Baike.dev
Log in
Back to tool/Back to issues
#3603·patroni

Patroni is not checking or reporting on pg_is_in_recovery() for current leader.

Author: guriandoroCreated May 6, 2026Updated Aug 18, 2026

What happened?

After a failover, the node that was flagged as leader by Patroni was in read-only mode. Patroni error log and patronictl list outputs showed as if the cluster was healthy, when in reality no writes were possible.

How can we reproduce it (as minimally and precisely as possible)?

  • Start with a 2 node patroni cluster and a pgbackrest to nfs
  • Configure pgbackrest and set archive_command and restore_command as needed.
  • using iptables on the replica, block returning traffic from the nfs
iptables -I INPUT 1 -p tcp -s "$SERVER" --sport 2049 -j DROP
  • kill -9 pidof_patroni on the leader
  • the replica will be correctly elected as leader from patroni side, but from postgres side the following will never complete:
archive-get command begin 2.58.0: [00000002.history, pg_wal/RECOVERYHISTORY] ...

Patroni shows a seemingly healthy node, wthen in reality the node is in read-only mode.

[root@node1 /]# patronictl list
+ Cluster: cluster1 (7636233032694706021) --------+----+-------------+-----+------------+-----+
|      Member     |    Host    |  Role  |  State  | TL | Receive LSN | Lag | Replay LSN | Lag |
+-----------------+------------+--------+---------+----+-------------+-----+------------+-----+
| cluster113949-1 | 172.16.2.3 | Leader | running |  1 |             |     |            |     |
+-----------------+------------+--------+---------+----+-------------+-----+------------+-----+

[root@node1 journalctl -u patroni | tail
May 06 02:51:53 node1 patroni[4000]: 2026-05-06 02:51:53,951 INFO: no action. I am (cluster113949-1), the leader with the lock
May 06 02:52:03 node1 patroni[4000]: 2026-05-06 02:52:03,951 INFO: no action. I am (cluster113949-1), the leader with the lock
May 06 02:52:13 node1 patroni[4000]: 2026-05-06 02:52:13,952 INFO: no action. I am (cluster113949-1), the leader with the lock
May 06 02:52:23 node1 patroni[4000]: 2026-05-06 02:52:23,951 INFO: no action. I am (cluster113949-1), the leader with the lock
May 06 02:52:33 node1 patroni[4000]: 2026-05-06 02:52:33,951 INFO: no action. I am (cluster113949-1), the leader with the lock
May 06 02:52:43 node1 patroni[4000]: 2026-05-06 02:52:43,951 INFO: no action. I am (cluster113949-1), the leader with the lock
May 06 02:52:53 node1 patroni[4000]: 2026-05-06 02:52:53,951 INFO: no action. I am (cluster113949-1), the leader with the lock
May 06 02:53:03 node1 patroni[4000]: 2026-05-06 02:53:03,950 INFO: no action. I am (cluster113949-1), the leader with the lock
May 06 02:53:13 node1 patroni[4000]: 2026-05-06 02:53:13,951 INFO: no action. I am (cluster113949-1), the leader with the lock
May 06 02:53:23 node1 patroni[4000]: 2026-05-06 02:53:23,951 INFO: no action. I am (cluster113949-1), the leader with the lock

[root@node1 /]# psql -Upostgres -c "select pg_is_in_recovery()";
 pg_is_in_recovery
-------------------
 t
(1 row)

[root@node1 /]# date
Wed May  6 02:53:46 UTC 2026

What did you expect to happen?

Patroni to check pg_is_in_recovery() on the leader node.

If nothing else, this should be logged as a warning. Additionally, it would be nice if patronictl list shows this in some way.

Patroni/PostgreSQL/DCS version

  • Patroni version: patroni 4.1.2
  • PostgreSQL version: (18.1)
  • DCS (and its version): etcdctl version: 3.5.12 -- API version: 3.5

Patroni configuration file

yaml
scope: cluster1
name: cluster113949-1
restapi:
 listen: 0.0.0.0:8008
 connect_address: 172.16.2.3:8008
etcd3:
 host: 172.16.2.3:2379
bootstrap:
 dcs:
    ttl: 30
    loop_wait: 10
    retry_timeout: 10
    maximum_lag_on_failover: 1048576
    postgresql:
     use_pg_rewind: true
     use_slots: true
     parameters:
       wal_level: replica
       hot_standby: "on"
       max_wal_senders: 10
       max_replication_slots: 10
       wal_log_hints: "on"
       archive_mode: "on"
       archive_timeout: 600s
       archive_command: "cp -f %p /home/postgres/archived/%f"
     recovery_conf:
       restore_command: "cp /home/postgres/archived/%f %p"
 initdb:
 - encoding: UTF8
 - data-checksums
 pg_hba:
 - host replication replicator 172.16.2.3/24 md5
 - host replication replicator 0.0.0.0/0 md5
 - host replication replicator 127.0.0.1/32 trust
 - host all all 172.16.2.3/24 md5
 - host all all 0.0.0.0/0 md5
 post_init: /usr/local/bin/setup_cluster.sh
 users:
    admin:
     password: verysecretpassword1^
     options:
       - createrole
       - createdb
postgresql:
 listen: 0.0.0.0:5432
 connect_address: 172.16.2.3:5432
 data_dir: "/var/lib/pgsql/18/data"
 bin_dir: "/usr/pgsql-18/bin"
 pgpass: /tmp/pgpass0
 authentication:
    replication:
     username: replicator
     password: verysecretpassword1^
    superuser:
     username: postgres
     password: verysecretpassword1^
 parameters:
    unix_socket_directories: '/var/run/postgresql'
watchdog:
 mode: off
tags:
    nofailover: false
    noloadbalance: false
    clonefrom: false
    nosync: false

patronictl show-config

yaml
loop_wait: 10
maximum_lag_on_failover: 1048576
postgresql:
  parameters:
    archive_command: pgbackrest --log-level-console=info --stanza=db archive-push %p
    archive_mode: 'on'
    archive_timeout: 600s
    hot_standby: 'on'
    max_replication_slots: 10
    max_wal_senders: 10
    wal_level: replica
    wal_log_hints: 'on'
  recovery_conf:
    restore_command: pgbackrest --log-level-console=info --stanza=db archive-get %f %p
  use_pg_rewind: true
  use_slots: true
retry_timeout: 10
ttl: 30

Patroni log files

bash
May 06 00:20:54 node1 patroni[4000]: 2026-05-06 00:20:54,003 INFO: no action. I am (cluster113949-1), a secondary, and following a leader (cluster1-0)
May 06 00:21:03 node1 patroni[4000]: 2026-05-06 00:21:03,960 INFO: no action. I am (cluster113949-1), a secondary, and following a leader (cluster1-0)
May 06 00:21:13 node1 patroni[4000]: 2026-05-06 00:21:13,775 WARNING: Request failed to cluster1-0: GET http://172.16.2.2:8008/patroni (HTTPConnection>
May 06 00:21:13 node1 patroni[4000]: 2026-05-06 00:21:13,906 INFO: promoted self to leader by acquiring session lock
May 06 00:21:13 node1 patroni[5831]: server promoting
May 06 00:21:13 node1 patroni[4000]: 2026-05-06 00:21:13,907 INFO: Lock owner: cluster113949-1; I am cluster113949-1
May 06 00:21:13 node1 patroni[4000]: 2026-05-06 00:21:13,992 INFO: updated leader lock during promote
May 06 00:21:23 node1 patroni[4000]: 2026-05-06 00:21:23,907 INFO: Lock owner: cluster113949-1; I am cluster113949-1
May 06 00:21:23 node1 patroni[4000]: 2026-05-06 00:21:23,952 INFO: updated leader lock during promote
May 06 00:21:33 node1 patroni[4000]: 2026-05-06 00:21:33,953 INFO: no action. I am (cluster113949-1), the leader with the lock
May 06 00:21:43 node1 patroni[4000]: 2026-05-06 00:21:43,952 INFO: no action. I am (cluster113949-1), the leader with the lock
May 06 00:21:53 node1 patroni[4000]: 2026-05-06 00:21:53,952 INFO: no action. I am (cluster113949-1), the leader with the lock

PostgreSQL log files

bash
2026-05-06 00:21:13.909 UTC [4046] LOG:  received promote request
2026-05-06 00:21:13.909 UTC [4073] FATAL:  terminating walreceiver process due to administrator command
2026-05-06 00:21:13.917 P00   INFO: archive-get command begin 2.58.0: [00000002.history, pg_wal/RECOVERYHISTORY] --exec-id=5832-125bcde4 --log-level-console=info --pg1-path=/var/lib/pgsql/18/data --repo1-path=/nfs-mnt/pgbackrest/repo --stanza=db
2026-05-06 00:22:55.596 UTC [5911] ERROR:  cannot execute CREATE TABLE in a read-only transaction
2026-05-06 00:22:55.596 UTC [5911] STATEMENT:  create table t1 (id int primary key)

Have you tried to use GitHub issue search?

  • Yes

Anything else we need to know?

No response

Source: patroni/patroni

View original on GitHubView discussion on GitHub