standby cluster in pause mode does not recover from wiped DCS

Author: mbanck-cdCreated Feb 12, 2025Updated Sep 16, 2026

Labelsbug

What happened?

If a primary cluster is in maintenance mode and the DCS is wiped (e.g. because one decides to migrate from etcd to etcd3), it re-writes the /leader, /initialize and /config keys to DCS on the next HA loop (along with the /members keys) and re-acquires the leader lock:

Feb 12 10:34:27 pg2 patroni@15-example_cluster[6738]: 2025-02-12 10:34:27,500 INFO: PAUSE: no action. I am (pg2), the leader with the lock
Feb 12 10:34:29 pg2 systemd[1]: Stopping patroni@15-example_cluster.service - Patroni instance 15-example_cluster...
Feb 12 10:34:30 pg2 patroni@15-example_cluster[6738]: 2025-02-12 10:34:30,167 INFO: Leader key is not deleted and Postgresql is not stopped due paused state
Feb 12 10:34:30 pg2 systemd[1]: patroni@15-example_cluster.service: Deactivated successfully.
[...]
Feb 12 10:34:30 pg2 systemd[1]: Started patroni@15-example_cluster.service - Patroni instance 15-example_cluster.
Feb 12 10:34:30 pg2 patroni@15-example_cluster[10747]: 2025-02-12 10:34:30,586 INFO: Selected new etcd server http://10.0.3.51:2379
Feb 12 10:34:30 pg2 patroni@15-example_cluster[10747]: 2025-02-12 10:34:30,598 INFO: No PostgreSQL configuration items changed, nothing to reload.
Feb 12 10:34:30 pg2 patroni@15-example_cluster[10753]: /var/run/postgresql/:5432 - accepting connections
Feb 12 10:34:30 pg2 patroni@15-example_cluster[10747]: 2025-02-12 10:34:30,615 INFO: establishing a new patroni heartbeat connection to postgres
Feb 12 10:34:30 pg2 patroni@15-example_cluster[10747]: 2025-02-12 10:34:30,623 INFO: Changed archive_mode from 'on' to 'True' (restart might be required)
Feb 12 10:34:30 pg2 patroni@15-example_cluster[10747]: 2025-02-12 10:34:30,632 INFO: Reloading PostgreSQL configuration.
Feb 12 10:34:30 pg2 patroni@15-example_cluster[10757]: server signaled
Feb 12 10:34:31 pg2 patroni@15-example_cluster[10747]: 2025-02-12 10:34:31,797 INFO: PAUSE: acquired session lock as a leader
Feb 12 10:34:41 pg2 patroni@15-example_cluster[10747]: 2025-02-12 10:34:41,842 INFO: PAUSE: no action. I am (pg2), the leader with the lock

patronictl:

root@pg2:~# patronictl -c /etc/patroni/15-example_cluster.yml list
+ Cluster: 15-example_cluster (7470460811060857477) ---------+
| Member | Host       | Role    | State     | TL | Lag in MB |
+--------+------------+---------+-----------+----+-----------+
| pg1    | 10.0.3.229 | Replica | streaming |  1 |         0 |
| pg2    | 10.0.3.192 | Leader  | running   |  1 |           |
+--------+------------+---------+-----------+----+-----------+
 Maintenance mode: on

etcdctl output:

/postgresql-common/15-example_cluster/config
{"ttl":30,"loop_wait":10,"retry_timeout":10,"maximum_lag_on_failover":1048576,"slots":{"standby_cluster"
:{"cluster_type":"primary","type":"physical"}},"postgresql":{"create_replica_methods":["pgbackrest","pg_
clonecluster"],"use_pg_rewind":true,"pgbackrest":{"command":"/usr/bin/pgbackrest --stanza=15-example_clu
ster --log-level-file=info --delta restore","keep_data":true,"no_params":true},"recovery_conf":{"restore
_command":"/usr/bin/pgbackrest --stanza=15-example_cluster archive-get %f %p"},"pg_clonecluster":{"comma
nd":"/usr/share/patroni/pg_clonecluster_patroni"},"remove_data_directory_on_rewind_failure":true,"remove
_data_directory_on_diverged_timelines":true,"parameters":{"max_connections":100,"max_wal_senders":10},"p
g_hba":["local   all             all                                     peer","host    all
all             127.0.0.1/32            scram-sha-256","host    all             all             ::1/128
                scram-sha-256","host    all             all             10.0.3.192/24               scra
m-sha-256","local   replication     all                                     peer","host    replication
   all             127.0.0.1/32            scram-sha-256","host    replication     all             ::1/1
28                 scram-sha-256","host    replication     all             10.0.3.192/24               s
cram-sha-256","host    all             postgres_rewind 10.0.3.192/24               scram-sha-256"]},"pau
se":true}
/postgresql-common/15-example_cluster/initialize
7470460811060857477
/postgresql-common/15-example_cluster/leader
pg2
/postgresql-common/15-example_cluster/members/pg1
{"conn_url":"postgres://10.0.3.229:5432/postgres","api_url":"http://10.0.3.229:8008/patroni","state":"ru
nning","role":"replica","version":"4.0.4","xlog_location":84194584,"replication_state":"streaming","time
line":1,"pause":true}
/postgresql-common/15-example_cluster/members/pg2
{"conn_url":"postgres://10.0.3.192:5432/postgres","api_url":"http://10.0.3.192:8008/patroni","state":"ru
nning","role":"primary","version":"4.0.4","xlog_location":84194584,"timeline":1,"pause":true}
/postgresql-common/15-example_cluster/status
{"optime":84194584,"slots":{"pg1":84194584,"standby_cluster":84194584,"pg2":84194584},"retain_slots":["p
g1","pg2"]}

A standby cluster seems to only write the /members keys. This results in a state where patronictl thinks the cluster is not initialized and patronictl resume does not work:

Feb 12 11:02:14 pg4 patroni@15-example_cluster_standby[6737]: 2025-02-12 11:02:14,381 INFO: PAUSE: no action. I am (pg4), the standby leader with the lock
Feb 12 11:02:21 pg4 systemd[1]: Stopping patroni@15-example_cluster_standby.service - Patroni instance 15-example_cluster_standby...
Feb 12 11:02:21 pg4 patroni@15-example_cluster_standby[6737]: 2025-02-12 11:02:21,707 INFO: Leader key is not deleted and Postgresql is not stopped due paused state
Feb 12 11:02:21 pg4 systemd[1]: patroni@15-example_cluster_standby.service: Deactivated successfully.
[...]
Feb 12 11:02:21 pg4 systemd[1]: Started patroni@15-example_cluster_standby.service - Patroni instance 15-example_cluster_standby.
Feb 12 11:02:22 pg4 patroni@15-example_cluster_standby[10797]: 2025-02-12 11:02:22,156 INFO: Selected new etcd server http://10.0.3.51:2379
Feb 12 11:02:22 pg4 patroni@15-example_cluster_standby[10797]: 2025-02-12 11:02:22,162 INFO: No PostgreSQL configuration items changed, nothing to reload.
Feb 12 11:02:22 pg4 patroni@15-example_cluster_standby[10802]: /var/run/postgresql/:5432 - accepting connections
Feb 12 11:02:22 pg4 patroni@15-example_cluster_standby[10797]: 2025-02-12 11:02:22,173 INFO: establishing a new patroni heartbeat connection to postgres
Feb 12 11:02:22 pg4 patroni@15-example_cluster_standby[10805]: server signaled
Feb 12 11:02:22 pg4 patroni@15-example_cluster_standby[10797]: 2025-02-12 11:02:22,235 INFO: PAUSE: no action. I am (pg4)

patronictl:

root@pg4:~# patronictl -c /etc/patroni/15-example_cluster_standby.yml list
+ Cluster: 15-example_cluster_standby (uninitialized) -------+
| Member | Host       | Role    | State     | TL | Lag in MB |
+--------+------------+---------+-----------+----+-----------+
| pg4    | 10.0.3.120 | Replica | streaming |  1 |         0 |
| pg5    | 10.0.3.26  | Replica | streaming |  1 |         0 |
+--------+------------+---------+-----------+----+-----------+
root@pg4:~# patronictl -c /etc/patroni/15-example_cluster_standby.yml resume
Error: Cluster is not paused

etcdctl output:

/postgresql-common/15-example_cluster_standby/members/pg4
{"conn_url":"postgres://10.0.3.120:5432/postgres","api_url":"http://10.0.3.120:8008/patroni","state":"ru
nning","role":"replica","version":"4.0.4","xlog_location":84194584,"replication_state":"streaming","time
line":1,"pause":true}
/postgresql-common/15-example_cluster_standby/members/pg5
{"conn_url":"postgres://10.0.3.26:5432/postgres","api_url":"http://10.0.3.26:8008/patroni","state":"runn
ing","role":"replica","version":"4.0.4","xlog_location":84194584,"replication_state":"streaming","timeli
ne":1,"pause":true}

To recover, it seems one has to do the following:

stop patroni on all standby cluster nodes
wipe standby cluster keys from DCS
manually edit patroni.dynamic.json and change "pause": true to "pause": false on all nodes
start patroni on all nodes
wait till a standby leader got elected
run patronictl resume

So the question is: is this a bug or a feature? If it is not a bug, I think this needs better documentation.

Maybe the standby cluster case is just not considered when writing stuff back into DCS?

How can we reproduce it (as minimally and precisely as possible)?

Deploy a standby cluster
Put it into maintenance mode
Wipe its DCS data
Restart patroni

What did you expect to happen?

Standby cluster recovers and can be resumed.

Patroni/PostgreSQL/DCS version

Patroni version: 4.0.4
PostgreSQL version: n/a
DCS (and its version): etcd-3.4.23 (v2/v3 API)

Patroni configuration file

yaml

patronictl show-config

yaml

Patroni log files

bash

see above

PostgreSQL log files

bash

n/a

Have you tried to use GitHub issue search?

Anything else we need to know?

No response

Source: patroni/patroni

View original on GitHub View discussion on GitHub