Baike.dev
Log in
Back to tool/Back to issues
#3717·patroni

Replica removes primary_conninfo and stops streaming after a brief DCS blip, with no failover and a confirmed-alive primary

Author: ceyxasmCreated Sep 8, 2026Updated Sep 17, 2026

What happened?

A brief Kubernetes API (DCS) outage caused the replica to remove primary_conninfo and primary_slot_name, kill the walreceiver, and fall back to archive recovery. It never reconnected. It stayed that way for 4h23m, reporting itself as a healthy replica following the leader, until we restarted the pod.

The distinguishing detail from #3686 and #3675: there was no failover. The primary was never demoted, no re-election occurred, and the timeline never changed. Throughout the window the primary logged only no action. I am (postgres-0), the leader with the lock.

The replica also had contact with the primary at the moment it disconnected. 158ms before stripping its config, its own health check of the primary returned:

{"role": "primary", "timeline": 14,
 "replication": [{"application_name": "postgres-1", "state": "streaming", "sync_state": "async"}]}

and it logged Primary (postgres-0) is still alive twice in that same second.

Impact: reads are served by the replica, so the application returned data minutes to an hour stale. Rows written to the primary and read back immediately were not found. Access tokens written to the primary were rejected as invalid for 30 minutes, until the archived WAL containing them reached the replica.

How can we reproduce it (as minimally and precisely as possible)?

We have not reproduced it deliberately. Observed conditions:

  1. Two-node cluster (one primary, one replica), Kubernetes DCS, failsafe_mode: true.
  2. Cluster healthy and streaming normally.
  3. The replica's Patroni loses access to the Kubernetes API for a few seconds ([Errno 111] Connection refused, then K8sConnectionFailed('No more API server nodes in the cluster')).
  4. The primary is unaffected and keeps its leader lock throughout.
  5. Within ~13 seconds the replica strips primary_conninfo / primary_slot_name and terminates the walreceiver.
  6. The DCS becomes reachable again seconds later. The replica does not restore the configuration.

What did you expect to happen?

Either:

  1. The replica keeps streaming. It had just confirmed the primary was alive and holding the leader lock, and no failover was in progress.
  2. Or, failing that, the HA loop restores primary_conninfo and resumes streaming once the DCS is reachable again, rather than remaining in archive recovery indefinitely while reporting itself healthy.

Patroni/PostgreSQL/DCS version

Patroni 4.0.5
PostgreSQL 15.13
DCS: Kubernetes (use_endpoints: true)

Patroni configuration file

yaml
scope: attentiontag-postgres
name: ${POD_NAME}
namespace: ${POD_NAMESPACE}

restapi:
  listen: 0.0.0.0:8008
  connect_address: ${POD_IP}:8008

kubernetes:
  use_endpoints: true
  labels:
    app: postgres
  role_label: role

bootstrap:
  dcs:
    failsafe_mode: true
    slots:
      postgres_0:
        type: physical
      postgres_1:
        type: physical
    postgresql:
      parameters:
        max_locks_per_transaction: 512
  initdb:
    - encoding: UTF8
  pg_hba:
    - local all all trust
    - host replication replicator 0.0.0.0/0 scram-sha-256
    - host all all 0.0.0.0/0 scram-sha-256
  post_bootstrap: /scripts/bootstrap.sh

postgresql:
  listen: 0.0.0.0:5432
  connect_address: ${POD_IP}:5432
  data_dir: /home/postgres/pgdata/data
  pgpass: /tmp/pgpass0
  authentication:
    replication:
      username: replicator
      password: ${PATRONI_REPLICATION_PASSWORD}
    superuser:
      username: ${PATRONI_SUPERUSER_USERNAME}
      password: ${PATRONI_SUPERUSER_PASSWORD}
  recovery_conf:
    restore_command: python3 /scripts/wal_archive.py restore %f %p
  parameters:
    shared_preload_libraries: timescaledb
    wal_level: replica
    archive_mode: "on"
    archive_command: python3 /scripts/wal_archive.py archive %p %f
    hot_standby_feedback: "on"
    max_replication_slots: 5
    max_connections: 200
    wal_keep_size: 128MB
    shared_buffers: 2GB
    effective_cache_size: 6GB
    maintenance_work_mem: 256MB
    work_mem: 10485kB
    wal_buffers: 16MB
    min_wal_size: 512MB
    default_statistics_target: "100"
    random_page_cost: "1.1"
    checkpoint_completion_target: "0.9"
    max_locks_per_transaction: "512"
    autovacuum_max_workers: "10"
    autovacuum_naptime: 10
    jit: "off"
    effective_io_concurrency: "256"
    idle_in_transaction_session_timeout: "300s"

patronictl show-config

patronictl could not be run from a one-shot kubectl exec (its Kubernetes object cache does not warm up in time and it exits with KubernetesError: Kubernetes API is not responding properly). The equivalent DCS configuration, from the /config REST endpoint:

yaml
failsafe_mode: true
postgresql:
  parameters:
    max_locks_per_transaction: 512
slots:
  postgres_0:
    type: physical
  postgres_1:
    type: physical

Patroni log files

Replica. DCS lost, then teardown 13 seconds later:

20:44:47,170 ERROR: Request to server failed: MaxRetryError(... [Errno 111] Connection refused)
20:44:48,173 ERROR: ObjectCache.run K8sConnectionFailed('No more API server nodes in the cluster')
20:44:48,175 ERROR: ObjectCache.run K8sConnectionFailed('No more API server nodes in the cluster')
20:45:00,529 WARNING: Primary (postgres-0) is still alive
20:45:00,616 INFO: following a different leader because i am not the healthiest node
20:45:00,627 WARNING: Primary (postgres-0) is still alive
20:45:00,666 INFO: following a different leader because i am not the healthiest node

Replica, for the following 4h23m, every 10 seconds:

00:52:20,638 INFO: no action. I am (postgres-1), a secondary, and following a leader (postgres-0)

Primary, across the same window — no demotion, no re-election:

20:44:40,699 INFO: no action. I am (postgres-0), the leader with the lock
20:45:00,977 INFO: no action. I am (postgres-0), the leader with the lock
20:49:50,723 INFO: no action. I am (postgres-0), the leader with the lock

PostgreSQL log files

Replica, teardown:

2026-09-07 20:45:00.665 UTC [25] LOG:  received SIGHUP, reloading configuration files
2026-09-07 20:45:00.670 UTC [25] LOG:  parameter "primary_conninfo" removed from configuration file, reset to default
2026-09-07 20:45:00.670 UTC [25] LOG:  parameter "primary_slot_name" removed from configuration file, reset to default
2026-09-07 20:45:00.684 UTC [29] LOG:  WAL receiver process shutdown requested
2026-09-07 20:45:00.687 UTC [73] FATAL:  terminating walreceiver process due to administrator command
2026-09-07 20:45:04.565 UTC [29] LOG:  invalid resource manager ID 101 at 8/E96B1EE8
2026-09-07 20:45:05.398 UTC [29] LOG:  waiting for WAL to become available at 8/E9002000

Replica, afterwards — only hourly archive restores, never streaming again:

2026-09-07 21:04:19.124 UTC [29] LOG:  restored log file "0000000E00000008000000E9" from archive
2026-09-07 22:01:35.761 UTC [29] LOG:  restored log file "0000000E00000008000000EA" from archive
2026-09-07 23:00:42.710 UTC [29] LOG:  restored log file "0000000E00000008000000EB" from archive
2026-09-07 23:48:39.231 UTC [29] LOG:  restored log file "0000000E00000008000000EC" from archive
2026-09-08 00:47:06.045 UTC [29] LOG:  restored log file "0000000E00000008000000ED" from archive
2026-09-08 00:52:22.044 UTC [29] LOG:  waiting for WAL to become available at 8/EE000739

Have you tried to use GitHub issue search?

  • Yes

Anything else we need to know?

SHOW primary_conninfo on the replica returned empty, and pg_stat_replication on the primary returned no rows, for the full duration.

/cluster did expose the degraded state, but role still read as a normal replica, so anything keying off Patroni's own status saw a healthy cluster:

json
{"name": "postgres-1", "role": "replica", "state": "in archive recovery", "lag": 2062944}

Restarting the replica pod resolved it immediately: configuration was rebuilt from the DCS, streaming resumed at 0.005s replay lag, and no rebuild was required.

Related: #3686 and #3675 report the same non-recovery after follow(node_to_follow=None) strips the recovery configuration. Both involve a failover and a new timeline. This case shows the same outcome with no failover, no re-election, no timeline change, and a primary that never lost its leader lock.

Source: patroni/patroni

View original on GitHubView discussion on GitHub