Baike.dev
Log in
Back to tool/Back to issues
#3478·patroni

Patroni failed to correctly handle the status information of a faulty standby node when using k8s as the DCS

Author: ice61Created Oct 23, 2025Updated Oct 24, 2025

What happened?

Hi,

First of all, thank you for the great product! The current high-availability cluster is deployed in a containerized manner, using k8s as the DCS, with each Postgres+Patroni node running as a separate pod. Under normal circumstances, each cluster has one leader node "a" and two replica nodes "b" and "c" (one sync standby and one replica), with both replica nodes in the "streaming" state. If the VM node hosting pod "b" undergoes network throttling (making Patroni unable to perform communication with the DCS), then even if replication between "a" and "b" has already been interrupted, patronictl list will still show node "b" as "streaming".

I suspect that when using k8s as the DCS, Patroni stores each node's data in the corresponding pod's annotations, and does not have a mechanism similar to "renewTime" during leader election (similar to leases in etcd). This results in a situation where, even if Patroni cannot refresh the node status in the touch_member function, the other nodes remain unaware that the faulty node's status has expired.

Would it be possible to consider modifying Patroni's code to set an expiration time for the status information of each node?

How can we reproduce it (as minimally and precisely as possible)?

Limit the network bandwidth(one Mb per second) of the virtual machine hosting the standby node pod, or simply shut down the virtual machine so that the pod enters the terminating state. In short, keep the pod present, but ensure that the Patroni inside the pod cannot communicate with the DCS.

What did you expect to happen?

Consistent with the behavior when using etcd as the DCS. If a node cannot communicate with the DCS (calls the touch_member function) for a long time, the patronictl list command will no longer display this node. This node will be removed from the cluster.

Patroni/PostgreSQL/DCS version

  • Patroni version: 4.04
  • PostgreSQL version: 14.5
  • DCS (and its version): k8s, 1.16.15

Patroni configuration file

yaml
bootstrap:
  dcs:
    failsafe_mode: true
    loop_wait: 10
    master_start_timeout: 120
    master_stop_timeout: 10
    maximum_lag_on_failover: 33554432
    postgresql:
      parameters:
        archive_mode: 'on'
        archive_timeout: 1800s
        cron.database_name: seasql
        hot_standby: 'on'
        hot_standby_feedback: true
        idle_in_transaction_session_timeout: 10min
        idle_session_timeout: 10min
        lock_timeout: 10min
        log_autovacuum_min_duration: 0
        log_checkpoints: 'on'
        log_connections: 'on'
        log_destination: stderr
        log_disconnections: 'on'
        log_line_prefix: '%t [%p]: [%l-1] %c %x %d %u %a %h '
        log_lock_waits: 'on'
        log_min_duration_statement: 1000
        log_replication_commands: true
        log_statement: ddl
        log_temp_files: 0
        log_truncate_on_rotation: true
        logging_collector: true
        max_connections: '2000'
        max_logical_replication_workers: 8
        max_prepared_transactions: 0
        max_replication_slots: 10
        max_slot_wal_keep_size: 20GB
        max_standby_archive_delay: 10min
        max_standby_streaming_delay: 3min
        max_sync_workers_per_subscription: 6
        max_wal_senders: 10
        max_wal_size: 10GB
        password_encryption: md5
        promote_trigger_file: promote.signal
        superuser_reserved_connections: 10
        tcp_keepalives_idle: 900
        tcp_keepalives_interval: 100
        timezone: Asia/Shanghai
        track_commit_timestamp: true
        track_functions: none
        unix_socket_directories: /tmp
        vacuum_defer_cleanup_age: 0
        wal_compression: 'on'
        wal_keep_size: 8GB
        wal_level: hot_standby
        wal_log_hints: 'on'
        wal_receiver_status_interval: 1s
        wal_receiver_timeout: 60s
      remove_data_directory_on_creating_replication_failed: true
      remove_data_directory_on_rewind_failure: true
      use_pg_rewind: true
      use_slots: true
    retry_timeout: 10
    synchronous_mode: true
    synchronous_node_count: 1
    ttl: 30
  initdb:
  - auth-host: md5
  - auth-local: trust
  - data-checksums
  - encoding: UTF8
  - locale: en_US.UTF-8
  post_init: /scripts/post_init.sh "zalandos"
  users:
    zalandos:
      options:
      - CREATEDB
      - NOLOGIN
      password: ''
kubernetes:
  bypass_api_service: true
  labels:
    application: spilo
  leader_label_value: master
  pod_ip: 177.177.150.184
  port: tcp://10.96.0.1:443
  port_443_tcp: tcp://10.96.0.1:443
  port_443_tcp_addr: 10.96.0.1
  port_443_tcp_port: '443'
  port_443_tcp_proto: tcp
  ports:
  - name: seasql
    port: 1523
  - name: patroni
    port: 8008
  role_label: spilo-role
  scope_label: cluster-name
  service_host: 10.96.0.1
  service_port: '443'
  service_port_https: '443'
  standby_leader_label_value: master
  use_endpoints: true
log:
  dateformat: '%Y-%m-%d %H:%M:%S %z'
  dir: /var/lib/seasql/ssdata/log
  file_num: 5
  file_size: 33554432
  format: '%(asctime)s %(levelname)s: %(message)s'
  level: INFO
namespace: service-software
postgresql:
  authentication:
    replication:
      password: Or10TL+hRs.=N@0l54
      username: standby
    superuser:
      password: Or10TL+hRs.=N@0l54
      username: ssadmin
  basebackup_fast_xlog:
    command: /scripts/basebackup.sh
    retries: 2
  bin_dir: /usr/lib/seasql/14/bin
  callbacks:
    on_role_change: /scripts/on_role_change.sh zalandos true
  connect_address: 177.177.150.184:1523
  create_replica_method:
  - basebackup_fast_xlog
  data_dir: /var/lib/seasql//ssdata/data
  database: seasql
  listen: '*:1523'
  name: seasql-base-0
  parameters:
    archive_command: /bin/true
    extwlist.custom_path: /scripts
    extwlist.extensions: btree_gin,btree_gist,citext,extra_window_functions,first_last_agg,hll,hstore,hypopg,intarray,ltree,pgcrypto,pgq,pgq_node,pg_trgm,postgres_fdw,tablefunc,uuid-ossp,timescaledb,pg_partman
    log_destination: csvlog
    log_directory: ../log
    log_file_mode: '0644'
    log_filename: seasql-%u.log
    log_rotation_age: 1d
    log_truncate_on_rotation: 'on'
    logging_collector: 'on'
    pg_stat_statements.track_utility: 'off'
    shared_buffers: 1638MB
    shared_preload_libraries: pg_stat_statements,pgextwlist,set_user,timescaledb,pg_cron,pg_stat_kcache
    ssl: 'on'
    ssl_cert_file: /run/certs/server.crt
    ssl_key_file: /run/certs/server.key
  pg_hba:
  - local   all             all                                   trust
  - host    all             all                0.0.0.0/0          md5
  - host    all             all                ::/0               md5
  - local   replication     standby                    trust
  - hostssl replication     standby all                md5
  pgpass: /run/seasql/pgpass
  use_unix_socket: true
  use_unix_socket_repl: true
restapi:
  connect_address: 177.177.150.184:8008
  listen: :8008
scope: seasql-base
seasql:
  parameters:
    shared_buffers: 32MB
tags:
  clonefrom: true
  nofailover: false
  noloadbalance: false
  nosync: false
watchdog:
  device: /dev/watchdog
  mode: false

patronictl show-config

yaml
failsafe_mode: true
loop_wait: 10
master_start_timeout: 120
master_stop_timeout: 10
maximum_lag_on_failover: 33554432
postgresql:
  parameters:
    archive_mode: 'on'
    archive_timeout: 1800s
    cron.database_name: seasql
    hot_standby: 'on'
    hot_standby_feedback: true
    idle_in_transaction_session_timeout: 10min
    idle_session_timeout: 10min
    lock_timeout: 10min
    log_autovacuum_min_duration: 0
    log_checkpoints: 'on'
    log_connections: 'on'
    log_destination: stderr
    log_directory: /seasql/log
    log_disconnections: 'on'
    log_filename: postgresql-%a-%H%M.log
    log_line_prefix: '%t [%p]: [%l-1] %c %x %d %u %a %h '
    log_lock_waits: 'on'
    log_min_duration_statement: 1000
    log_replication_commands: true
    log_rotation_size: 100MB
    log_statement: ddl
    log_temp_files: 0
    log_truncate_on_rotation: true
    logging_collector: true
    max_connections: '2000'
    max_logical_replication_workers: 8
    max_prepared_transactions: 0
    max_replication_slots: 10
    max_slot_wal_keep_size: 20GB
    max_standby_archive_delay: 10min
    max_standby_streaming_delay: 3min
    max_sync_workers_per_subscription: 6
    max_wal_senders: 10
    max_wal_size: 10GB
    password_encryption: md5
    promote_trigger_file: promote.signal
    superuser_reserved_connections: 10
    tcp_keepalives_idle: 900
    tcp_keepalives_interval: 100
    timezone: Asia/Shanghai
    track_commit_timestamp: true
    track_functions: none
    unix_socket_directories: /tmp
    vacuum_defer_cleanup_age: 0
    wal_compression: 'on'
    wal_keep_size: 8GB
    wal_level: hot_standby
    wal_log_hints: 'on'
    wal_receiver_status_interval: 1s
    wal_receiver_timeout: 60s
  remove_data_directory_on_creating_replication_failed: true
  remove_data_directory_on_rewind_failure: true
  use_pg_rewind: true
  use_slots: true
retry_timeout: 10
synchronous_mode: true
synchronous_node_count: 1
ttl: 30

Patroni log files

bash
2025-10-23 14:28:14 +0800 INFO: DCS is not accessible
2025-10-23 14:28:14 +0800 WARNING: Loop time exceeded, rescheduling immediately.
2025-10-23 14:28:15 +0800 ERROR: ObjectCache.run K8sConnectionFailed('No more API server nodes in the cluster')
2025-10-23 14:28:16 +0800 WARNING: Retrying (Retry(total=0, connect=None, read=None, redirect=0, status=None)) after connection broken by 'ReadTimeoutError("HTTPSConnectionPool(host='10.96.0.1', port=443): Read timed out. (read timeout=2.5)")': /api/v1/namespaces/service-software/endpoints?labelSelector=application%3Dspilo%2Ccluster-name%3Dseasql-base
2025-10-23 14:28:16 +0800 ERROR: Request to server https://10.96.0.1:443 failed: ClosedPoolError("HTTPSConnectionPool(host='10.96.0.1', port=443): Pool is closed.")
2025-10-23 14:28:17 +0800 ERROR: ObjectCache.run K8sConnectionFailed('No more API server nodes in the cluster')
2025-10-23 14:28:17 +0800 WARNING: Retrying (Retry(total=0, connect=None, read=None, redirect=0, status=None)) after connection broken by 'ReadTimeoutError("HTTPSConnectionPool(host='10.96.0.1', port=443): Read timed out. (read timeout=2.5)")': /api/v1/namespaces/service-software/pods?labelSelector=application%3Dspilo%2Ccluster-name%3Dseasql-base
2025-10-23 14:28:17 +0800 ERROR: Request to server https://10.96.0.1:443 failed: ClosedPoolError("HTTPSConnectionPool(host='10.96.0.1', port=443): Pool is closed.")
2025-10-23 14:28:18 +0800 ERROR: ObjectCache.run K8sConnectionFailed('No more API server nodes in the cluster')
2025-10-23 14:28:19 +0800 WARNING: Retrying (Retry(total=0, connect=None, read=None, redirect=0, status=None)) after connection broken by 'ReadTimeoutError("HTTPSConnectionPool(host='10.96.0.1', port=443): Read timed out. (read timeout=2.5)")': /api/v1/namespaces/service-software/endpoints?labelSelector=application%3Dspilo%2Ccluster-name%3Dseasql-base
2025-10-23 14:28:19 +0800 ERROR: Request to server https://10.96.0.1:443 failed: ClosedPoolError("HTTPSConnectionPool(host='10.96.0.1', port=443): Pool is closed.")
2025-10-23 14:28:20 +0800 ERROR: ObjectCache.run K8sConnectionFailed('No more API server nodes in the cluster')
2025-10-23 14:28:21 +0800 WARNING: Retrying (Retry(total=0, connect=None, read=None, redirect=0, status=None)) after connection broken by 'ReadTimeoutError("HTTPSConnectionPool(host='10.96.0.1', port=443): Read timed out. (read timeout=2.5)")': /api/v1/namespaces/service-software/pods?labelSelector=application%3Dspilo%2Ccluster-name%3Dseasql-base
2025-10-23 14:28:21 +0800 ERROR: Request to server https://10.96.0.1:443 failed: ClosedPoolError("HTTPSConnectionPool(host='10.96.0.1', port=443): Pool is closed.")
2025-10-23 14:28:22 +0800 ERROR: ObjectCache.run K8sConnectionFailed('No more API server nodes in the cluster')
2025-10-23 14:28:23 +0800 WARNING: Retrying (Retry(total=0, connect=None, read=None, redirect=0, status=None)) after connection broken by 'ReadTimeoutError("HTTPSConnectionPool(host='10.96.0.1', port=443): Read timed out. (read timeout=2.5)")': /api/v1/namespaces/service-software/endpoints?labelSelector=application%3Dspilo%2Ccluster-name%3Dseasql-base
2025-10-23 14:28:23 +0800 ERROR: Request to server https://10.96.0.1:443 failed: ClosedPoolError("HTTPSConnectionPool(host='10.96.0.1', port=443): Pool is closed.")
2025-10-23 14:28:24 +0800 ERROR: ObjectCache.run K8sConnectionFailed('No more API server nodes in the cluster')
2025-10-23 14:28:24 +0800 ERROR: get_cluster
Traceback (most recent call last):
  File "/usr/lib/python3.9/site-packages/patroni/dcs/kubernetes.py", line 980, in __load_cluster
    self._wait_caches(stop_time)
  File "/usr/lib/python3.9/site-packages/patroni/dcs/kubernetes.py", line 860, in _wait_caches
    raise RetryFailedError('Exceeded retry deadline')
patroni.utils.RetryFailedError: Exceeded retry deadline
2025-10-23 14:28:24 +0800 INFO: establishing a new patroni heartbeat connection to postgres
2025-10-23 14:28:24 +0800 WARNING: Retrying (Retry(total=0, connect=None, read=None, redirect=0, status=None)) after connection broken by 'ReadTimeoutError("HTTPSConnectionPool(host='10.96.0.1', port=443): Read timed out. (read timeout=2.5)")': /api/v1/namespaces/service-software/pods?labelSelector=application%3Dspilo%2Ccluster-name%3Dseasql-base

PostgreSQL log files

bash
Nothing important.

Have you tried to use GitHub issue search?

  • Yes

Anything else we need to know?

[root@seasql-base-1 seasql]# patronictl -c /var/lib/seasql/seasql.yml list Cluster: seasql-base (7538711463146062125) ---------+------------------+----+-----------+-----------------+ | Member | Host | Role | State | TL | Lag in MB | Tags | +---------------+----------------------+--------------+------------------+----+-----------+-----------------+ | seasql-base-0 | 177.177.150.184:1523 | Replica | streaming | 11 | 55 | clonefrom: true | <===== can not communicate with the DCS | seasql-base-1 | 177.177.182.88:1523 | Sync Standby | streaming | 11 | 0 | clonefrom: true | | seasql-base-2 | 177.177.109.246:1523 | Leader | running | 11 | | clonefrom: true | +---------------+----------------------+--------------+------------------+----+-----------+-----------------+

As you can see, the streaming replication between the leader and seasql-base-0 has been disconnected due to network throttling.

seasql=# select * from pg_replication_slots ; -[ RECORD 1 ]-------+-------------- slot_name | seasql_base_1 plugin | slot_type | physical datoid | database | temporary | f active | t active_pid | 479 xmin | 6659989 catalog_xmin | restart_lsn | A/BD0D8240 confirmed_flush_lsn | wal_status | reserved safe_wal_size | 21490728384 two_phase | f -[ RECORD 2 ]-------+-------------- slot_name | seasql_base_0 plugin | slot_type | physical datoid | database | temporary | f active | f active_pid | xmin | 6584625 catalog_xmin | restart_lsn | A/AD42A4A8 confirmed_flush_lsn | wal_status | reserved safe_wal_size | 21222292928 two_phase | f

Source: patroni/patroni

View original on GitHubView discussion on GitHub