wait_for_user_backends_to_close() returns None when there are no user backends, so on_safepoint() is skipped and the watchdog stays armed

What happened?

Since #3661 (released in 4.1.4), Postgresql._do_stop() only calls the on_safepoint callback when PostmasterProcess.wait_for_user_backends_to_close() returns a truthy value:

# patroni/postgresql/__init__.py:920
if on_safepoint and postmaster.wait_for_user_backends_to_close(stop_timeout):
    on_safepoint()

But that method has three code paths that return a falsy None, and one of them is the normal, expected case of having no client backends at all:

# patroni/postgresql/postmaster.py:180-235
def wait_for_user_backends_to_close(self, stop_timeout: Optional[float]) -> Optional[bool]:
    """...
    :returns: ``True`` if there were no user backends or all of them closed in time; ``None`` if
        the list of children could not be retrieved or some backends were still alive after
        *stop_timeout*.
    """
    try:
        children = self.children()
    except psutil.Error:
        return logger.debug('Failed to get list of postmaster children')   # (1) -> None

    user_backends: List[psutil.Process] = []
    for child in children:
        ...
    if user_backends:                                                      # (2) empty list skips this
        ...
        if stop_timeout and live:
            logger.warning('Backends still alive after %s: %s', ...)       # (3) -> None
        else:
            logger.debug("Backends closed")
            return True                                                    # only truthy return

When user_backends is empty the if block is skipped entirely and the function falls through and returns None — contradicting its own docstring, which documents True for exactly that case.

The consequence is that on_safepoint() — which is how Ha.demote() disarms the hardware watchdog (on_safepoint=self.watchdog.disable, patroni/ha.py:1645) — is never called. The watchdog stays armed and fires, hard-resetting the host.

Impact

On a Patroni leader with watchdog.mode: automatic (or required) and a softdog/hardware watchdog, losing DCS connectivity causes the node to hard-reboot instead of cleanly demoting:

1. leader loses contact with etcd
2. Ha._handle_dcs_error() → demote('offline') → PostgreSQL is demoted, on_safepoint is skipped because there are no client backends
3. the watchdog is still armed with ttl - safety_margin seconds left
4. the kernel watchdog reboots the machine ~25s later

This is a regression: in 4.1.0 the call was unconditional and the watchdog was always disarmed.

Note that this only reproduces when the node has no client backends at demotion time. With synchronous_mode: on, primary_stop_timeout() returns a real value and stop_timeout is not None, but that does not change whether user_backends is empty — so sync mode does not avoid it.

How can we reproduce it (as minimally and precisely as possible)?

Reproduction

3-node cluster (PostgreSQL 16, Patroni 4.1.5, watchdog.mode: automatic, ttl: 30, loop_wait: 10, safety_margin: 5 → watchdog timeout 25s, synchronous_mode: false).

1. Confirm the leader holds the watchdog:
   复制

   # fuser -v /dev/watchdog
   /dev/watchdog:  postgres 1461 F.... patroni
   # cat /sys/class/watchdog/watchdog0/state /sys/class/watchdog/watchdog0/timeout
   active
   25

2. Disconnect the leader's network interface (or drop DCS traffic).
3. Observe patroni.log:
   复制

   INFO  ha:_handle_dcs_error: demoting self because DCS is not accessible and I was a leader
   INFO  ha:demote: Demoting self (offline)
   DEBUG postmaster:wait_for_user_backends_to_close: Waiting for user backends ...   <- ABSENT
   DEBUG postmaster:wait_for_user_backends_to_close: Backends closed                 <- ABSENT

4. Observe patroni.log:
   复制

   INFO  ha:_handle_dcs_error: demoting self because DCS is not accessible and I was a leader
   INFO  ha:demote: Demoting self (offline)
   DEBUG postmaster:wait_for_user_backends_to_close: Waiting for user backends ...   <- ABSENT
   DEBUG postmaster:wait_for_user_backends_to_close: Backends closed                 <- ABSENT

   Neither debug line appears, confirming the empty-user_backends path was taken. There is no further output — the demote never reports completion.
5. The watchdog stays active and the host hard-reboots ~25s later. last reboot shows a boot entry with no matching shutdown record.

What did you expect to happen?

no kernel reboot

Patroni/PostgreSQL/DCS version

• Patroni version: since 4.1.4
• PostgreSQL version: PostgreSQL 16
• DCS (and its version): Patroni 4.1.5

Patroni configuration file

NA

patronictl show-config

NA

Patroni log files

NA

PostgreSQL log files

NA

Have you tried to use GitHub issue search?

• Yes

Anything else we need to know?

No response