[Bug]: the primary isolation check does not stop an isolated primary from accepting writes

Is there an existing issue already for this bug?

• I have searched for an existing issue, and could not find anything. I believe this is a new bug.

I have read the troubleshooting guide

• I have read the troubleshooting guide and I think this is a new bug.

I am running a supported version of CloudNativePG

• I have read the troubleshooting guide and I think this is a new bug.

Version

1.30.0

What version of Kubernetes are you using?

1.34

What is your Kubernetes environment?

Self-managed: k3s

How did you install the operator?

YAML manifest

What happened?

The version and environment fields above describe the third-party reproduction this report is based on, not a deployment of mine. This comes from reading the code after that reproduction was published, and I am filing it because the mechanism checks out against main and against every supported release.

The primary isolation check is documented as fencing an isolated primary and stopping it from accepting writes. It does not do that. It only returns HTTP 500 from the liveness probe (pkg/management/postgres/webserver/probes/liveness.go). The kubelet then signals the container, and the instance manager handles that signal by calling TryShuttingDownSmartFast (internal/cmd/manager/instance/run/lifecycle/lifecycle.go), which issues pg_ctl stop -m smart -w -t 180 first (pkg/management/postgres/instance.go). A PostgreSQL smart shutdown refuses new connections but lets every session that is already open run to completion, so sessions established before the partition keep committing on the isolated primary for up to smartShutdownTimeout, which defaults to 180 seconds.

This is not an argument for changing the shutdown mode generally. Smart shutdown is the right behaviour for the other paths that reach this code, a rolling update, a pod deletion, a node drain, where not killing user sessions is the entire point. The problem is that the isolation path, whose purpose is to stop writes as quickly as possible, inherits a shutdown routine designed for planned termination.

The documentation does not describe this. docs/src/failover.md says "The isolation check stops an isolated primary from continuing to accept writes", and docs/src/instance_manager.md says the isolation check "fences" a primary and that the effect is to "shut it down" when the liveness probe fails. On an asynchronous cluster none of that holds for the sessions that are already connected: they go on committing, and go on being acknowledged, for the length of the smart shutdown.

Evidence

Coroot published a measured reproduction at https://coroot.com/blog/reproducing-split-brain-on-cloudnativepg/, on 1.30.0 with asynchronous replication, and their numbers match the mechanism. They applied a network partition at 18:11:20, the isolation check fired 34 seconds in ("18:11:54 Killing Container postgres failed liveness probe"), and the isolated primary acknowledged its last write at 18:14:55, t+215s. 215 minus 34 is 181, against the 180 second smartShutdownTimeout. Their replacement primary took its first write at 18:13:17, so 99 seconds passed with both primaries acknowledging commits. Of the writes acknowledged after the partition they report 291 rows that pg_rewind later deleted and 562 for which, in their words, "that id now belongs to another client".

On synchronous replication

Synchronous replication closes this, and it is the project's standing advice for not losing commits: it is what the sample cluster and the quickstart configure on main, and #11148 warns when a multi-instance cluster has none. With it configured the isolated primary blocks on COMMIT and never acknowledges, and the blocked transactions are not visible to any other session either, since the transaction stays in the procarray for the whole wait.

This report is therefore about asynchronous clusters, which is what was measured above and what every released version still defaults to.

Suggested direction

Have the isolation check request a fast shutdown rather than only failing the probe, so that stopping the primary does not depend on the kubelet's timing or on a shutdown routine shared with planned termination.

Most of the machinery for that already exists: Instance.RequestFastImmediateShutdown() in pkg/management/postgres/instance.go puts the request on the instance command channel. Note that it is not unused, reconcileOldPrimary already demotes a former primary through it, so the isolation path needs a budget of its own rather than the one that path relies on. Its interaction with the lifecycle loop needs attention too, since an instance stopped for isolation must not be restarted in place.

This only collapses the second of two windows. From the partition until the liveness probe gives up, roughly 30 seconds with the default probe settings, the primary is fully open. After that, and this is the part the fix addresses, only the sessions already established on the isolated side keep writing, for up to smartShutdownTimeout.

Scope

The isolation check landed in 1.27.0, so this affects every supported release as well as main. The primary lease added in 1.30.0 neither helps nor is meant to: an isolated instance manager cannot reach the API server, so it classifies its lease read as unverifiable and keeps running, which docs/src/failover.md already documents.

Until this is fixed, spec.smartShutdownTimeout: 0 skips straight to a fast shutdown and collapses the second window to the probe detection latency.

Cluster resource

$ kubectl get cluster postgres-splitbrain -o json | jq '.spec | {failoverDelay, smartShutdownTimeout, probes}'
{
  "failoverDelay": 0,
  "smartShutdownTimeout": 180,
  "probes": {
    "liveness": {
      "isolationCheck": {"connectionTimeout": 1000, "enabled": true, "requestTimeout": 1000}
    }
  }
}

(Coroot's cluster as they published it; the rest was default.)

Relevant log output

# partition applied at 18:11:20
18:11:54  Killing         Container postgres failed liveness probe
18:11:57  FailingOver     Current primary isn't healthy, initiating a failover
18:11:58  FailoverTarget  Failing over from postgres-splitbrain-1 to postgres-splitbrain-2

# new primary takes its first write at 18:13:17
# isolated primary acknowledges its last write at 18:14:55, t+215s

Code of Conduct

• I agree to follow this project's Code of Conduct