Removed workers can roll back transactions belonging to the original coordinator
Problem
Removing a worker resets its local group ID to 0 (the coordinator identity), but can leave its old node-directory entries in pg_dist_node. If the worker remains running, its transaction-recovery process can still contact the old cluster and mistake the original coordinator's prepared transactions for its own.
A prepared transaction contains durable changes that are still waiting for a final commit or rollback decision.
Example
- Coordinator C durably records that a transaction on worker A must commit, but A is still waiting for the final commit instruction.
- An unrelated worker B is removed. Its group ID becomes
0, but it retains its route to A. - Recovery on B finds the prepared transaction on A and treats it as B's own. B has no matching commit-decision row in its local
pg_dist_transactiontable; that row belongs to C. B can therefore incorrectly tell A to roll back.
This can discard changes that the original coordinator already decided to commit, potentially leaving a distributed transaction only partially applied.
Expected behavior
Removing B must not let B interfere with transactions owned by C. A detached worker must stop using its old cluster routes, and recovery must not act on stale membership or a changing local identity. Detachment must preserve physical user tables.
CI context
Identified while investigating #8850. In the PG17 regression job, multi_colocated_shard_rebalance reported a missing prepared transaction during commit, followed by a missing shard table.
The mechanism above is a source-supported explanation, but the exact rollback actor in that CI run is not confirmed. Affected releases and production impact have not been established.
Source: citusdata/citus