Connection of one in-process transaction being handed to a separate transaction, causing corruption
Version: [email protected] · Node 24 · PostgreSQL 16 (AWS Aurora)
Summary
We observed one connection being used by two sql.begin() transactions,
and that second transaction's COMMIT then committed the first transaction's partial, unfinished writes
What we run
One pool (max: 5) shared by several concurrent sql.begin() workloads:
- a long-lived insert transaction that pipelines a few thousand
INSERTs, then a single finalUPDATEmarker, all in onesql.begin; - a concurrent short purge transaction (a
DELETEwithinsql.begin); - occasional table scans done in transaction snapshots
Options: max_lifetime 300–600s (randomized), idle_timeout: 60, TLS on, over a network
where connections can half-close. (We can't say whether the pool was saturated at the moment
of the collision; we only know two specific transactions, the writer and the purge, ended up
sharing one connection.)
Evidence (from the PostgreSQL server log)
Backend 74671 was in the middle of an insert transaction:
19:00:28 [74671]:LOG: AUDIT: SESSION,1366,1,WRITE,INSERT,,,"INSERT INTO "
19:00:29 [74671]:LOG: AUDIT: SESSION,1367,1,WRITE,INSERT,,,"INSERT INTO "Then on that same backend:
19:00:29 [74671] WARNING: there is already a transaction in progress (BeginTransactionBlock)Corresponding exactly to server log lines indicating the purger was executing its transaction:
2026-08-29T19:00:29.52707903Z | INFO | Purging PG changes before 724at0ueo ...
2026-08-29T19:00:29.528707661Z | WARN | {"level":"WARN","pid":27,"worker":"change-streamer","workerIndex":0,"severity":"WARNING","code":"25001","message":"pg notice","routine":"BeginTransactionBlock"}
2026-08-29T19:00:29.616657299Z | INFO | Purged 6334 PG changes before 724at0ueo (89.49 ms)(The DELETE statements from the purger are wrapped in a CTE and thus is filtered from audit logs as "reads")
After that WARNING, backend 74671 logged nothing further except a clean disconnection:
19:01:30 [74671] LOG: disconnection: session time: 0:05:11.707 ... user=... database=...Note: a plain disconnection, not a FATAL, not an idle-in-transaction termination, not a rollback.
Database state
- The INSERTs were committed, but without the final UPDATE (of a different table) that the code ensures
- The purger's delete was also committed.
74671closed with a clean disconnection even though the tx of [1] had not been committed
If the backend disconnected while a transaction is still open, the inserter's rows
would have been rolled back. They weren't. So at disconnect time 74671 held
no open transaction: its server-side transaction had already been
committed. But the inserter never issued that COMMIT. The only other operation on
74671 was the purge sql.begin(), whose BEGIN is the 25001 above — its COMMIT
persisted both the purge's changes and the insert transaction's partial changes.
Reproduction
We were not able to reproduce this synthetically on [email protected] (a small pool
saturated with concurrent writer/purge/reader sql.begin() transactions, plus connection
recycling and backend-termination storms all held atomicity). It appears to require a
specific pool-state/timing (possibly Aurora- or half-close-specific) we couldn't force.
The server-log evidence above is from production.
Questions
- With #274 fixed in 3.4.7, are there remaining paths where a connection still inside one
sql.begin()transaction can be handed to a secondsql.begin(), so that the second'sBEGIN/COMMIToperate on the first's open transaction? - Is a
25001fromBeginTransactionBlockon a pooledsql.begin()ever expected, or is it always a connection-state desync? - Is this the same underlying cause as #827 (
sql.begin()hanging under similar load)?
Our mitigation
We're moving the writer loop and the purge onto separate postgres() client instances,
each with max: 1, so the two workloads live in different in-memory pools and this
connection-sharing cannot occur. Flagging the driver behavior in case it points at a
residual bug others hit.
Source: porsager/postgres