CDC decoder: translated tuples leak into ReorderBufferChange, causing SIGSEGV in SlabFree during reorder buffer cleanup
Summary
TranslateChangesIfSchemaChanged() overwrites change->data.tp.newtuple / oldtuple with tuples allocated by heap_form_tuple() in the current memory context, and never restores the original pointers. PostgreSQL later frees those fields itself in ReorderBufferFreeChange(), expecting memory it allocated from the reorder buffer's own tuple context. Freeing a foreign pointer through the slab allocator produces a garbage context pointer and segfaults the walsender.
This is a pre-existing latent bug, not a regression from any recent PR. It surfaces intermittently, so it can look like a flaky test.
Evidence
Observed in src/test/cdc/t/006_cdc_schema_change_and_move.pl (PG18), which combines DROP COLUMN, a force_logical shard move, and a subsequent UPDATE — exactly the path that triggers translation. The test aborts after two assertions when the publisher crashes; the visible CI symptom is only a poll_query_until catch-up timeout with "connection refused", because the server logs are not uploaded as artifacts.
Backtrace from a local reproduction with a core dump:
SlabFree
ReorderBufferFreeChange
ReorderBufferCleanupTXN
ReorderBufferProcessTXN
xact_decode
LogicalDecodingProcessRecord
XLogSendLogical
WalSndLoopServer log:
LOG: client backend (PID ...) was terminated by signal 11: Segmentation fault
DETAIL: Failed process was running: START_REPLICATION SLOT "cdc_replication_slot" LOGICAL 0/0
(proto_version '4', streaming 'parallel', origin 'any', publication_names '"cdc_publication"')
LOG: shutting down because "restart_after_crash" is offCore inspection identifies the freed object precisely:
ReorderBufferChangewith actionREORDER_BUFFER_CHANGE_UPDATE, XID 797- the fault occurs on the field at offset
0x38, confirmed via DWARF to bedata.tp.newtuple - the new tuple is still a valid
HeapTupleData(len 58), i.e. our translated tuple - its chunk header selects allocator ID 5, from which
SlabFreederives a bogus context pointer0x1000000and faults
So the crash is freeing the translated tuple, not the ReorderBufferChange container.
Code path
cdc_decoder.cGetTupleForTargetSchemaForCdc()builds a replacement tuple withheap_form_tuple()TranslateChangesIfSchemaChanged()assigns it tochange->data.tp.newtuple(andoldtuplefor UPDATE with REPLICA IDENTITY FULL, and for DELETE)TranslateAndPublishRelationForCDC()calls the pgoutput callback and returns without restoring the original pointers- PostgreSQL's
ReorderBufferProcessTXN()callsAbortCurrentTransaction()and thenReorderBufferCleanupTXN(), which frees both tuple fields
The lifetime mismatch means PostgreSQL frees memory Citus allocated, and Citus's tuples are additionally orphaned if the context is already gone. Intermittency comes from memory reuse, not from concurrency — it does not require a race to fail.
Fix direction
Restore the original tuple pointers on the ReorderBufferChange after publishing, and free the translated tuples ourselves, so each allocator only frees what it owns.
Notes
- Reproduced with the exact CI-built extension binaries; the previous passing CI artifact reproduces it too, and their
.textsections are byte-identical — confirming this is not introduced by recent packaging changes. .github/actions/save_logs_and_resultsdoes not uploadsrc/test/cdc/log/, so CDC TAP server logs are lost in CI. Worth fixing separately to make future crashes diagnosable.
Source: citusdata/citus