#8844·citus

CDC decoder: translated tuples leak into ReorderBufferChange, causing SIGSEGV in SlabFree during reorder buffer cleanup

Author: ihalatciCreated Sep 10, 2026Updated Sep 10, 2026
Labelsbug

Summary

TranslateChangesIfSchemaChanged() overwrites change->data.tp.newtuple / oldtuple with tuples allocated by heap_form_tuple() in the current memory context, and never restores the original pointers. PostgreSQL later frees those fields itself in ReorderBufferFreeChange(), expecting memory it allocated from the reorder buffer's own tuple context. Freeing a foreign pointer through the slab allocator produces a garbage context pointer and segfaults the walsender.

This is a pre-existing latent bug, not a regression from any recent PR. It surfaces intermittently, so it can look like a flaky test.

Evidence

Observed in src/test/cdc/t/006_cdc_schema_change_and_move.pl (PG18), which combines DROP COLUMN, a force_logical shard move, and a subsequent UPDATE — exactly the path that triggers translation. The test aborts after two assertions when the publisher crashes; the visible CI symptom is only a poll_query_until catch-up timeout with "connection refused", because the server logs are not uploaded as artifacts.

Backtrace from a local reproduction with a core dump:

SlabFree
ReorderBufferFreeChange
ReorderBufferCleanupTXN
ReorderBufferProcessTXN
xact_decode
LogicalDecodingProcessRecord
XLogSendLogical
WalSndLoop

Server log:

LOG:  client backend (PID ...) was terminated by signal 11: Segmentation fault
DETAIL:  Failed process was running: START_REPLICATION SLOT "cdc_replication_slot" LOGICAL 0/0
         (proto_version '4', streaming 'parallel', origin 'any', publication_names '"cdc_publication"')
LOG:  shutting down because "restart_after_crash" is off

Core inspection identifies the freed object precisely:

  • ReorderBufferChange with action REORDER_BUFFER_CHANGE_UPDATE, XID 797
  • the fault occurs on the field at offset 0x38, confirmed via DWARF to be data.tp.newtuple
  • the new tuple is still a valid HeapTupleData (len 58), i.e. our translated tuple
  • its chunk header selects allocator ID 5, from which SlabFree derives a bogus context pointer 0x1000000 and faults

So the crash is freeing the translated tuple, not the ReorderBufferChange container.

Code path

  • cdc_decoder.c GetTupleForTargetSchemaForCdc() builds a replacement tuple with heap_form_tuple()
  • TranslateChangesIfSchemaChanged() assigns it to change->data.tp.newtuple (and oldtuple for UPDATE with REPLICA IDENTITY FULL, and for DELETE)
  • TranslateAndPublishRelationForCDC() calls the pgoutput callback and returns without restoring the original pointers
  • PostgreSQL's ReorderBufferProcessTXN() calls AbortCurrentTransaction() and then ReorderBufferCleanupTXN(), which frees both tuple fields

The lifetime mismatch means PostgreSQL frees memory Citus allocated, and Citus's tuples are additionally orphaned if the context is already gone. Intermittency comes from memory reuse, not from concurrency — it does not require a race to fail.

Fix direction

Restore the original tuple pointers on the ReorderBufferChange after publishing, and free the translated tuples ourselves, so each allocator only frees what it owns.

Notes

  • Reproduced with the exact CI-built extension binaries; the previous passing CI artifact reproduces it too, and their .text sections are byte-identical — confirming this is not introduced by recent packaging changes.
  • .github/actions/save_logs_and_results does not upload src/test/cdc/log/, so CDC TAP server logs are lost in CI. Worth fixing separately to make future crashes diagnosable.