#31395·yugabyte-db

[XCluster] DDL Replication - Snapshot-too-old error on retry of partially-processed batch

Author: hulien22Created May 1, 2026Updated Sep 18, 2026
Labelskind/enhancementarea/docdbxClusterpriority/highest

Jira Link: DB-21229 XClusterDDLQueueHandler::ExecuteCommittedDDLs publishes safe time per-commit_time as it iterates a batch. If processing fails partway through, the batch is persisted with all commit_times still present. After enough time elapses for compaction to advance to the published safe time, every retry reads at the front of commit_times and hits kSnapshotTooOld on commit_times that have already been processed.

sample repro:

  1. Source generates a batch of DDLs producing commit_times [c1, c2, c3, c4, c5].
  2. One of the later DDLs fails reliably (suppose at c3)
  3. ExecuteCommittedDDLs processes c1, c2 successfully → update_safe_time_func_(c2) publishes safe time = c2 → fails at c3 → returns error → replicated_ddls (1,1) still retains [c1, c2, c3, c4, c5]
  4. Wait for timestamp_history_retention_interval_sec. Namespace xCluster safe time advances to ~c2, HistoryCutoffOperation propagates, compaction discards history older than c2.
  5. Next retry of ExecuteCommittedDDLs iterates from c1 → GetRowsToProcess(c1) issues SET yb_read_time TO 'c1 ht'; SELECT … FROM ddl_queue q WHERE NOT EXISTS (SELECT 1 FROM replicated_ddls …) → returns kSnapshotTooOld.

eg error: W0501 21:41:17.836359 2201727 xcluster_poller.cc:343] P [8fde64f80c5e0897ba4adbe87c92ec4f:e4f59ffc9626467abe07d9cc63df82ca] C [0000490000003000800000000000d22c:6d7903118872429389dd349c8266420d]: Failed to process existing DDL queue: Network error (yb/yql/pgwrapper/libpq_utils.cc:482): Fetch '/*+ MergeJoin(q r) */ SELECT q.ddl_end_time, q.query_id, q.yb_data FROM yb_xcluster_ddl_replication.ddl_queue AS q WHERE NOT EXISTS ( SELECT 1 FROM yb_xcluster_ddl_replication.replicated_ddls AS r WHERE r.ddl_end_time = q.ddl_end_time AND r.query_id = q.query_id ) ORDER BY q.ddl_end_time ASC;' failed: 7, message: ERROR: Snapshot too old. Read point: { physical: 1777530246950278 }, earliest read time allowed: { physical: 1777556098975454 }, delta (usec): 25852.025s: kSnapshotTooOld (pgsql error 72000) (aux msg ERROR: Snapshot too old. Read point: { physical: 1777530246950278 }, earliest read time allowed: { physical: 1777556098975454 }, delta (usec): 25852.025s: kSnapshotTooOld)

Proposed fix: We should skip commit_times that are already processed so that we never re-read at a commit_time whose history could have been GC'd. Eg in the example above, we would skip reading at c1 or c2 since we've already processed those and bumped the safe time past those times. (skip commit times ≤ the currently published ddl_queue safe time read back from the safe time table)