v4 backfill (M2): the completion hook deadlocks — STOP MERGES blocks the queue that SYNC REPLICA STRICT waits on

What happens

20260701_v4_step_2_rewrite_observations_to_pid_tid_sorting completes every chunk and then hangs in its completion hook until something times out. The migration is recorded as failed, and clearing failedAt reproduces it — the step cannot be retried into success.

Why

onBackfillSucceeded freezes merges and then waits for the replication queue to empty:

// worker/src/backgroundMigrations/rewriteObservationsToPidTidSorting.ts:294
await this.stopMergesOnScratchTable();
await this.syncReplicasOnScratchTable();

buildSyncReplicaQuery emits SYSTEM SYNC REPLICA ... STRICT (utils/v4BackfillDdl.ts:107-110), and STRICT waits until the queue is completely empty. If a MERGE_PARTS entry is already queued when the freeze lands, it can never be executed, so the queue can never empty, so the sync never returns.

The freeze causes the condition it is waiting on.

From system.replication_queue on a production cluster, mid-hang:

type:            MERGE_PARTS
new_part_name:   202606_0_153_4
num_tries:       1
num_postponed:   153
last_exception:  (empty)
postpone_reason: Not executing log entry queue-0000010131 of type MERGE_PARTS
                 for part 202606_0_153_4 because merges and mutations are
                 cancelled now.

One entry, on one replica, postponed 153 times and counting. The other three hosts finished their per-host SYSTEM SYNC REPLICA in 0–0.1s.

Why retrying does not clear it

The freeze is partly persistent. buildFreezeReplicatedMergesQuery writes max_replicated_merges_in_queue = 0, always_fetch_merged_part = 1 into table metadata (utils/v4BackfillDdl.ts:99), which survives the worker process. An attempt that dies after freezing leaves the scratch table in exactly the state that prevents the next attempt from draining, so every retry hangs the same way.

Why it survives smaller installs

It needs the replication queue to be non-empty at the moment the freeze lands. On a scratch table that was just filled by the backfill, that window is wide; on a small one it is usually empty and the hook succeeds.

What it looks like from outside

Whatever timeout fires first, which made this hard to attribute. Observed in one deployment, in this order, as each limit was raised:

All three are the same hang. The failedReason recorded in Postgres is identical in the first and third case, so the ClickHouse query_log duration is the only way to tell them apart.

Patch

Sync first, then freeze, and make the freeze reversible so a failed attempt does not poison the next one:

await this.resumeMergesOnScratchTable(); // lift both halves of any prior freeze
await this.syncReplicasOnScratchTable(); // drain the queue, merges included
await this.stopMergesOnScratchTable();   // freeze the converged state

resumeMergesOnScratchTable issues SYSTEM START MERGES plus ALTER TABLE ... RESET SETTING max_replicated_merges_in_queue, always_fetch_merged_part (and the shared_merge_tree_disable_merges_and_mutations_assignment equivalent on SharedMergeTree). RESET SETTING rather than assigning defaults back, so the table returns to the server's default instead of a value hard-coded in the migration.

There is a window between the sync returning and the freeze landing in which a new merge could be scheduled. That seems acceptable because M3 re-verifies replica convergence before it reads system.parts, and verifyCompletedChunk checks the part is still active — but if the freeze-first order was chosen deliberately to close that window, an alternative would be SYSTEM SYNC REPLICA ... LIGHTWEIGHT, which waits for fetches rather than for merges.

Environment

• ClickHouse 25.8, self-hosted, two shards with two replicas each
• v4 historic backfill chain M1→M5