[Bug][Zeta] After master takeover, CDC job with idle-finished source subtasks never triggers checkpoints again (alreadyStarted=false)
Search before asking
- I searched the issues and found no similar issues. (#12139 is a different restore hang.)
What happened
With two master nodes, when the active master dies and the standby takes over, a running MySQL-CDC job with parallelism > 1 is restored as RUNNING but never triggers another checkpoint. Source and sink tasks keep processing (SourceReceivedCount / SinkWriteCount grow), but SinkCommittedCount stops because the 2PC sink only commits on checkpoint completion. The job silently freezes until some task fails and the pipeline is restarted.
Root cause
On master active switch SubPlan restores the task execution states from the IMap and reports
jobMaster.getCheckpointManager().reportedPipelineRunning(pipelineId, allTaskRunning.get());where allTaskRunning is true only if every coordinator and physical vertex is RUNNING.
For MySQL-CDC with parallelism 2, after the snapshot phase only one reader keeps consuming the binlog and the other reader becomes idle and FINISHED (log on the new master: SourceTask (2/2) ... is in state FINISHED when init state future). So alreadyStarted = false, CheckpointCoordinator#restoreCoordinator(false) sets isAllTaskReady = false and waits for all tasks to report READY_START. The running tasks are not redeployed, so they never report READY_START again, allTaskReady() never passes and scheduleTriggerPendingCheckpoint is never called.
Control experiment: the same job with parallelism = 1 (no idle-finished subtask) is restored with alreadyStarted: true and checkpoints continue immediately after the takeover.
How to reproduce
- Separated mode with 2 masters + 3 workers, checkpoint storage + IMap persistence enabled.
- Submit MySQL-CDC -> Doris (2PC) streaming job with
parallelism = 2, wait until the snapshot is done (one source subtask becomes FINISHED). kill -9the active master.- Standby logs
This node become a new active master node,begin restore job, thenreceived restore CheckpointCoordinator with alreadyStarted: false, and no furtherwait checkpoint id: N completedlines. The job stays RUNNING,SinkCommittedCountstays constant.
Suggested fix (for discussion)
Treat subtasks that were closed as idle (tracked in readyToCloseIdleTask / FINISHED after the reader signalled no more splits) as "started" when computing allTaskRunning, or let restoreCoordinator(false) re-request READY_START from tasks that are still running. Happy to work on a PR once the direction is agreed.
SeaTunnel Version
dev (c7304ace6, 2026-09-17); also reproduced with the fix for #12385 applied (state is then loaded correctly, but checkpointing still does not resume).
SeaTunnel Config
env { job.mode = "STREAMING", parallelism = 2, checkpoint.interval = 10000 }
source { MySQL-CDC { ... } }
sink { Doris { sink.enable-2pc = "true", ... } }Running Command
bin/seatunnel.sh --config cdc.conf --asyncError Exception
received restore CheckpointCoordinator with alreadyStarted: false
(no checkpoint triggered afterwards)Zeta or Flink or Spark Version
Zeta, separated cluster mode, 2 masters
Java or Scala Version
JDK 8 (1.8.0)
Screenshots
No response
Are you willing to submit PR?
- Yes I am willing to submit a PR!
Code of Conduct
- I agree to follow this project's Code of Conduct
Source: apache/seatunnel