DYNAMIC_FORK: one SUB_WORKFLOW branch never created, force-TIMED_OUT after 60 min (regression in 3.32.0 from #1369)

Report (from a customer in the Conductor OSS community Slack)

Hi guys, we have DYNAMIC_FORK Task, which usually runs 80 SUB_WORKFLOW Tasks, but today conductor server haven't created sub-workflow for one of them and changed task and worfklow status to Timeout after 60 mins. What has happend? Tell me if you need more information about this incedent. We use the last version 3.32.0 of conductor server and we have never faced with this in previous one 3.30.2

Cause

Regression in 3.32.0 from #1369 ("reserve an async system task's queue message while it runs", fixing #1321). The new redelivery logic in AsyncSystemTaskExecutor assumes a blocking start() never leaves a task SCHEDULED (comment at AsyncSystemTaskExecutor.java:159), so on redelivery it force-times-out a still-SCHEDULED task instead of re-running it (line 164).

That assumption is false for SUB_WORKFLOW: SubWorkflow.start() deliberately leaves the task SCHEDULED on a transient error (SubWorkflow.java:206-207). Combined with the new 1-hour message reserve (default responseTimeoutSeconds = TaskDef.ONE_HOUR = 3600s), a transient hiccup during child creation now becomes an unrecoverable 60-minute timeout instead of a fast, self-healing retry.

Sequence for the one failed branch out of 80:

1. Task picked up; startTime persisted, message reserved for 1 hour.
2. Child creation hits a transient error (plausible when 80 children are minted at once) or the worker restarts mid-launch -- task stays SCHEDULED.
3. Message stays invisible for the full hour (on 3.30.2 it would redeliver in seconds and retry).
4. On redelivery, hasExceededResponseTimeout is true, so the task is set TIMED_OUT instead of re-running start() -- the child is never created and the JOIN never completes, so the workflow times out.

This matches every detail: 1 of 80, sub-workflow never created, TIMED_OUT, and exactly 60 minutes (ONE_HOUR doubles as both the reserve window and the timeout threshold). Not fixed in 3.32.1-3.32.3.

Confirming from logs

Look for Timing out SUB_WORKFLOW/... no response within responseTimeout (emitted only by this path), a node restart/GC/load spike ~60 min before the timeout, and a ~1-hour gap between the task's startTime and endTime.

Proposed direction (for discussion)

Keep the #1321 reserve, but on redelivery of a still-SCHEDULED SUB_WORKFLOW re-run start() (it's idempotent via the deterministic child id) rather than blanket-timing-out. Flagging for maintainer input first, since the reserve was added deliberately.

Environment

Conductor server 3.32.0 (regression from 3.30.2, still present through 3.32.3); DYNAMIC_FORK fanning out to ~80 SUB_WORKFLOW tasks.