orca: `StartExecution` retry can put a completed pipeline back into `RUNNING`

Affected version: Spinnaker 2026.2.0 / Orca 2026.2.0

What happened

We hit a race in Orca when a StartExecution handler took longer than the Redis queue's 60-second acknowledgement timeout. Redis delivered the same message to another worker, and both workers read the execution while it was still NOT_STARTED.

The first worker started the pipeline and the pipeline finished successfully. The second worker then saved the older copy it had already loaded, changing the top-level execution status from SUCCEEDED back to RUNNING.

We saw this happen to 39 pipelines on Orca 2026.2.0. Every leaf stage was SUCCEEDED or SKIPPED, but the pipeline itself remained RUNNING.

How the race happens

Redis queue         Worker A                SQL store             Worker B
    |                   |                       |                     |
    |-- StartExecution->|                       |                     |
    |                   |-- read state -------->|                     |
    |                   |<-- NOT_STARTED -------|                     |

    |                   |-- admission SQL query>|                     |
    |                   |   waiting for SQL result...                 |

    |-- retry same StartExecution after 60s ------------------------->|
    |                   |                       |<-- read state -------|
    |                   |                       |--- NOT_STARTED ----->|
    |                   |<-- admission result --|                     |

    |                   |-- write RUNNING ----->|                     |
    |                   |-- queue StartStage; pipeline completes      |
    |                   |-- write SUCCEEDED --->|                     |
    |                   |                       |<-- stale RUNNING ----|
    |<-- duplicate StartStage ---------------------------------------|
    |-- duplicate is ignored because the stage is already SUCCEEDED

The duplicate stage start is harmless because StartStageHandler notices that the stage has already succeeded. The stale execution write is the problem: it changes a terminal pipeline back to RUNNING.

What we saw in the logs

The same pattern appeared in all 39 pipelines:

• The final two StartExecution deliveries were 60.175 to 65.406 seconds apart.
• Both workers loaded the execution as NOT_STARTED.
• The first worker wrote RUNNING 61.668 to 66.393 seconds after receiving the message.
• The pipeline finished before the retrying worker wrote its copy.
• The retry wrote stale RUNNING state 4.912 to 40.267 seconds after the pipeline finished.
• StartStageHandler ignored the duplicate stage start because the stage had already succeeded.

The first RUNNING write and the later stale write came from different Orca pods in every case.

What was slow

The slow part was the SQL-backed concurrency check used by shouldQueue(). retrievePipelinesForPipelineConfigId took about 66 to 69 seconds during the two bursts we checked. It loads and materializes up to maxConcurrentExecutions + 2 complete executions and their stages, even though shouldQueue() only needs to know how many are active.

An admission query should generally never take this long. The elevated SQL latency is a performance problem in its own right, but it also exposed a separate correctness bug: a duplicate StartExecution delivery can overwrite newer execution state. Fixing the SQL latency would make the race less likely, but duplicate delivery is normal queue behavior and should not be able to move a completed pipeline back to RUNNING. That race is worth fixing independently.

The metrics from those bursts were:

• sql_executions_retrievePipelinesForPipelineConfigId2_timing_seconds_max: 65.798 to 68.738 seconds
• sql_pool_default_connectionUsageTiming_seconds_max: 61.780 to 65.865 seconds
• sql_pool_default_connectionAcquiredTiming_seconds_max: 27 to 57 milliseconds
• sql_executions_store1_timing_seconds_max: 0.584 to 1.947 seconds
• sql_pool_default_blocked: zero on every pod except for a maximum of two callers on one pod during one burst

This points to the admission read path rather than waiting for a pool connection or writing the execution afterward. We cannot tell from the current metrics how much of that time was database execution and result transfer versus Orca decompressing, deserializing, and loading stages.

Why the retry can overwrite completed state

Four pieces of behavior make the race possible:

1. StartExecution uses the Redis queue's default one-minute acknowledgement timeout.
2. QueueProcessor acknowledges the message only after StartExecutionHandler returns.
3. Each handler keeps the execution copy it loaded while it performs the admission check.
4. The SQL save does not check whether the stored status or execution version changed in the meantime.

A duplicate delivery should be safe. Once an execution is terminal, a retry must not move it back to RUNNING or enqueue its first stages again.

Suggested fix

Make the NOT_STARTED to RUNNING update conditional on the stored execution still being in the state or version that the worker originally read. If another worker has already changed it, treat the message as an already-handled duplicate and do not enqueue the first stages.

A regression test could run two handlers like this:

1. Both handlers load the same NOT_STARTED execution.
2. The first handler starts and completes it.
3. The second handler tries to save its stale copy and is prevented from changing the terminal status.

Increasing the acknowledgement timeout would make this race less likely, but it would not make a duplicate delivery safe. The conditional state update is the important part.

Separately, the admission check could use a count or lightweight projection instead of loading complete execution bodies and stages.

Relevant code paths

• messages.kt: StartExecution acknowledgement timeout
• RedisQueue.kt: default acknowledgement timeout
• QueueProcessor.kt: acknowledgement after handler completion
• StartExecutionHandler.kt: execution load, RUNNING transition, save, and first-stage enqueue
• OrcaMessageHandler.shouldQueue(): concurrency-admission check
• SqlExecutionRepository.kt: execution retrieval and update

Related issues

• https://github.com/spinnaker/spinnaker/issues/6443
• https://github.com/spinnaker/spinnaker/issues/6373

Those issues cover related zombie recovery and queueing behavior, but neither appears to describe this stale StartExecution overwrite race.