[BUG] Durable task eviction uses a stale retry count after worker-loss reassignment
Describe the issue
A durable task can stop progressing after worker-loss reassignment because eviction uses a cached retry count. The eviction SQL matches no active runtime row, but the dispatcher still sends an eviction acknowledgment. The worker then clears its local wait while the engine still considers the current attempt assigned to that worker.
Environment
- Self-hosted engine: reproduced on
v0.106.8. - TypeScript SDK:
1.31.1, with local patches for durable child references and desired-worker-label propagation. These SDK patches were unchanged between the failing and passing engine runs. - Source inspection: the same lookup, SQL predicate, and acknowledgment behavior remain in
v0.107.0and currentmainas of September 15, 2026. The runtime reproduction has not been rerun on those versions.
Expected behavior
Eviction should use the active attempt's retry count. The worker should not abandon an active wait on the basis of an eviction acknowledgment when that attempt remains assigned and was not evicted.
Code to Reproduce, Logs, or Screenshots
Reproduced in a local integration harness with separate coordinator and child-worker processes. A standalone reproducer has not yet been extracted.
- Run a durable coordinator that spawns a long-running child on a separate worker and uses eviction-enabled waits. Keep the dispatcher process running throughout so its task lookup cache survives.
- Send
SIGKILLto the coordinator worker while the child is running. - Start a replacement coordinator worker and allow Hatchet to reassign the coordinator. The child continues on its existing worker.
- Complete the child and let the replayed coordinator reach a later evictable wait.
- Observe the coordinator stop progressing after eviction is acknowledged.
In the stalled run, the current v1_task_runtime row had retry_count = 2, a non-null worker_id, and evicted_at = NULL. The SDK had removed the local callback before its completion arrived.
The source path explains the mismatch:
evictDurableTask()callsGetTaskByExternalId(..., false)and passes the returnedRetryCounttoEvictTask().GetTaskByExternalId()caches a record that includesRetryCount; a cached record can outlive reassignment to another retry.- The eviction SQL requires an exact retry-count match. An old retry therefore does not evict the active runtime row.
handleEvictInvocation()discards the eviction result and sends an acknowledgment whenever the call returns no error, including whenWasEvictedis false.
Changing the eviction lookup to GetTaskByExternalId(..., true) to bypass the cache made the same worker-kill regression complete successfully. The SDK patches and test scenario were held constant.
Additional context
This appears distinct from #4814, which handles eviction after all callbacks are satisfied, and #4918, which retries incomplete child creation. Both fixes are already in the tested engine version.
Besides fetching the current retry, could the eviction acknowledgment path distinguish an active attempt that was not evicted from valid no-op cases such as an already-evicted task or an obsolete invocation?
AI Disclosure
- I acknowledge that an LLM was used in the creation of this issue, in accordance with Hatchet's AI_POLICY.md.
- Details: OpenCode assisted with the local integration harness, failure diagnosis, engine patch, upstream source and issue searches, and this report.
Source: hatchet-dev/hatchet