[BUG] Batched application failures can use another task's retry budget
Describe the issue
FailTaskAppFailure can retry a task after its own application retry budget is exhausted when another task in the same failure batch has a larger budget. A task configured with retries=0 can therefore run again.
The query updates v1_task FROM tasks_to_steps without joining the target task to its matching row. The comparison tasks_to_steps.retries > v1_task.app_retry_count can use another task's budget. The missing join also allows a task excluded from locked_tasks because it has no active runtime to be updated.
Environment
- Observed with self-hosted engine v0.106.3 and Python SDK 1.35.1; the persisted Step had
retries=0, while its task reachedapp_retry_count=1andinternal_retry_count=0. - Deterministically reproduced through the repository query on upstream
mainat929f9d2c5d0fc65640efce885c33bde76809baa9, using PostgreSQL 15.6 and Go 1.26.5. - The same query is present in v0.107.4. This is separate from the SDK default-resolution issue #4727.
Expected behavior
Each active attempt should use its own Step's budget; non-retryable failures and attempts without a runtime should not be retried.
Reproduction
Create tasks with the following stored state, then submit all five to one FailTaskAppFailure call, using their current IDs, insertion timestamps and retry counts:
| Task | Step retries | App retries used | Internal retries used | Active runtime | Non-retryable | Expected retry | Actual retry |
|---|---|---|---|---|---|---|---|
| A | 0 | 0 | 0 | yes | no | no | yes |
| B | 1 | 1 | 0 | yes | no | no | yes |
| C | 2 | 1 | 2 | yes | no | yes | yes |
| D | 3 | 0 | 0 | yes | yes | no | no |
| E | 3 | 0 | 0 | no | no | no | yes |
Proposed fix
Carry inserted_at through tasks_to_steps and join it to the update target on (id, inserted_at). A repository regression test fails before this change and passes afterward; the existing queued-task replay test also passes.
I would like to contribute the fix and regression test. Could a maintainer triage this and assign it to toliv if accepted?
- An LLM was used in preparing this issue, in accordance with Hatchet's AI_POLICY.md.
- Details: OpenAI Codex investigated the query, searched for existing reports, drafted this issue, and prepared and ran the fix and regression test.
Source: hatchet-dev/hatchet