#4989·hatchet

[BUG] Batched application failures can use another task's retry budget

Author: tolivCreated Sep 20, 2026Updated Sep 20, 2026

Describe the issue

FailTaskAppFailure can retry a task after its own application retry budget is exhausted when another task in the same failure batch has a larger budget. A task configured with retries=0 can therefore run again.

The query updates v1_task FROM tasks_to_steps without joining the target task to its matching row. The comparison tasks_to_steps.retries > v1_task.app_retry_count can use another task's budget. The missing join also allows a task excluded from locked_tasks because it has no active runtime to be updated.

Environment

  • Observed with self-hosted engine v0.106.3 and Python SDK 1.35.1; the persisted Step had retries=0, while its task reached app_retry_count=1 and internal_retry_count=0.
  • Deterministically reproduced through the repository query on upstream main at 929f9d2c5d0fc65640efce885c33bde76809baa9, using PostgreSQL 15.6 and Go 1.26.5.
  • The same query is present in v0.107.4. This is separate from the SDK default-resolution issue #4727.

Expected behavior

Each active attempt should use its own Step's budget; non-retryable failures and attempts without a runtime should not be retried.

Reproduction

Create tasks with the following stored state, then submit all five to one FailTaskAppFailure call, using their current IDs, insertion timestamps and retry counts:

Task Step retries App retries used Internal retries used Active runtime Non-retryable Expected retry Actual retry
A 0 0 0 yes no no yes
B 1 1 0 yes no no yes
C 2 1 2 yes no yes yes
D 3 0 0 yes yes no no
E 3 0 0 no no no yes

Proposed fix

Carry inserted_at through tasks_to_steps and join it to the update target on (id, inserted_at). A repository regression test fails before this change and passes afterward; the existing queued-task replay test also passes.

I would like to contribute the fix and regression test. Could a maintainer triage this and assign it to toliv if accepted?

AI Disclosure
  • An LLM was used in preparing this issue, in accordance with Hatchet's AI_POLICY.md.
  • Details: OpenAI Codex investigated the query, searched for existing reports, drafted this issue, and prepared and ran the fix and regression test.