[BUG] DAG runs with on_failure handlers can never reach a terminal status when cancelled before the on_failure task spawns

Describe the issue

A DAG run whose workflow defines an on_failure task can never reach a terminal display status if it is cancelled before the on_failure task spawns. The run shows RUNNING forever in the dashboard/API even though every task that exists for it is CANCELLED. Bulk-cancelling a backlog produces these at scale: in our production incident, 24,482 runs were stuck displaying RUNNING with 100% of their underlying tasks CANCELLED.

Root cause (two parts, both verified in source at v0.94.10 and unchanged at v0.97.0):

1. total_tasks counts ON_FAILURE steps. The trigger path enumerates every step of the workflow version with no job-kind filter (ListStepsByWorkflowVersionIds joins all jobs), and total_tasks is set to the full step count (trigger.go — TotalTasks: len(opt.TaskIds)). But an ON_FAILURE step only creates a task when another step fails — on the success path it materializes late as SKIPPED, and on the cancel path it never materializes at all.
2. The DAG status rollup checks task_count != total_tasks → 'RUNNING' before any terminal check (olap.sql#L1065, olap.sql#L1217).

For a 1-step workflow with an on_failure handler: total_tasks = 2, the cancelled run has task_count = 1, and 1 != 2 returns RUNNING on every recompute, forever. Nothing can ever fix such a run — cancelling again doesn't transition it (the tasks are already terminal), and only partition retention removes the row.

Environment

• SDK: Python v1.20+
• Engine: Self-hosted v0.94.10 (AKS, Postgres message queue). Code path verified byte-identical through v0.97.0.

Expected behavior

A cancelled DAG run whose spawned tasks are all terminal should display CANCELLED.

Code to Reproduce, Logs, or Screenshots

1. Define any workflow with one task plus an on_failure task, with a concurrency limit.
2. Enqueue runs past the concurrency cap so some sit QUEUED.
3. Bulk-cancel the queued runs.
4. The cancelled runs display RUNNING permanently; v1_dags_olap.total_tasks = 2 while only 1 task row exists.

Confirming query — every stuck run is "hollow" (all existing tasks terminal):

SELECT count(*) FROM v1_dags_olap d
WHERE d.readable_status = 'RUNNING'
  AND NOT EXISTS (
    SELECT 1 FROM v1_tasks_olap t
    WHERE t.workflow_run_id = d.external_id
      AND t.readable_status NOT IN ('CANCELLED','COMPLETED','FAILED')
  );

Additional context

This has been reported at least three times before and each report dead-ended:

• #1996 (v0.67.8, same shape: onFailureJob, on-failure skipped, run stuck RUNNING) — closed by the stale-bot.
• #2573 (stuck RUNNING, cancel updates finished_at but not status) — closed not-planned.
• PR #3276 identified this exact root cause with the same file references and included a fix + tests, but was closed by its author after he attributed his symptoms to a proxy misconfiguration. The != comparison it targeted is still live at v0.97.0.

A staged fix is available on my fork: compare view. It counts only unconditionally-created steps in total_tasks (leaving TaskExternalIDs and match-condition wiring untouched, unlike #3276) and changes the rollup comparison to strict less-than so the failure path (where the spawned on_failure task pushes task_count above total_tasks) doesn't hit the same dead-end. I walked all rollup CASE paths (success with SKIPPED→COMPLETED materialization, failure, cancel, mid-creation, eviction) — no regressions. Happy to open the PR if this issue is accepted.

• I acknowledge that an LLM was used in the creation of this Issue, in accordance with Hatchet's AI_POLICY.md.

• Details: Claude Code was used extensively for the source-level root-cause analysis, the staged patch, and drafting this issue, directed by a human operator investigating a production incident (the incident data, verification against our deployment, and remediation are ours).