Improve max_active_runs visibility across logs, API, and UI
Today, when a Dag is at max_active_runs, there is very little signal anywhere that explains it, across three different surfaces:
- Logs: there is currently no log message anywhere for this. The scheduler's periodic bookkeeping (
_set_exceeds_max_active_runs) updatesDagModel.exceeds_max_non_backfillsilently, with no log output at all. #72387 tracks adding a log (with blocking run/task IDs) for the moment a Dag newly becomes blocked; the PR attempting that, #72403, was closed without merging. Either way, that would only cover a Dag newly becoming blocked — a Dag Run created after the Dag is already at capacity (e.g. a second manual trigger while the first run is still active) produces no log output anywhere explaining why it never starts. - API:
DagModel.exceeds_max_non_backfill(added in 3.2.0 by migration0099_3_2_0_add_exceeds_max_runs_flag_to_dag_model) is persisted to thedagtable but is never exposed onDAGResponse(GET /dags/{dag_id}) or anywhere else in the public API. There is no way for a client to ask "is this Dag currently blocked on max_active_runs" without independently computing active-run counts. - UI: the Dag header's "Active Runs" stat (
airflow-core/src/airflow/ui/src/pages/Dag/Header.tsx) showsX of Yonce a Dag has active runs, but nothing explains what happens to a run that's queued up behind the limit.
Related: #72387 tracks adding blocking run/task IDs to the "Dag newly became blocked" bookkeeping, once a log exists there at all. That would be complementary to, not overlapping with, the gaps above (different trigger point, different code path).
Historical note: the original motivating example for this issue was seeing X of Y with X > Y (e.g. "2 of 1") in the 3.2.2 UI, which reads as an outright illogical number. On current main that specific presentation can no longer happen — active_runs_count (the X) is RUNNING-only and capped by the scheduler's promotion gate — thanks to two already-shipped, independent upstream fixes: #69769 (narrowed the count to RUNNING-only; previously counted RUNNING+QUEUED) and #72371 (relabeled the stat from "Max Active Runs" to "Active Runs" to match). Older Airflow/Astro Runtime builds cut before both of those will still show the old, confusing behavior. The underlying problem this issue tracks is still real on main — a Dag can have runs genuinely stuck queued behind the limit with only an inferred indication in the UI (X = Y) — it just no longer surfaces as a strange-looking number.
Follow-up work
One PR per item, so each can be reviewed independently:
- Log when a newly-created Dag Run will not be scheduled yet because the Dag is already at
max_active_runs(covers the manual-trigger-while-blocked case that #72387 doesn't). → #73689 (merged/open) - Expose active-run-related fields on
DAGDetailsResponse:is_at_max_active_runs(aliased toexceeds_max_non_backfill) andqueued_runs_count(live-computed, same pattern as the existingactive_runs_count). Also fixes two real bugs found while building this: the dag-processor's active-run-count calculation hardcoded0for any Dag whose timetable can't be scheduled (e.g.schedule=None, triggered only manually/via the API), which madeexceeds_max_non_backfillpermanently unreliable for exactly the Dags that most need it; and a pre-existing N+1 query in the same code path (oneactive_runs_of_dagsquery per Dag per parse cycle instead of one batched query per parse). → #73692 - Add an info icon + tooltip next to the "Active Runs" stat in the Dag header UI, shown whenever
queued_runs_count > 0, and display the queued count inline (1 of 1 (2 queued)) so the information doesn't require a hover. Keyed off the livequeued_runs_countrather than the periodically-recomputedis_at_max_active_runsflag, so the UI's correctness doesn't depend on that flag's refresh cadence. Went with an info icon rather than a warning, since (per the historical note above) the number itself never looks alarming onmain— the queued state is invisible, not alarming-looking, so "info" fits better than "warning". → #73693 (builds on #73692)
Drafted-by: Claude Sonnet 5 (reviewed by @seanmuth)
Source: apache/airflow