#44336·airflow

Different `max_active_runs` for scheduled vs manual runs

Author: hterikCreated Nov 25, 2024Updated Sep 16, 2026
Labelsarea:schedulerkind:feature

Description

Hi We have a use case where a DAG is scheduled similar to ContinuousTimetable, where it should run as often as possible, but it shouldn't be scheduled to run more than one at a time. The same DAG can be started manually, with a set of parameters to override the default behaviour. When someone decides to start manually, it is OK to have 2 or more of them running concurrently.

I can't find any way to implement this, setting max_active_runs=1, constrains also the manual runs.

I would like a new parameter (max_active_scheduled_runs?), or some new extension point in the TimeTable API, that can take all ongoing runs into consideration and use that to withhold scheduling new runs.

Use case/motivation

Some example use cases

  • The scheduled runs are running a complete test suite that takes many hours. Using Params, a user can select one specific test to run, that will only take minutes to run.
  • Shortening lead times. Imagine a run taking 1 hour. Half an hour in you realize you need to do a new run with some new data that became available. Instead of waiting for the current run to finish or aborting the current run, you can manually override to start a second run in parallel. Always having a half-hour cadence active is not desirable due to cost.

Related issues

No response

Are you willing to submit a PR?

  • Yes I am willing to submit a PR!

Code of Conduct