Performance Bottleneck in Airflow Lineage Backend with Large DAG Histories

Affected module Airflow Lineage Backend

Describe the bug The Airflow Lineage Backend attempts to send lineage information for the full DAG regardless of whether the current task has lineage metadata configured. That happens even if the DAG has no inlets/outlets configured at all. Besides that all task also try to get all task instances without filter, that has proven a huge no go for our production enviroment because of massive delays (more than 10 minutes) for each task in the DAG. Even setting the max_status to 0, it still try to get all task instances.

To Reproduce Enable airflow to send lineage information to Openmetadata in a Airflow cluster with DAGs with big number of DAG runs and task instances.

Expected behavior

• Allow disabling status emission entirely when max_status=0
• Skip status and lineage generation for tasks without lineage (inlets or outlets)
• Scope task instance retrieval by execution date (e.g., last N days)

Version:

• Airflow: 2.10.5
• OpenMetadata version: 1.8.0
• OpenMetadata Ingestion package version: 1.8.0

Additional context Reading the code i found that the delay was between the logs "DAG has not changed since last run" and "Processing XLet data". That points directly to: self.add_all_pipeline_status(pipeline). https://github.com/open-metadata/OpenMetadata/blob/cc80bd6f3c311537717c8f554053daa51a964705/ingestion/src/airflow_provider_openmetadata/lineage/runner.py#L411C14-L411C37