Migration 0049 fails with ProgramLimitExceeded on PostgreSQL when an XCom value exceeds the jsonb size limit
Under which category would you file this issue?
Core
Apache Airflow version
upgrading from 2.11.2 to [3.x from Runtime 3.3-7] Database: PostgreSQL
What happened and how to reproduce it?
On PostgreSQL, xcom.value is bytea in Airflow 2, which allows up to 1 GB per value. Migration 0049 (remove_pickled_data_from_xcom_table) converts it to jsonb, which caps a single value at 268,435,455 bytes (~268 MB). XCom values that were valid in Airflow 2 cannot be converted, and the migration aborts with:
ProgramLimitExceeded: total size of jsonb array elements exceeds the maximum of 268435455 bytes
Alembic stays at 9fc3fc5de720 and every retry fails at the same step.
The upgrade cannot complete until the rows are removed.
Column definition in airflow-core/src/airflow/models/xcom.py: value = mapped_column(JSON().with_variant(postgresql.JSONB, "postgresql"), nullable=True)
This is specific to PostgreSQL. Other backends use generic JSON.
Details:
- The xcom table was ~30 GB in total. The limit is per value, not per table.
- 55 rows failed, each a task return_value close to the limit.
How to reproduce: On an Airflow 2 PostgreSQL metadata DB, store an XCom value larger than 268,435,455 bytes, then run the Airflow 3 migrations.
Query to find affected rows before upgrading:
SELECT dag_id, task_id, run_id, key, octet_length(value) AS bytes
FROM xcom
WHERE octet_length(value) > 268435455
ORDER BY bytes DESC;
Workaround: Copied the affected rows to a backup table, deleted them from xcom, and re-ran the migration. It completed.
What you think should happen instead?
Migration 0049 should move values that cannot fit in jsonb into a quarantine table (like the existing airflow_moved_* tables) and continue, or fail early with a clear list of affected rows. The upgrade docs should state the jsonb per-value limit for PostgreSQL.
Operating System
No response
Deployment
None
Apache Airflow Provider(s)
No response
Versions of Apache Airflow Providers
No response
Official Helm Chart version
Not Applicable
Kubernetes Version
No response
Helm Chart configuration
No response
Docker Image customizations
No response
Anything else?
No response
Are you willing to submit PR?
- Yes I am willing to submit a PR!
Code of Conduct
- I agree to follow this project's Code of Conduct
Source: apache/airflow