Airflow Xcom - Exclusive

By default, Airflow writes XCom data directly into its metadata database (SQLAlchemy-supported databases like PostgreSQL, MySQL, or SQLite). The data is serialized into text (typically JSON) and stored in the xcom table.

To create an exclusive, secure cloud-backed XCom system, you override the BaseXCom class: airflow xcom exclusive

[core] xcom_backend = my_plugins.s3_xcom_backend.S3XComBackend Use code with caution. 5. Airflow XCom Best Practices & Anti-Patterns By default, Airflow writes XCom data directly into

By default, Airflow uses the metadata database to store XComs via the BaseXCom class. While this works well for development, it comes with significant limitations in production. The most notable constraint is the 48KB size limit for stored values. Furthermore, the default backend can become a bottleneck when dealing with a large number of XComs, slowing down the database and, by extension, the entire scheduler. These limitations form the primary reason for seeking a more "exclusive" and robust solution. The most notable constraint is the 48KB size

Historically, Airflow allowed XCom values to be serialized using Python's pickle module, which could lead to security vulnerabilities and version incompatibilities. Modern Airflow , and pickling support is deprecated. Always ensure your XCom values are JSON‑serializable unless you have a very good reason to do otherwise.

# Task A and Task B run in parallel task_a >> task_c task_b >> task_c

If you would like to customize this workflow further, let me know: