Airflow - Xcom In

✅ or ensure upstream dependencies with >> . ❌ Using XComs for many small values across many tasks Each XCom is a DB row. 10 000 tasks × 5 XComs = 50 000 rows – fine. But 100 000 tasks × 10 XComs = 1 million rows – slow. Advanced: XCom Backends Airflow 2.0+ lets you store XComs outside the metadata DB. Useful if you need slightly larger values or lower DB load.

push = PythonOperator(task_id='push_task', python_callable=push_function) pull = PythonOperator(task_id='pull_task', python_callable=pull_function) xcom in airflow

Now go build DAGs that actually share information – cleanly and reliably. ✅ or ensure upstream dependencies with >>

@task def aggregate(results: list[str]): print(f"All results: results") But 100 000 tasks × 10 XComs = 1 million rows – slow

def pull_function(**context): user_id = context['ti'].xcom_pull(task_ids='push_task', key='user_id') print(f"Received user_id")

Here’s a structured, useful blog post about — written for data engineers who want to move beyond basic tasks and build real DAGs. Mastering XComs in Apache Airflow: Cross‑Task Communication Without the Pain One of the first surprises when learning Airflow is that tasks run isolated from each other. You can’t just set task_2.data = task_1.data . So how do you pass a value from one task to another? XComs .