Scenario:
Implement with Redis:
Add reliability:
By default, Airflow tasks push and pull XComs via the metadata database (usually PostgreSQL or MySQL). A simple pattern is: airflow xcom exclusive
# Task A (Push)
def push_path(**context):
file_path = f"/data/report_context['ds'].csv"
context['ti'].xcom_push(key='report_path', value=file_path)
return file_path
from airflow.models.xcom import BaseXCom
from airflow.exceptions import AirflowException
class ExclusiveXCom(BaseXCom):
ALLOWED_PULLS =
("dag_etl", "extract", "load"): ["rows_count"],
("dag_etl", "transform", "report"): ["aggregated_metrics"],
@classmethod
def get_value(cls, key, dag_id, task_id, run_id, map_index):
# Enforce exclusive pull: only if (dag_id, calling_task, target_task) is allowed
calling_task = task_id # Note: in real implementation, you'd need to resolve caller
allowed_keys = cls.ALLOWED_PULLS.get((dag_id, calling_task), [])
if key not in allowed_keys:
raise AirflowException(
f"XCom exclusive violation: Task calling_task not allowed to pull key 'key'"
)
return super().get_value(key, dag_id, task_id, run_id, map_index)
XCom rows are uniquely identified by this combination of columns in Airflow database:
Implication: XComs are scoped to a specific DAG run and task instance; different execution_date/run_id or task_id isolates them.
Apache Airflow has become the de facto standard for workflow orchestration. At its heart lies a simple but powerful mechanism for task-to-task communication: XCom (short for "Cross-Communication"). By default, Airflow allows any task to push any piece of data—whether it’s a filename, a model accuracy score, or a JSON blob—to be pulled by any downstream task. Scenario:
But this flexibility comes at a cost. In large-scale data pipelines, the default XCom behavior can lead to bloated metadata databases, security vulnerabilities, race conditions, and debugging nightmares.
Enter XCom Exclusive Mode—a feature designed to enforce stricter boundaries, improve performance, and make your DAGs more predictable. But what exactly is it? How do you enable it? And is it right for your team?
This article dives deep into XCom exclusive mode, comparing it with the standard model, walking through practical examples, and revealing advanced patterns to level up your Airflow engineering. Implement with Redis: