Page MenuHomePhabricator

Upgrade to Airflow 3.3.x
Open, In Progress, Needs TriagePublic

Description

As per this chart, Airflow version 2.x will go EoL on April 22 2026.

We need to be able to move to the 3.x line of Airflow, which is the current stable branch, in order to be able to keep up-to-date with security patches, as well as being able to support the new features available in the 3.x line.

The stable version at the time of writing is 3.3.1

This ticket will track the work to upgrade our Airflow services to this version. Notes are tracked in google doc and translated to the task.

Plan

  • Setup pre-requisites to unblock further work. This is a short research phase to get the code working, as the preferred ways of running the Airflow is in DSE K8S. Acceptance criteria: *Make Airflow 3.3 on airflow-devenv able to run minimal generic DAG*
    • T432993 Setup up minimal infra for Airflow 3.x to explore the ways of keeping the code changes for docker image, DAGs, and charts.
    • T433388 Productionise the changeset to make it suitable for both airflow-devenv and helmfile deployment
  • Consider all possible upgrades before moving to 3.3.1
  • Make test_k8s instance work on Airflow 3.3. This is still research and fix phase, which will start affecting shared codebase and shape the rest of the migration.
    • If you configured Single-Sign-On (SSO) using OAuth, OIDC, or LDAP, make sure that the authentication is working as expected. If you use a custom webserver_config.py you need to replace from airflow.www.security import AirflowSecurityManager with from airflow.providers.fab.auth_manager.security_manager.override import FabAirflowSecurityManagerOverride.
  • Per-instance upgrade. After upgrading test_k8s instance we will know how to work run WMF code on Airflow 3.3. We might need but will try to avoid to keep the DAGs code for Airflow 3.3 separate and in sync with main 2.x branch, at least for shared components.

Old task description is archived in P96212

Details

Related Objects

StatusSubtypeAssignedTask
ResolvedBTullis
In Progressatsuko
Resolvedatsuko
OpenNone
OpenNone
Resolvedatsuko
OpenNone
OpenNone
Resolvedatsuko
OpenNone
In Progressatsuko
OpenNone
In ProgressRKemper
OpenNone
In ProgressNone
In ProgressJAllemandou
OpenNone
OpenNone
OpenNone
OpenNone
In ProgressNone
OpenNone
In ProgressNone
OpenNone
Openamastilovic

Event Timeline

There are a very large number of changes, so older changes are hidden. Show Older Changes

Claiming it as it is my quaterly goal

atsuko renamed this task from Upgrade to Airflow 3.2.x to Upgrade to Airflow 3.3.x.Jul 28 2026, 2:25 PM
atsuko updated the task description. (Show Details)

As the result of T433388, airflow 3.3.0 is available for test with airflow-devenv:

airflow-devenv create \
    --branch main_airflow_3.3 --extra-values values-airflow3.yaml \
    -d test_k8s

Directory (-d test_k8s) is an example only, can be any other instance.

As of now Dags are still not working, but I'm working on it as part of said task.

Airflow 3.3.1 has been released a few days ago and fixes a range of security issues, so let's rebase the 3.3 deployment to 3.3.1 while still very people are using it?

CVE-2026-54183: Airflow Variables were not masked in the UI for authenticated users
Severity: low

Apache Airflow's secrets masker hides values stored under sensitive key names when they are displayed in the UI. The masker's recursion-depth limit did not descend into values nested inside a list, tuple, or set beyond that limit, so an Airflow Variable holding such a deeply-nested value was shown unmasked in the Variables UI. The exposure is limited to the UI: any authenticated user who can see the Variable in the UI can already read its full value through the Variables REST API, so this does not disclose data the user could not otherwise obtain — the masking is a shoulder-surfing defense for the UI, not an access-control boundary.
This is an incomplete-fix follow-up to CVE-2026-42358, whose fix made only the dictionary walk unbounded; lists, tuples, and sets beyond the depth limit remained unmasked in the UI. Deployments that applied the CVE-2026-42358 fix should also upgrade to address this residual case. Upgrade to apache-airflow 3.3.1 or later.

CVE-2026-58076: Unguarded import_string() of airflow_exc_ser / base_exc_ser exception nodes in BaseSerialization.deserialize enables DAG-author RCE on Scheduler / API Server
Severity: important

Apache Airflow's serialization layer reconstructed exception nodes by calling import_string() on a class name taken from the serialized blob and instantiating it with arguments from the same blob, with no restriction on what could be imported. An operator's executor_config reaches that branch, so a Dag author could place a value there that causes an arbitrary callable to be imported and invoked -- for example subprocess.check_output, or builtins.eval on the builtins-prefixed variant. The code runs in the Scheduler, which reconstructs serialized Dags in its normal loop with no request involved, and in the API server, on any authenticated read of the Dag such as GET /api/v2/dags/{dag_id}/details. Both are components the Airflow security model states must never execute Dag-author code, and both hold the metadata database credentials and the JWT signing secret. No non-default configuration is required. This is a different sink from CVE-2026-33264, which covered only the trigger branch of the same deserializer: deployments that upgraded in response to that advisory are still affected through the exception branch and must upgrade again. Users are advised to upgrade to apache-airflow 3.3.1 or later, which restricts the imported class to a subclass of BaseException.

CVE-2026-59242: Arbitrary airflow.* class instantiation on the API server via the XCom deserialize endpoint
Severity: important

Apache Airflow's XCom GET /api/v2/{...}/xcomEntries/{key}?deserialize=true endpoint passed a string-literal payload through BaseXCom.deserialize_value without the _check_forbidden_xcom_keys guard, allowing an authenticated API user with XCom write-and-read access to instantiate arbitrary airflow.* classes on the API server (CWE-502). An authenticated user who can write an XCom value and then read it back with deserialize=true triggers the unsafe instantiation. Users are advised to upgrade to apache-airflow 3.3.1 or later, which rejects reserved XCom serialization keys submitted as JSON string literals.

CVE-2026-59244: Secrets masker: var.json Variable values not masked in the Rendered Templates UI
Severity: moderate

Apache Airflow's secrets masker did not mask var.json Variable values whose value is a dict in the Rendered Templates UI — the dict value failed an isinstance(str) guard — so a secret stored as a JSON Variable and referenced in a template via var.json was displayed in cleartext to any user with access to that task's Rendered Templates view. Users are advised to upgrade to apache-airflow 3.3.1 or later, which masks nested Variable values regardless of type.

CVE-2026-65017: Config API: team-scoped Celery broker secret disclosed to a Viewer (multi-team masking bypass)
Severity: moderate

Apache Airflow's Config API did not mask team-scoped sensitive configuration values in multi-team deployments. When an administrator has enabled multi-team mode and exposed the Config API, an authenticated Viewer holding only configuration-read access — with no prior access to the secret — could read a team-scoped Celery broker URL, including its embedded credentials, in cleartext, while the equivalent global option was correctly masked. The secrets masker matched only base section and option names and did not normalize team-prefixed sections before the sensitivity check (CWE-200). This is a distinct masker bypass from CVE-2026-48828 and CVE-2026-48892: deployments that upgraded to apache-airflow 3.3.0 to address those issues remain affected by this team-scoped variant. Users are advised to upgrade to apache-airflow 3.3.1 or later, which normalizes team-scoped sections before masking.

CVE-2026-67260: DAG-author remote code execution on the Scheduler via awaiting_input next_kwargs deserialization
Severity: important

Apache Airflow 3.3.0 moved human-in-the-loop tasks from the triggerer to a new awaiting_input task state swept by the scheduler. That sweep deserializes the task instance's next_kwargs without an allow-list, so a Dag author — who controls that value through the task execution API — can cause an arbitrary module import and object instantiation inside the scheduler process, or terminate the scheduler job. No non-default configuration is required: the sweep runs unconditionally every 15 seconds, and the default allowed_deserialization_classes setting does not cover this code path. Versions before 3.3.0 are not affected, because human-in-the-loop tasks deferred onto the triggerer instead. This is a different code path from CVE-2026-58076, which covers the same unguarded exception-node deserialization reached elsewhere — deployments that applied that fix must upgrade for this issue as well. Users are advised to upgrade to apache-airflow 3.3.1 or later.

CVE-2026-67587: DAG-author remote code execution on the Scheduler via a Serde Callback deserialization gadget
Severity: important

Apache Airflow's Task SDK rebuilt a Callback object from serialized data by re-running its constructor, which imports the module named by the stored callback path. Because SyncCallback is itself an Airflow class it passes the default allowed_deserialization_classes allow-list, so tightening that setting does not help. A Dag author — who controls a task instance's next_kwargs through the task execution API — can therefore cause an arbitrary module to be imported inside the scheduler process, when the scheduler's awaiting_input timeout sweep deserializes that value. No non-default configuration is required; the sweep runs unconditionally. Versions before 3.3.0 are not affected: the class existed, but the scheduler sweep that reaches it did not. This is a separate code path from CVE-2026-58076 and CVE-2026-67260, which cover different gadgets reaching deserialization — applying either of those fixes does not address this one. Users are advised to upgrade to apache-airflow 3.3.1 or later.

CVE-2026-68076: Connections test API: team-scope guard bypass resolves another team's environment Connection
Severity: moderate

Apache Airflow's environment-variable secrets backend resolved a team-scoped Connection or Variable from the wrong team's scope. The guard meant to prevent this only ran when no team scope was supplied, and its pattern could not match a team name containing an underscore, which team names are allowed to contain. When the guard did not apply, the lookup fell through to an unconditional global read that resolved the stored AIRFLOW_CONN__<TEAM>___<ID> variable regardless of which team asked. In multi-team mode an authenticated user of one team could therefore have POST /api/v2/connections/test resolve another team's Connection and authenticate outward with that team's credentials; the endpoint uses the credentials rather than returning them. Exploitation requires [core] multi_team enabled, [core] test_connection set to Enabled (it ships Disabled), team-scoped secrets provisioned as environment variables in the API-server process, and knowledge of the encoded identifier. Redirecting the test at an attacker-controlled host is separately blocked. Users are advised to upgrade to apache-airflow 3.3.1 or later.

CVE-2026-68968: Authorization bypass in the Backfill API through conflicting interpretations of the backfill id
Severity: important

Apache Airflow's Backfill API authorized a request against a Dag id supplied by the caller whenever the backfill_id path segment failed to parse. The authorization dependency parsed it with int() while the route handler parsed it as pydantic's NonNegativeInt, which accepts values int() rejects (1.0 coerces to 1); FastAPI resolves dependencies before endpoint validation, so the two acted on different Dags. An authenticated user holding edit permission on any single Dag could therefore read, pause and cancel backfills belonging to any other Dag, including moving another Dag's queued runs to failed. No non-default configuration is required and backfill ids are sequential, so finding a
target is trivial. Users are advised to upgrade to apache-airflow 3.3.1 or later, which parses the backfill id with the same type the routes declare.

CVE-2026-68969: Bulk Variable and Connection endpoints record secret values in the audit log in cleartext
Severity: moderate

Apache Airflow wrote Variable values and Connection extra contents to the audit log in cleartext when they were submitted through the bulk endpoints (PATCH /api/v2/variables and PATCH /api/v2/connections). The audit-log masking recognised only top-level request fields, and a bulk request nests its entities two levels below, so no masking was applied to them. Any authenticated user with audit-log read access -- who need not hold Variables or Connections read at all -- could recover those secrets verbatim, and the Connection extra copy is stored unencrypted in the log while the connection table encrypts it. The Airflow UI's *Import Variables* action posts to this endpoint, so an ordinary operator import wrote every secret in the file to the log. This is a different code path from CVE-2026-50204: that fix shipped in 3.3.0 and covers the single-entity endpoints only, so deployments that upgraded in response to that advisory remain affected and must upgrade again. Users are advised to upgrade to apache-airflow 3.3.1 or later.

CVE-2026-68970: Values of a list-shaped Variable are not masked in task logs and the Rendered Templates UI
Severity: moderate

Apache Airflow's Task SDK did not mask the contents of a Variable whose JSON value is a list, so secrets stored in that shape appeared in cleartext in task logs and in the Rendered Templates UI. Masking was applied only when the deserialized value was a string or a dict; a list at the top level matched neither and was returned unmasked. Any authenticated user able to read the logs or rendered templates of a task that references such a Variable could recover the values, with no special configuration required. This is the list-shaped counterpart of CVE-2026-59244, whose fix covered the dict case only, so deployments that upgraded in response to that advisory remain affected and must upgrade again. Users are advised to upgrade to apache-airflow 3.3.1 or later.

CVE-2026-68971: Cross-team authorization bypass in the asset materialization and dag-run result endpoints
Severity: moderate

Apache Airflow's asset materialization endpoint (POST /api/v2/assets/{asset_id}/materialize) and the XCom result check on wait_dag_run_until_finished authorized the target Dag without its team, unlike every other authorization site. A team-aware auth manager distinguishes a team-scoped Dag from a global one by that field -- the Keycloak auth manager, for example, checks the DAG resource instead of DAG:<team> -- so the team-scoped permission that should gate the request was never consulted. In a deployment running multi-team mode with a team-aware auth manager, an authenticated user in one team could trigger Dag runs belonging to another team, supplying their own dag_run_id and conf, and could read another team's XCom values. Deployments using the FAB auth manager are unaffected, as it has no multi-team support. Users are advised to upgrade to apache-airflow 3.3.1 or later, which resolves the Dag's team at both sites.

Created a new MR, will update today. Previous version is in the tag.

Built docker-registry.wikimedia.org/repos/data-engineering/airflow-dags:airflow-3.3.1-py3.11-2026-08-17-132624-cb7bea2af5eb4f962449b6d162518c19ca70547f@sha256:61ebd3f988933724d441c3ec2d1c3f31c0ca36b4818b39910e74c0d6b079b59b and merged in https://gerrit.wikimedia.org/r/c/operations/deployment-charts/+/1326300

Change #1335780 had a related patch set uploaded (by Atsuko; author: Atsuko):

[operations/deployment-charts@master] airflow3: updating airflow-client

https://gerrit.wikimedia.org/r/1335780