As per this chart, Airflow version 2.x will go EoL on April 22 2026.
We need to be able to move to the 3.x line of Airflow, which is the current stable branch, in order to be able to keep up-to-date with security patches, as well as being able to support the new features available in the 3.x line.
The stable version at the time of writing is 3.3.1
This ticket will track the work to upgrade our Airflow services to this version. Notes are tracked in google doc and translated to the task.
Plan
- Setup pre-requisites to unblock further work. This is a short research phase to get the code working, as the preferred ways of running the Airflow is in DSE K8S. Acceptance criteria: *Make Airflow 3.3 on airflow-devenv able to run minimal generic DAG*
- Consider all possible upgrades before moving to 3.3.1
-
T433385 Debian/Javaout of scope now - T434283 Upgrade airflow-dags codebase to both Airflow 2.x and 3.x compatible
- Step 3: Dag authors - Check your Airflow Dags for compatibility
- Use the ruff tool with AIR rules to check all DAGs.
- Update import statements.
-
- Make test_k8s instance work on Airflow 3.3. This is still research and fix phase, which will start affecting shared codebase and shape the rest of the migration.
- If you configured Single-Sign-On (SSO) using OAuth, OIDC, or LDAP, make sure that the authentication is working as expected. If you use a custom webserver_config.py you need to replace from airflow.www.security import AirflowSecurityManager with from airflow.providers.fab.auth_manager.security_manager.override import FabAirflowSecurityManagerOverride.
- Per-instance upgrade. After upgrading test_k8s instance we will know how to work run WMF code on Airflow 3.3. We might need but will try to avoid to keep the DAGs code for Airflow 3.3 separate and in sync with main 2.x branch, at least for shared components.
Old task description is archived in P96212