Page MenuHomePhabricator
Create Task
Maniphest T423243

Upgrade Airflow to 2.11.2
Closed, ResolvedPublic
Actions

Description

We need to upgrade our Airflow instances to version 2.11.2, which is expected to be the last version in the 2.x line.

See: https://airflow.apache.org/docs/apache-airflow/stable/installation/supported-versions.html#version-life-cycle

This will be a pre-requisite for upgrading to the Airflow 3.x line.

We are currently on version 2.10.5 of Airflow.

  • Read https://airflow.apache.org/docs/apache-airflow/2.11.2/installation/upgrading.html
  • Read the release notes from 2.11.0
  • Read the release notes from 2.11.1
  • Read the release notes from 2.11.2
  • Create an updated Airflow image using version 2.11.2
  • Update the airflow chart as required - No updates seem to be required
  • Deploy the new version to an airflow-devenv environment
  • Deploy the new version to all instances
    • test-k8s
    • analytics-test
    • main
    • research
    • search
    • platform-eng
    • analytics-product
    • wmde
    • wikidata
    • sre
    • fr-tech
    • ml

Details

Related Changes in Gerrit:
Show related patches Customize query in gerrit

Related Objects
Search...

StatusSubtypeAssignedTask
Restricted Task
 ResolvedBTullisT423243 Upgrade Airflow to 2.11.2
 In ProgressatsukoT423248 Upgrade to Airflow 3.3.x
 ResolvedatsukoT432993 Package and explore Airflow 3.3.0 in airflow-devenv
 ResolvedatsukoT433383 Run test-k8s instance on Airflow 3.3
 StalledNoneT433385 Upgrade airflow-dags image to trixie/python 3.13
 ResolvedatsukoT433388 Adapt airflow chart for 3.3.x
 OpenNoneT434283 Upgrade airflow-dags codebase to both Airflow 2.x and 3.x compatible
 OpenNoneT434642 Create diffs for the DAG owners to migrate from SLA to Deadline
 ResolvedatsukoT434786 Degradation in upgraded Airflow Spark provider
 In ProgressatsukoT434896 Fix direct database access in Airflow 3
 ResolvedatsukoT437236 Implement K8s JWT authentication for Airflow 3 REST API
 ResolvedatsukoT436269 Upgrade apache-airflow-client to a 3.3.x version
 ResolvedatsukoT435172 SparkSubmitOperator job is failing on Airflow3
 OpenNoneT435476 Upgrade airflow-analytics-product to 3.3.x
 In ProgressRKemperT435477 Upgrade airflow-analytics-test to 3.3.x
 OpenNoneT435478 Upgrade airflow-dumps to 3.3.x
 In ProgressNoneT435479 Upgrade airflow-fr-tech to 3.3.x
 In ProgressAhoelzlT435480 Upgrade airflow-main to 3.3.x
 OpenNoneT435481 Upgrade airflow-ml to 3.3.x
 OpenNoneT435482 Upgrade airflow-platform-eng to 3.3.x
 OpenNoneT435483 Upgrade airflow-research to 3.3.x
 OpenNoneT435484 Upgrade airflow-search to 3.3.x
 In ProgressNoneT435485 Upgrade airflow-sre to 3.3.x
 OpenNoneT435486 Upgrade airflow-wikidata to 3.3.x
 In ProgressNoneT435487 [Analytics] [Tech Debt] Upgrade airflow-wmde to 3.3.x
 OpenNoneT435942 [Analytics] [Tech Debt] Check of WMDE wiki_page_cite_references DAG in Airflow 3.3
 ResolvedRKemperT437981 Airflow 3: remove Amazon provider warnings from CLI startup
 In ProgressatsukoT437994 CI for Airflow 3
 ResolvedatsukoT439576 airflow3: data interval length is zero and logical date points to actual time when the dag starts
 ResolvedbrouberolT423279 Create the airflow 2.11.2 docker image

Event Timeline

BTullis created this task.Apr 14 2026, 10:04 AM
BTullis triaged this task as Medium priority.
BTullis created subtask T423248: Upgrade to Airflow 3.3.x.Apr 14 2026, 11:22 AM
BTullis mentioned this in T423248: Upgrade to Airflow 3.3.x.
brouberol added a subtask: T423279: Create the airflow 2.11.2 docker image.Apr 14 2026, 12:52 PM
brouberol changed the status of subtask T423279: Create the airflow 2.11.2 docker image from Open to In Progress.Apr 14 2026, 1:26 PM
BTullis edited projects, added: Data-Platform-SRE (2026-03-27 - 2026-04-17); removed: Data-Platform-SRE.Apr 14 2026, 1:40 PM
BTullis moved this task from Backlog - project to In Progress on the Data-Platform-SRE (2026-03-27 - 2026-04-17) board.
brouberol closed subtask T423279: Create the airflow 2.11.2 docker image as Resolved.Apr 14 2026, 2:06 PM
BTullis mentioned this in T405509: Provide an access to MaxMind GeoIP in DSE K8S pods.Apr 17 2026, 9:58 AM
BTullis claimed this task.Apr 17 2026, 11:16 AM

I have tested the new image in a devenv and everything seems good so far.

I used this command:

airflow-devenv create -d test_k8s -b geoip_test_dag --set app.version=airflow-2.11.2-py3.11-2026-04-16-134304-f4802bd4391b8fc561804913728472d5a2721d42@sha256:976f939064787fa65c1c551e72f697120e611e5610441d6869c7523d723d429b,app.executor_pod_image_version=airflow-2.11.2-py3.11-2026-04-16-134304-f4802bd4391b8fc561804913728472d5a2721d42@sha256:976f939064787fa65c1c551e72f697120e611e5610441d6869c7523d723d429b

...and I documented this testing process here: https://wikitech.wikimedia.org/wiki/Data_Platform/Systems/Airflow/Developer_guide#Testing_a_pre-release_Airflow_image_in_your_development_environment

image.png (2,387×1,208 px, 247 KB)

image.png (2,375×460 px, 246 KB)

All seems fine so far. I have launched a task with the standard KubernetesExecutor and another with the KubernetesPodOperator and both were fine.

I'll claim the task and look to roll out the other instances next week, all being well.

BTullis updated the task description. (Show Details)Apr 17 2026, 11:17 AM
BTullis updated the task description. (Show Details)Apr 17 2026, 11:50 AM
gerritbot added a comment.Apr 21 2026, 11:11 AM

Change #1275854 had a related patch set uploaded (by Btullis; author: Btullis):

[operations/deployment-charts@master] Deploy the new Airflow version as the default for devenvs

https://gerrit.wikimedia.org/r/1275854

gerritbot added a project: Patch-For-Review.Apr 21 2026, 11:11 AM

Change #1275855 had a related patch set uploaded (by Btullis; author: Btullis):

[operations/deployment-charts@master] Deploy the new Airflow version to the test-k8s instance

https://gerrit.wikimedia.org/r/1275855

gerritbot added a comment.Apr 21 2026, 11:11 AM

Change #1275856 had a related patch set uploaded (by Btullis; author: Btullis):

[operations/deployment-charts@master] Deploy the new Airflow version to the analytics-test instance

https://gerrit.wikimedia.org/r/1275856

gerritbot added a comment.Apr 21 2026, 11:11 AM

Change #1275857 had a related patch set uploaded (by Btullis; author: Btullis):

[operations/deployment-charts@master] Deploy the new airflow version to the main instance

https://gerrit.wikimedia.org/r/1275857

gerritbot added a comment.Apr 21 2026, 11:12 AM

Change #1275858 had a related patch set uploaded (by Btullis; author: Btullis):

[operations/deployment-charts@master] Deploy the new Airflow version to the platform-eng instance

https://gerrit.wikimedia.org/r/1275858

gerritbot added a comment.Apr 21 2026, 11:12 AM

Change #1275859 had a related patch set uploaded (by Btullis; author: Btullis):

[operations/deployment-charts@master] Deploy the new airflow version to the search instance

https://gerrit.wikimedia.org/r/1275859

gerritbot added a comment.Apr 21 2026, 11:12 AM

Change #1275860 had a related patch set uploaded (by Btullis; author: Btullis):

[operations/deployment-charts@master] Deploy the new Airflow version to the analytics-product instance

https://gerrit.wikimedia.org/r/1275860

gerritbot added a comment.Apr 21 2026, 11:12 AM

Change #1275861 had a related patch set uploaded (by Btullis; author: Btullis):

[operations/deployment-charts@master] Deploy the new Airflow version to the research instance

https://gerrit.wikimedia.org/r/1275861

gerritbot added a comment.Apr 21 2026, 11:12 AM

Change #1275862 had a related patch set uploaded (by Btullis; author: Btullis):

[operations/deployment-charts@master] Deploy the new Airflow version to the ml instance

https://gerrit.wikimedia.org/r/1275862

gerritbot added a comment.Apr 21 2026, 11:12 AM

Change #1275863 had a related patch set uploaded (by Btullis; author: Btullis):

[operations/deployment-charts@master] Deploy the new Airflow version to the sre instance

https://gerrit.wikimedia.org/r/1275863

gerritbot added a comment.Apr 21 2026, 11:12 AM

Change #1275864 had a related patch set uploaded (by Btullis; author: Btullis):

[operations/deployment-charts@master] Deploy the new Airflow version to the wmde instance

https://gerrit.wikimedia.org/r/1275864

gerritbot added a comment.Apr 21 2026, 11:12 AM

Change #1275865 had a related patch set uploaded (by Btullis; author: Btullis):

[operations/deployment-charts@master] Deploy the new Airflow version to the wikidata instance

https://gerrit.wikimedia.org/r/1275865

gerritbot added a comment.Apr 21 2026, 11:12 AM

Change #1275866 had a related patch set uploaded (by Btullis; author: Btullis):

[operations/deployment-charts@master] Deploy the new Airflow version to the fr-tech instance

https://gerrit.wikimedia.org/r/1275866

AndrewTavis_WMDE subscribed.Apr 23 2026, 11:03 AM
gerritbot added a comment.Apr 23 2026, 11:13 AM

Change #1275854 merged by jenkins-bot:

[operations/deployment-charts@master] Deploy the new Airflow version as the default for devenvs

https://gerrit.wikimedia.org/r/1275854

gerritbot added a comment.Apr 23 2026, 11:13 AM

Change #1275855 merged by jenkins-bot:

[operations/deployment-charts@master] Deploy the new Airflow version to the test-k8s instance

https://gerrit.wikimedia.org/r/1275855

gerritbot added a comment.Apr 23 2026, 11:14 AM

Change #1275856 merged by jenkins-bot:

[operations/deployment-charts@master] Deploy the new Airflow version to the analytics-test instance

https://gerrit.wikimedia.org/r/1275856

BTullis updated the task description. (Show Details)Apr 23 2026, 1:05 PM
Gehel edited projects, added: Data-Platform-SRE (2026-04-24 - 2026-05-15); removed: Data-Platform-SRE (2026-03-27 - 2026-04-17).Apr 24 2026, 9:16 AM
Gehel moved this task from Backlog - project to In Progress on the Data-Platform-SRE (2026-04-24 - 2026-05-15) board.
gerritbot added a comment.Apr 27 2026, 1:10 PM

Change #1275857 merged by jenkins-bot:

[operations/deployment-charts@master] Deploy the new airflow version to the main instance

https://gerrit.wikimedia.org/r/1275857

BTullis updated the task description. (Show Details)Apr 27 2026, 1:15 PM
gerritbot added a comment.Apr 27 2026, 1:45 PM

Change #1275858 merged by jenkins-bot:

[operations/deployment-charts@master] Deploy the new Airflow version to the platform-eng instance

https://gerrit.wikimedia.org/r/1275858

gerritbot added a comment.Apr 27 2026, 1:45 PM

Change #1275859 merged by jenkins-bot:

[operations/deployment-charts@master] Deploy the new airflow version to the search instance

https://gerrit.wikimedia.org/r/1275859

BTullis updated the task description. (Show Details)Apr 27 2026, 1:57 PM
gerritbot added a comment.Apr 27 2026, 2:06 PM

Change #1275860 merged by jenkins-bot:

[operations/deployment-charts@master] Deploy the new Airflow version to the analytics-product instance

https://gerrit.wikimedia.org/r/1275860

gerritbot added a comment.Apr 27 2026, 2:06 PM

Change #1275861 merged by jenkins-bot:

[operations/deployment-charts@master] Deploy the new Airflow version to the research instance

https://gerrit.wikimedia.org/r/1275861

BTullis updated the task description. (Show Details)Apr 27 2026, 2:10 PM
gerritbot added a comment.Apr 27 2026, 2:13 PM

Change #1275862 merged by jenkins-bot:

[operations/deployment-charts@master] Deploy the new Airflow version to the ml instance

https://gerrit.wikimedia.org/r/1275862

gerritbot added a comment.Apr 27 2026, 2:13 PM

Change #1275863 merged by jenkins-bot:

[operations/deployment-charts@master] Deploy the new Airflow version to the sre instance

https://gerrit.wikimedia.org/r/1275863

gerritbot added a comment.Apr 27 2026, 2:13 PM

Change #1275864 merged by jenkins-bot:

[operations/deployment-charts@master] Deploy the new Airflow version to the wmde instance

https://gerrit.wikimedia.org/r/1275864

gerritbot added a comment.Apr 27 2026, 2:14 PM

Change #1275865 merged by jenkins-bot:

[operations/deployment-charts@master] Deploy the new Airflow version to the wikidata instance

https://gerrit.wikimedia.org/r/1275865

gerritbot added a comment.Apr 27 2026, 2:14 PM

Change #1275866 merged by jenkins-bot:

[operations/deployment-charts@master] Deploy the new Airflow version to the fr-tech instance

https://gerrit.wikimedia.org/r/1275866

BTullis closed this task as Resolved.Apr 27 2026, 2:29 PM
BTullis updated the task description. (Show Details)
BTullis moved this task from In Progress to Done on the Data-Platform-SRE (2026-04-24 - 2026-05-15) board.
Maintenance_bot removed a project: Patch-For-Review.Apr 27 2026, 2:31 PM
Gehel moved this task from Done to Reported on the Data-Platform-SRE (2026-04-24 - 2026-05-15) board.May 1 2026, 7:20 AM
atsuko changed the status of subtask T423248: Upgrade to Airflow 3.3.x from Open to In Progress.Jul 28 2026, 2:50 PM
Content licensed under Creative Commons Attribution-ShareAlike (CC BY-SA) 4.0 unless otherwise noted; code licensed under GNU General Public License (GPL) 2.0 or later and other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL · Credits