Page MenuHomePhabricator
Create Task
Maniphest T433388

Adapt airflow chart for 3.3.x
Closed, ResolvedPublic
Actions

Description

Continues work from T432993

  • Use the airflow config update and airflow config update --fix commands to vlidate the configuration. P95569
  • The Dag processor must now be started independently, even for local or development setups: airflow dag-processor
    • airflow-devenv should wait for dag-processor to start before _Forcing DAG serialization_
    • import_helper.py is compatible with Airflow 3.3
  • Deploy to the airflow-devenv environment and execute airflow db migrate
  • Re-work chart patch
  • DAGs
    • setup python3 P95842
    • Setup core.execution_api_server_url to $baseurl/execution P95856
    • Setup jwt token (client P95864, server P95924, pod description P95856)

Acceptance criteria

  • all components are present
  • minimal test_k8s Dag is running

Details

Related Changes in Gerrit:
SubjectAuthorRepoBranchLines +/-
airflow: dag-processor for Airflow 3Atsukooperations/deployment-chartsmaster+122 -2
dag-processorAtsukooperations/deployment-chartsmaster+107 -2
airflow3 supportAtsukooperations/deployment-chartsmaster+7 -4
airflow3 supportAtsukooperations/deployment-chartsmaster+3 -4
airflow: support for v3.3.0Atsukooperations/deployment-chartsmaster+25 -6
airflow: undo dev instance version pinningAtsukooperations/deployment-chartsmaster+0 -3
Customize query in gerrit
Related Changes in GitLab:
TitleReferenceAuthorSource BranchDest Branch
Updating Spark provider to 5.6 due to import errorrepos/data-engineering/airflow-dags!2497atsukoT433388main_airflow_3.3
Create deployment with extra config filerepos/data-engineering/airflow-devenv!19atsukoT433388main
Customize query in GitLab

Related Objects
Search...

StatusSubtypeAssignedTask
Restricted Task
 ResolvedBTullisT423243 Upgrade Airflow to 2.11.2
 In ProgressatsukoT423248 Upgrade to Airflow 3.3.x
 ResolvedatsukoT433388 Adapt airflow chart for 3.3.x

Event Timeline

atsuko created this task.Jul 28 2026, 2:43 PM
atsuko mentioned this in T423248: Upgrade to Airflow 3.3.x.Jul 28 2026, 2:47 PM
atsuko removed atsuko as the assignee of this task.Jul 28 2026, 2:49 PM
atsuko updated the task description. (Show Details)
CodeReviewBot added a project: Patch-For-Review.Jul 29 2026, 10:30 AM

atsuko opened https://gitlab.wikimedia.org/repos/data-engineering/airflow-dags/-/merge_requests/2445

Draft: Building Airflow 3.3.0 image

CodeReviewBot added a comment.Jul 29 2026, 11:00 AM

atsuko merged https://gitlab.wikimedia.org/repos/data-engineering/airflow-dags/-/merge_requests/2445

Building Airflow 3.3.0 image

atsuko updated the task description. (Show Details)Jul 29 2026, 11:02 AM
Maintenance_bot removed a project: Patch-For-Review.Jul 29 2026, 11:31 AM
atsuko updated the task description. (Show Details)Jul 29 2026, 11:37 AM
atsuko updated the task description. (Show Details)Jul 29 2026, 1:11 PM
atsuko changed the task status from Open to In Progress.Jul 29 2026, 1:56 PM
atsuko moved this task from Backlog - project to In Progress on the Data-Platform-SRE (2026-07-03 - 2026-07-31) board.
atsuko edited projects, added: Data-Platform-SRE (2026-07-03 - 2026-07-31); removed: Data-Platform-SRE.
atsuko updated the task description. (Show Details)Jul 29 2026, 1:58 PM
CodeReviewBot added a project: Patch-For-Review.Jul 29 2026, 2:40 PM

atsuko opened https://gitlab.wikimedia.org/repos/data-engineering/airflow-dags/-/merge_requests/2446

Upgrade Context to Airflow 3.3

CodeReviewBot added a comment.Jul 31 2026, 2:15 PM

atsuko opened https://gitlab.wikimedia.org/repos/data-engineering/airflow-devenv/-/merge_requests/19

Create deployment with extra config file

gerritbot added a comment.Jul 31 2026, 2:18 PM

Change #1319858 had a related patch set uploaded (by Atsuko; author: Atsuko):

[operations/deployment-charts@master] airflow: undo dev instance version pinning

https://gerrit.wikimedia.org/r/1319858

atsuko added a comment.EditedJul 31 2026, 2:31 PM

Tried running DAGs, k8s pods are failing with python not being found P95842

Providers list doesn't match a little

-apache-airflow-providers-ftp
-apache-airflow-providers-imap
+apache-airflow-providers-openlineage
+apache-airflow-providers-standard
gerritbot added a comment.Jul 31 2026, 2:32 PM

Change #1318719 had a related patch set uploaded (by Atsuko; author: Atsuko):

[operations/deployment-charts@master] airflow: support for v3.3.0

https://gerrit.wikimedia.org/r/1318719

gerritbot added a comment.Jul 31 2026, 3:12 PM

Change #1319858 merged by jenkins-bot:

[operations/deployment-charts@master] airflow: undo dev instance version pinning

https://gerrit.wikimedia.org/r/1319858

gerritbot added a comment.Jul 31 2026, 3:13 PM

Change #1318719 merged by jenkins-bot:

[operations/deployment-charts@master] airflow: support for v3.3.0

https://gerrit.wikimedia.org/r/1318719

atsuko added a comment.Jul 31 2026, 4:51 PM

After merging leftover diffs, the command to try airflow 3.3.0 is

airflow-devenv create \
    --branch main_airflow_3.3 --extra-values values-airflow3.yaml \
    -d test-k8s -f dags/kubernetes_pod_operator_dag.py

Directory -d and list of dags -f can be different.

atsuko updated the task description. (Show Details)Jul 31 2026, 4:53 PM
CodeReviewBot added a comment.Aug 3 2026, 8:46 AM

atsuko merged https://gitlab.wikimedia.org/repos/data-engineering/airflow-dags/-/merge_requests/2446

Upgrade plugins dependencies to Airflow 3.3

CodeReviewBot added a comment.Aug 3 2026, 8:47 AM

atsuko merged https://gitlab.wikimedia.org/repos/data-engineering/airflow-devenv/-/merge_requests/19

Create deployment with extra config file

Maintenance_bot removed a project: Patch-For-Review.Aug 3 2026, 9:32 AM
atsuko added a comment.EditedAug 3 2026, 9:55 AM

Gr-r-r, airflow hardcodes python in every invocation, so P95842 happens, rebuilding the image.

Upd: new image has helped, now it needs to actually be able to reach API server P95856.

CodeReviewBot added a project: Patch-For-Review.Aug 3 2026, 10:22 AM

atsuko opened https://gitlab.wikimedia.org/repos/data-engineering/airflow-dags/-/merge_requests/2456

Provide python in PATH

CodeReviewBot added a comment.Aug 3 2026, 10:35 AM

atsuko merged https://gitlab.wikimedia.org/repos/data-engineering/airflow-dags/-/merge_requests/2456

Provide python in PATH

atsuko updated the task description. (Show Details)Aug 3 2026, 11:03 AM
atsuko added a comment.EditedAug 3 2026, 11:31 AM

To make the executor work P95856, we need to address the difference between Airflow 2 and 3 (see F75945493 and F75945523 for in-process API server and links for API server), where all the requests need to go thru the API server.

For this, we need to run envoy to the API server (or the API server instance maybe) in a sidecar.

  • Airflow 2 architecture

image.png (1,940×1,162 px, 288 KB)

  • Airflow 3 architecture

image.png (2,098×1,172 px, 372 KB)

Maintenance_bot removed a project: Patch-For-Review.Aug 3 2026, 11:32 AM
atsuko updated the task description. (Show Details)Aug 3 2026, 11:32 AM
gerritbot added a comment.Aug 3 2026, 11:41 AM

Change #1320131 had a related patch set uploaded (by Atsuko; author: Atsuko):

[operations/deployment-charts@master] update airflow3 image

https://gerrit.wikimedia.org/r/1320131

gerritbot added a project: Patch-For-Review.Aug 3 2026, 11:41 AM
atsuko updated the task description. (Show Details)Aug 3 2026, 11:43 AM
atsuko mentioned this in T433383: Run test-k8s instance on Airflow 3.3.Aug 3 2026, 1:11 PM
atsuko added a comment.EditedAug 4 2026, 10:30 AM

After applying temporary patch that will ID the containers, P95860, I can say that airflow.sdk.execution_time.execute_workload is using taskpod template. Now thats easy.

Progress in https://gerrit.wikimedia.org/r/c/operations/deployment-charts/+/1320131

ToDo

  • Setup core.execution_api_server_url to $baseurl/execution

[]x Setup proxies if needed in kubernetes-executor.pod-template

  • Figure out auth problem with the token (client P95864, server P95924, pod description P95856)
atsuko claimed this task.Aug 4 2026, 2:22 PM
atsuko added a comment.EditedAug 6 2026, 11:37 AM

Added airflow__api_auth__jwt_secret to airflow-test-k8s private config, will need to add to the rest of the deployments.

commit 68797c03e6617b049077224de9cb1637f04d4eba (HEAD -> master)
Date:   Thu Aug 6 11:35:01 2026 +0000

    (atsuko) airflow-test-k8s: adding JWT secret for Airflow 3
atsuko updated the task description. (Show Details)Aug 6 2026, 1:44 PM
gerritbot added a comment.Aug 6 2026, 1:48 PM

Change #1320131 merged by jenkins-bot:

[operations/deployment-charts@master] airflow3 support

https://gerrit.wikimedia.org/r/1320131

Maintenance_bot removed a project: Patch-For-Review.Aug 6 2026, 2:32 PM
CodeReviewBot added a project: Patch-For-Review.Aug 7 2026, 11:06 AM

atsuko opened https://gitlab.wikimedia.org/repos/data-engineering/airflow-dags/-/merge_requests/2497

Updating Spark provider to 5.6 due to import error

CodeReviewBot added a comment.Aug 7 2026, 11:30 AM

atsuko merged https://gitlab.wikimedia.org/repos/data-engineering/airflow-dags/-/merge_requests/2497

Updating Spark provider to 5.6 due to import error

Maintenance_bot removed a project: Patch-For-Review.Aug 7 2026, 11:31 AM
CodeReviewBot added a project: Patch-For-Review.Aug 7 2026, 1:29 PM

atsuko opened https://gitlab.wikimedia.org/repos/data-engineering/airflow-dags/-/merge_requests/2500

Draft: fixing cached_property imports

CodeReviewBot added a comment.Aug 7 2026, 1:29 PM

atsuko opened https://gitlab.wikimedia.org/repos/data-engineering/airflow-dags/-/merge_requests/2501

Draft: fixing cached_property imports

gerritbot added a comment.Aug 7 2026, 6:16 PM

Change #1322840 had a related patch set uploaded (by Atsuko; author: Atsuko):

[operations/deployment-charts@master] airflow3 support

https://gerrit.wikimedia.org/r/1322840

gerritbot added a comment.Aug 8 2026, 9:37 AM

Change #1322840 merged by jenkins-bot:

[operations/deployment-charts@master] airflow3 support

https://gerrit.wikimedia.org/r/1322840

Gehel edited projects, added: Data-Platform-SRE (2026-08-07 - 2026-08-28); removed: Data-Platform-SRE (2026-07-03 - 2026-07-31).Aug 11 2026, 6:54 AM
Gehel moved this task from Backlog - project to In Progress on the Data-Platform-SRE (2026-08-07 - 2026-08-28) board.
CodeReviewBot added a comment.Aug 11 2026, 11:00 AM

atsuko merged https://gitlab.wikimedia.org/repos/data-engineering/airflow-dags/-/merge_requests/2501

fixing cached_property imports

CodeReviewBot added a comment.Aug 11 2026, 11:01 AM

atsuko merged https://gitlab.wikimedia.org/repos/data-engineering/airflow-dags/-/merge_requests/2500

fixing cached_property imports

atsuko added a comment.Aug 11 2026, 11:11 AM

Added secrets for the rest of the deployments

commit f70ab2c72d3016f21d904ccf0dff5d70dcc7d44c (HEAD -> master)
Date:   Tue Aug 11 11:09:44 2026 +0000

    (atsuko) airflow: adding JWT secret for Airflow 3
Maintenance_bot removed a project: Patch-For-Review.Aug 11 2026, 11:31 AM
atsuko added a comment.Aug 12 2026, 9:55 AM

The only leftover is dag-processor CR, gonna merge it today and we are done here.

gerritbot added a comment.Aug 12 2026, 12:57 PM

Change #1324711 had a related patch set uploaded (by Atsuko; author: Atsuko):

[operations/deployment-charts@master] dag-processor

https://gerrit.wikimedia.org/r/1324711

gerritbot added a project: Patch-For-Review.Aug 12 2026, 12:57 PM

Change #1324711 abandoned by Atsuko:

[operations/deployment-charts@master] dag-processor

Reason:

duplicate of Ie7c17175d84c1e3f5fc5a8d7bb64f5836f7459b2

https://gerrit.wikimedia.org/r/1324711

gerritbot added a comment.Aug 12 2026, 12:59 PM

Change #1324678 had a related patch set uploaded (by Atsuko; author: Atsuko):

[operations/deployment-charts@master] dag-processor

https://gerrit.wikimedia.org/r/1324678

atsuko updated the task description. (Show Details)Aug 12 2026, 1:17 PM
atsuko closed this task as Resolved.Aug 12 2026, 1:19 PM
atsuko moved this task from In Progress to Done on the Data-Platform-SRE (2026-08-07 - 2026-08-28) board.
gerritbot added a comment.Aug 12 2026, 1:27 PM

Change #1324678 merged by jenkins-bot:

[operations/deployment-charts@master] airflow: dag-processor for Airflow 3

https://gerrit.wikimedia.org/r/1324678

Maintenance_bot removed a project: Patch-For-Review.Aug 12 2026, 1:31 PM
Gehel moved this task from Done to Reported on the Data-Platform-SRE (2026-08-07 - 2026-08-28) board.Aug 14 2026, 5:43 AM
Content licensed under Creative Commons Attribution-ShareAlike (CC BY-SA) 4.0 unless otherwise noted; code licensed under GNU General Public License (GPL) 2.0 or later and other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL · Credits