Page MenuHomePhabricator
Create Task
Maniphest T433383

Run test-k8s instance on Airflow 3.3
Closed, ResolvedPublic
Actions

Description

TBC, dependent on T433388

  • what to do with shared code, how to keep code in sync
  • decisions on SLA->Deadline, Database access

ToDo

Acceptance criteria

  • test_k8s running on main_airflow_3.3 branch with 3.3 codebase

Details

Related Changes in Gerrit:
SubjectAuthorRepoBranchLines +/-
airflow3: smtp_default connectionAtsukooperations/deployment-chartsmaster+18 -3
airflow-test-k8s: upgrade to airflow 3Atsukooperations/deployment-chartsmaster+23 -2
Customize query in gerrit

Related Objects
Search...

StatusSubtypeAssignedTask
Restricted Task
 ResolvedBTullisT423243 Upgrade Airflow to 2.11.2
 In ProgressatsukoT423248 Upgrade to Airflow 3.3.x
 ResolvedatsukoT433383 Run test-k8s instance on Airflow 3.3

Event Timeline

atsuko created this task.Jul 28 2026, 2:31 PM
atsuko removed a project: Epic.
atsuko removed atsuko as the assignee of this task.Jul 28 2026, 2:50 PM
atsuko updated the task description. (Show Details)Aug 3 2026, 1:11 PM
atsuko updated the task description. (Show Details)Aug 11 2026, 9:15 AM
atsuko updated the task description. (Show Details)Aug 12 2026, 10:03 AM

There is a task for setting up the Airflow instance for Experiment Platform T416709

atsuko updated the task description. (Show Details)Aug 12 2026, 10:06 AM
Gehel edited projects, added: Data-Platform-SRE (2026-08-07 - 2026-08-28); removed: Data-Platform-SRE.Aug 13 2026, 7:02 AM
atsuko added a comment.EditedAug 13 2026, 1:10 PM

spark_submit_pi_yarn_cluster_mode_via_skein fails with T434786

Gehel edited projects, added: Data-Platform-SRE (2026-08-28 - 2026-09-18); removed: Data-Platform-SRE (2026-08-07 - 2026-08-28).Aug 28 2026, 1:01 PM
Gehel edited projects, added: Data-Platform-SRE (2026-09-18 - 2026-10-09); removed: Data-Platform-SRE (2026-08-28 - 2026-09-18).Fri, Sep 18, 1:20 PM
atsuko changed the task status from Open to In Progress.Wed, Sep 23, 2:23 PM
atsuko claimed this task.
atsuko moved this task from Backlog - project to In Progress on the Data-Platform-SRE (2026-09-18 - 2026-10-09) board.
atsuko updated the task description. (Show Details)Wed, Sep 23, 2:37 PM
gerritbot added a comment.Fri, Sep 25, 11:09 AM

Change #1345077 had a related patch set uploaded (by Atsuko; author: Atsuko):

[operations/deployment-charts@master] airflow-test-k8s: upgrade to airflow 3

https://gerrit.wikimedia.org/r/1345077

gerritbot added a project: Patch-For-Review.Fri, Sep 25, 11:09 AM
atsuko added a comment.EditedFri, Sep 25, 11:17 AM

Upgrade plan

Pre-requisite

For database migration, I first create the devenv on 2.11, then upgrade it in-place with not-yet released airflow-devenv upgrade:

$ airflow-devenv create --dags-folder test_k8s
$ python3 -m airflow_devenv upgrade --dags-folder test_k8s --branch T434896_maintenance_dags --extra-values values-airflow3.yaml

Upgrade

  • Disable all DAGs
  • Remove webserver and scheduler deployments
  • Roll out new values (will need fix)

Verification

  • OIDC
    • Validate user auto-creation
  • RESTExternalTaskSensor

Cleanup

  • Re-enable all DAGs after testing fix for T439576
  • Remove Kerberos-related users
atsuko updated the task description. (Show Details)Fri, Sep 25, 11:20 AM
Stashbot added a comment.Tue, Sep 29, 10:43 AM

Mentioned in SAL (#wikimedia-operations) [2026-09-29T10:43:43Z] <atsukoito> shutting down https://airflow-test-k8s.wikimedia.org for upgrade, https://phabricator.wikimedia.org/T433383#12363530

gerritbot added a comment.Tue, Sep 29, 10:43 AM

Change #1345077 merged by jenkins-bot:

[operations/deployment-charts@master] airflow-test-k8s: upgrade to airflow 3

https://gerrit.wikimedia.org/r/1345077

Maintenance_bot removed a project: Patch-For-Review.Tue, Sep 29, 11:31 AM
atsuko added a comment.EditedTue, Sep 29, 12:33 PM

Based on my understanding, Airflow+flask machinery thinks that the service is HTTP-only, and url_for returns wrong URL

return self.appbuilder.sm.oauth_remotes[provider].authorize_redirect(
    redirect_uri=url_for(  # redirect_uri: http://airflow-test-k8s.wikimedia.org/auth/oauth-authorized/CAS 
        ".oauth_authorized", provider=provider, _external=True
    ),
    state=state.decode("ascii") if isinstance(state, bytes) else state,
)

Fixes are tracked in 1345956

atsuko created subtask T439576: airflow3: data interval length is zero and logical date points to actual time when the dag starts.Tue, Sep 29, 3:30 PM
atsuko removed a subtask: T439576: airflow3: data interval length is zero and logical date points to actual time when the dag starts.EditedTue, Sep 29, 3:43 PM

Found an issue with DAG scheduling T439576 and unmerged changes T439685

atsuko updated the task description. (Show Details)Tue, Sep 29, 4:24 PM
atsuko mentioned this in T423248: Upgrade to Airflow 3.3.x.Tue, Sep 29, 4:40 PM
atsuko added a comment.EditedWed, Sep 30, 10:34 AM

Fix SMTP config: airflow3: smtp_default connection

[2026-09-30T00:51:50.793808Z] INFO - ::group::Post Execute
[2026-09-30T00:51:50.942790Z] ERROR - Failed to send notification (sync): SMTP connection is not found.
[2026-09-30T00:51:50.942935Z] ERROR - Failed to send email notification via SmtpNotifierAirflowException: SMTP connection is not found.
    File /home/app/.local/lib/python3.11/site-packages/airflow/sdk/execution_time/task_runner.py, line 2154 in _send_error_email_notification
    File /home/app/.local/lib/python3.11/site-packages/airflow/sdk/bases/notifier.py, line 138 in __call__
    File /home/app/.local/lib/python3.11/site-packages/airflow/providers/smtp/notifications/smtp.py, line 155 in notify
    File /home/app/.local/lib/python3.11/site-packages/airflow/providers/smtp/hooks/smtp.py, line 79 in __enter__
    File /home/app/.local/lib/python3.11/site-packages/airflow/providers/smtp/hooks/smtp.py, line 119 in get_conn
,AirflowNotFoundException: The conn_id `smtp_default` isn't defined
    File /home/app/.local/lib/python3.11/site-packages/airflow/providers/smtp/hooks/smtp.py, line 117 in get_conn
    File /home/app/.local/lib/python3.11/site-packages/airflow/sdk/bases/hook.py, line 61 in get_connection
    File /home/app/.local/lib/python3.11/site-packages/airflow/sdk/definitions/connection.py, line 296 in get
    File /home/app/.local/lib/python3.11/site-packages/airflow/sdk/execution_time/context.py, line 231 in _get_connection
gerritbot added a comment.Wed, Sep 30, 5:14 PM

Change #1346713 had a related patch set uploaded (by Atsuko; author: Atsuko):

[operations/deployment-charts@master] airflow3: smtp_default connection

https://gerrit.wikimedia.org/r/1346713

gerritbot added a project: Patch-For-Review.Wed, Sep 30, 5:14 PM
atsuko closed this task as Resolved.Wed, Sep 30, 5:18 PM
atsuko moved this task from In Progress to Done on the Data-Platform-SRE (2026-09-18 - 2026-10-09) board.
gerritbot added a comment.Thu, Oct 1, 12:53 PM

Change #1346713 merged by jenkins-bot:

[operations/deployment-charts@master] airflow3: smtp_default connection

https://gerrit.wikimedia.org/r/1346713

Maintenance_bot removed a project: Patch-For-Review.Thu, Oct 1, 1:32 PM
atsuko mentioned this in T435480: Upgrade airflow-main to 3.3.x.Thu, Oct 1, 3:50 PM
Gehel moved this task from Done to Reported on the Data-Platform-SRE (2026-09-18 - 2026-10-09) board.Fri, Oct 2, 7:24 AM
Content licensed under Creative Commons Attribution-ShareAlike (CC BY-SA) 4.0 unless otherwise noted; code licensed under GNU General Public License (GPL) 2.0 or later and other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL · Credits