Page MenuHomePhabricator
Create Task
Maniphest T380624

Migrate the airflow-platform-eng scheduler to Kubernetes
Closed, ResolvedPublic
Actions

Description

Following the procedure outlined on https://wikitech.wikimedia.org/wiki/Data_Platform/Systems/Airflow/Kubernetes/Operations#Migrate_the_scheduler_and_kerberos_components_to_Kubernetes

  • Create the kerberos principals and the base64 representation of the instance keytab
  • Create the S3 user
  • add the secrets
  • Create the S3 bucket
  • Sync all the scheduler and DAG task logs to S3
  • stop all the airflow systemd services and sync the logs again
  • deploy the helmfile.d changes (to be deployed)
  • deploy the puppet changes ervices_ensure: absent (to be deployed
  • Verify Dags are running

Details

Related Changes in Gerrit:
SubjectRepoBranchLines +/-
airflow-platform-eng: set up all services except systemdoperations/puppetproduction+1 -0
airflow-product-eng: migrate scheduler and db to k8soperations/deployment-chartsmaster+26 -36
Customize query in gerrit

Related Objects
Search...

Event Timeline

brouberol created this task.Nov 22 2024, 3:54 PM
Gehel triaged this task as High priority.Nov 25 2024, 1:33 PM
mfossati subscribed.Dec 16 2024, 4:53 PM
Gehel moved this task from Incoming to Scratch on the Data-Platform-SRE board.Jan 8 2025, 8:39 AM
Gehel edited projects, added Data-Platform-SRE (2024.11.30 - 2024.12.20); removed Data-Platform-SRE.
Gehel edited projects, added Data-Platform-SRE (2025.01.11 - 2025.01.31); removed Data-Platform-SRE (2024.11.30 - 2024.12.20).Jan 10 2025, 10:39 AM
Gehel edited projects, added Data-Platform-SRE (2025.02.10 - 2025.02.28); removed Data-Platform-SRE (2025.01.11 - 2025.01.31).Feb 11 2025, 10:05 AM
• Stevemunene claimed this task.Feb 13 2025, 2:32 PM
Gehel edited projects, added Data-Platform-SRE (2025.03.01 - 2025.03.21); removed Data-Platform-SRE (2025.02.10 - 2025.02.28).Feb 28 2025, 1:27 PM
• Stevemunene moved this task from Backlog - project to Backlog - operations on the Data-Platform-SRE (2025.03.01 - 2025.03.21) board.Mar 4 2025, 4:45 PM
Gehel moved this task from Backlog - operations to In Progress on the Data-Platform-SRE (2025.03.01 - 2025.03.21) board.Mar 4 2025, 4:47 PM
gerritbot added a comment.Mar 7 2025, 1:35 PM

Change #1125424 had a related patch set uploaded (by Stevemunene; author: Stevemunene):

[operations/deployment-charts@master] airflow-product-eng: migrate scheduler and db to k8s

https://gerrit.wikimedia.org/r/1125424

gerritbot added a project: Patch-For-Review.Mar 7 2025, 1:35 PM
• Stevemunene moved this task from In Progress to Blocked/Waiting on the Data-Platform-SRE (2025.03.01 - 2025.03.21) board.Mar 10 2025, 1:17 PM

Instance owners have requested us to postpone our move to dse-k8s until they are done testing and running currently blocked pipelines related to https://phabricator.wikimedia.org/T387470 and https://phabricator.wikimedia.org/T386255

Gehel edited projects, added Data-Platform-SRE (2025.03.22 - 2025.04.11); removed Data-Platform-SRE (2025.03.01 - 2025.03.21).Mar 21 2025, 9:56 AM
Gehel moved this task from Backlog - project to Blocked/Waiting on the Data-Platform-SRE (2025.03.22 - 2025.04.11) board.
xcollazo mentioned this in T390727: Canary failure on airflow platform_eng intsance after migrating to Kubernetes.Apr 1 2025, 2:25 PM
• Stevemunene moved this task from Blocked/Waiting to In Progress on the Data-Platform-SRE (2025.03.22 - 2025.04.11) board.Apr 2 2025, 9:42 AM
• Stevemunene updated the task description. (Show Details)Apr 2 2025, 11:45 AM
gerritbot added a comment.Apr 2 2025, 11:59 AM

Change #1133367 had a related patch set uploaded (by Stevemunene; author: Stevemunene):

[operations/puppet@production] airflow-platform-eng: set up all services except systemd

https://gerrit.wikimedia.org/r/1133367

gerritbot added a comment.Apr 2 2025, 1:00 PM

Change #1125424 merged by jenkins-bot:

[operations/deployment-charts@master] airflow-product-eng: migrate scheduler and db to k8s

https://gerrit.wikimedia.org/r/1125424

gerritbot added a comment.Apr 2 2025, 2:14 PM

Change #1133367 merged by Stevemunene:

[operations/puppet@production] airflow-platform-eng: set up all services except systemd

https://gerrit.wikimedia.org/r/1133367

Maintenance_bot removed a project: Patch-For-Review.Apr 2 2025, 2:31 PM
• Stevemunene updated the task description. (Show Details)Apr 2 2025, 2:43 PM
• Stevemunene added a subtask: T390880: [L] Verify all Airflow DAGs after migration to Kubernetes.Apr 3 2025, 8:07 AM

Some DAGS need to be verified and are being worked on here https://phabricator.wikimedia.org/T390880 we shall close once verified

• Stevemunene mentioned this in T390880: [L] Verify all Airflow DAGs after migration to Kubernetes.Apr 3 2025, 8:08 AM
mfossati changed the status of subtask T390880: [L] Verify all Airflow DAGs after migration to Kubernetes from Open to In Progress.Apr 3 2025, 9:08 AM
mfossati mentioned this in T339129: [Spike] Periodically regenerate various variable data sets/files.Apr 7 2025, 5:51 PM
• Stevemunene closed this task as Resolved.Apr 8 2025, 3:41 PM
• Stevemunene updated the task description. (Show Details)
• Stevemunene moved this task from In Progress to Done on the Data-Platform-SRE (2025.03.22 - 2025.04.11) board.

Scheduler is moved and all the DAGS have been verified to be running as expected

brouberol moved this task from Done to Reported on the Data-Platform-SRE (2025.03.22 - 2025.04.11) board.Apr 10 2025, 1:02 PM
brouberol moved this task from Reported to Done on the Data-Platform-SRE (2025.03.22 - 2025.04.11) board.Apr 10 2025, 1:05 PM
Gehel moved this task from Done to Reported on the Data-Platform-SRE (2025.03.22 - 2025.04.11) board.Apr 11 2025, 9:15 AM
mfossati closed subtask T390880: [L] Verify all Airflow DAGs after migration to Kubernetes as Resolved.Apr 28 2025, 4:40 PM
Content licensed under Creative Commons Attribution-ShareAlike (CC BY-SA) 4.0 unless otherwise noted; code licensed under GNU General Public License (GPL) 2.0 or later and other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL · Credits