Page MenuHomePhabricator
Create Task
Maniphest T380620

Migrate the airflow-research scheduler to Kubernetes
Closed, ResolvedPublic
Actions

Details

Related Changes in Gerrit:
SubjectRepoBranchLines +/-
airflow: enable the injection of custom config files in the worker podoperations/deployment-chartsmaster+86 -1
airflow: revert to having the scheduling using an http checkoperations/deployment-chartsmaster+6 -2
airflow-research: fix typooperations/puppetproduction+1 -1
airflow-research: disable the airflow systemd servicesoperations/puppetproduction+19 -19
airflow: enable the injection of custom config files in the worker podoperations/deployment-chartsmaster+62 -1
Re-enable the airflow-resarch scheduler probesoperations/deployment-chartsmaster+0 -4
airflow-research: temporarily disable liveness/readiness probesoperations/deployment-chartsmaster+4 -0
Add missing postgresql networkpolicies to airflow-researchoperations/deployment-chartsmaster+5 -1
Migrate the airflow-research scheduler to dse-k8soperations/deployment-chartsmaster+4 -12
Create k8s tokens for airflow-research on dse-k8soperations/puppetproduction+8 -0
Customize query in gerrit

Related Objects
Search...

Event Timeline

brouberol created this task.Nov 22 2024, 3:53 PM
brouberol removed a subscriber: JMeybohm.
Gehel triaged this task as High priority.Nov 25 2024, 1:33 PM
BTullis claimed this task.Jan 7 2025, 10:26 AM
BTullis edited projects, added Data-Platform-SRE (2024.11.30 - 2024.12.20); removed Data-Platform-SRE.
BTullis subscribed.

I'll carry out this Airflow scheduler migration, as per the instructions here: https://wikitech.wikimedia.org/wiki/Data_Platform/Systems/Airflow/Kubernetes/Operations#Migrate_the_scheduler_and_kerberos_components_to_Kubernetes

gerritbot added a comment.Jan 7 2025, 10:42 AM

Change #1108725 had a related patch set uploaded (by Btullis; author: Btullis):

[operations/puppet@production] Create k8s tokens for airflow-research on dse-k8s

https://gerrit.wikimedia.org/r/1108725

gerritbot added a project: Patch-For-Review.Jan 7 2025, 10:42 AM

Change #1108725 abandoned by Btullis:

[operations/puppet@production] Create k8s tokens for airflow-research on dse-k8s

Reason:

Already added.

https://gerrit.wikimedia.org/r/1108725

Maintenance_bot removed a project: Patch-For-Review.Jan 7 2025, 11:30 AM
BTullis added subscribers: kzimmerman, leila, fkaelin and 8 others.Jan 7 2025, 12:15 PM
gerritbot added a comment.Jan 8 2025, 10:52 AM

Change #1109041 had a related patch set uploaded (by Btullis; author: Btullis):

[operations/deployment-charts@master] Migrate the airflow-research scheduler to dse-k8s

https://gerrit.wikimedia.org/r/1109041

gerritbot added a project: Patch-For-Review.Jan 8 2025, 10:52 AM
ops-monitoring-bot added a comment.Jan 8 2025, 11:08 AM

Icinga downtime and Alertmanager silence (ID=e7f372ee-26b5-4320-9944-59eb99c3d211) set by btullis@cumin1002 for 2:00:00 on 1 host(s) and their services with reason: Migrating to kubernetes

an-airflow1002.eqiad.wmnet
gerritbot added a comment.Jan 8 2025, 12:06 PM

Change #1109041 merged by jenkins-bot:

[operations/deployment-charts@master] Migrate the airflow-research scheduler to dse-k8s

https://gerrit.wikimedia.org/r/1109041

Maintenance_bot removed a project: Patch-For-Review.Jan 8 2025, 12:30 PM
gerritbot added a comment.Jan 8 2025, 12:35 PM

Change #1109056 had a related patch set uploaded (by Btullis; author: Btullis):

[operations/deployment-charts@master] Add missing postgresql networkpolicies to airflow-research

https://gerrit.wikimedia.org/r/1109056

gerritbot added a project: Patch-For-Review.Jan 8 2025, 12:35 PM
gerritbot added a comment.Jan 8 2025, 12:38 PM

Change #1109056 merged by jenkins-bot:

[operations/deployment-charts@master] Add missing postgresql networkpolicies to airflow-research

https://gerrit.wikimedia.org/r/1109056

Maintenance_bot removed a project: Patch-For-Review.Jan 8 2025, 1:30 PM
gerritbot added a comment.Jan 8 2025, 1:41 PM

Change #1109069 had a related patch set uploaded (by Brouberol; author: Brouberol):

[operations/deployment-charts@master] airflow-research: temporarily disable liveness/readiness probes

https://gerrit.wikimedia.org/r/1109069

gerritbot added a project: Patch-For-Review.Jan 8 2025, 1:41 PM
gerritbot added a comment.Jan 8 2025, 2:19 PM

Change #1109069 merged by jenkins-bot:

[operations/deployment-charts@master] airflow-research: temporarily disable liveness/readiness probes

https://gerrit.wikimedia.org/r/1109069

Maintenance_bot removed a project: Patch-For-Review.Jan 8 2025, 2:31 PM
gerritbot added a comment.Jan 8 2025, 3:53 PM

Change #1109110 had a related patch set uploaded (by Btullis; author: Btullis):

[operations/deployment-charts@master] Re-enable the airflow-resarch scheduler probes

https://gerrit.wikimedia.org/r/1109110

gerritbot added a project: Patch-For-Review.Jan 8 2025, 3:53 PM
gerritbot added a comment.Jan 8 2025, 3:57 PM

Change #1109110 merged by jenkins-bot:

[operations/deployment-charts@master] Re-enable the airflow-resarch scheduler probes

https://gerrit.wikimedia.org/r/1109110

Maintenance_bot removed a project: Patch-For-Review.Jan 8 2025, 4:30 PM
BTullis closed this task as Resolved.Jan 8 2025, 5:27 PM
BTullis moved this task from Backlog - project to Done on the Data-Platform-SRE (2024.11.30 - 2024.12.20) board.
Gehel moved this task from Done to Reported on the Data-Platform-SRE (2024.11.30 - 2024.12.20) board.Jan 10 2025, 8:56 AM
gerritbot added a comment.Jan 10 2025, 10:24 AM

Change #1109670 had a related patch set uploaded (by Brouberol; author: Brouberol):

[operations/deployment-charts@master] airflow: enable the injection of custom config files in the worker pod

https://gerrit.wikimedia.org/r/1109670

gerritbot added a project: Patch-For-Review.Jan 10 2025, 10:24 AM
gerritbot added a comment.Jan 10 2025, 1:47 PM

Change #1109705 had a related patch set uploaded (by Brouberol; author: Brouberol):

[operations/deployment-charts@master] airflow: enable the injection of custom config files in the worker pod

https://gerrit.wikimedia.org/r/1109705

gerritbot added a comment.Jan 10 2025, 1:49 PM

Change #1109670 abandoned by Brouberol:

[operations/deployment-charts@master] airflow: enable the injection of custom config files in the worker pod

Reason:

Superseded by https://gerrit.wikimedia.org/r/c/operations/deployment-charts/+/1109705 due to a change in change-id

https://gerrit.wikimedia.org/r/1109670

gerritbot added a comment.Jan 10 2025, 2:33 PM

Change #1109714 had a related patch set uploaded (by Brouberol; author: Brouberol):

[operations/puppet@production] airflow-research: disable the airflow systemd services

https://gerrit.wikimedia.org/r/1109714

gerritbot added a comment.Jan 14 2025, 1:34 PM

Change #1109714 merged by Brouberol:

[operations/puppet@production] airflow-research: disable the airflow systemd services

https://gerrit.wikimedia.org/r/1109714

gerritbot added a comment.Jan 14 2025, 1:40 PM

Change #1111223 had a related patch set uploaded (by Brouberol; author: Brouberol):

[operations/puppet@production] airflow-research: fix typo

https://gerrit.wikimedia.org/r/1111223

gerritbot added a comment.Jan 14 2025, 1:41 PM

Change #1111223 merged by Brouberol:

[operations/puppet@production] airflow-research: fix typo

https://gerrit.wikimedia.org/r/1111223

gerritbot added a comment.Jan 14 2025, 2:51 PM

Change #1111245 had a related patch set uploaded (by Brouberol; author: Brouberol):

[operations/deployment-charts@master] airflow: revert to having the scheduling using an http check

https://gerrit.wikimedia.org/r/1111245

gerritbot added a comment.Jan 14 2025, 2:59 PM

Change #1111245 merged by Brouberol:

[operations/deployment-charts@master] airflow: revert to having the scheduling using an http check

https://gerrit.wikimedia.org/r/1111245

gerritbot added a comment.Jan 21 2025, 3:57 PM

Change #1109705 merged by Brouberol:

[operations/deployment-charts@master] airflow: enable the injection of custom config files in the worker pod

https://gerrit.wikimedia.org/r/1109705

Maintenance_bot removed a project: Patch-For-Review.Jan 21 2025, 4:30 PM
Content licensed under Creative Commons Attribution-ShareAlike (CC BY-SA) 4.0 unless otherwise noted; code licensed under GNU General Public License (GPL) 2.0 or later and other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL · Credits