Page MenuHomePhabricator
Create Task
Maniphest T404084

Move the dumps_v1 DAGs from the Airflow test_k8s instance to a dedicated instance
Closed, ResolvedPublic
Actions

Assigned To
brouberol
Authored By
BTullis
Sep 9 2025, 1:29 PM
Project Tags
Referenced Files
F95063156: Screenshot 2026-07-21 at 18.05.52.png
Tue, Jul 21, 4:05 PM
F95050364: Screenshot 2026-07-21 at 16.42.55.png
Tue, Jul 21, 2:43 PM
F95049654: image.png
Tue, Jul 21, 2:39 PM
F95047538: Screenshot 2026-07-21 at 16.25.05.png
Tue, Jul 21, 2:25 PM
Subscribers
Aklapper
brouberol
BTullis
Gehel
WMDE-leszek
xcollazo

Description

Whilst carrying out the work on T352650, we developed the DAGs on the kest-k8s Airflow instance.

The reason for this initial configuration is that, at that time, we were using novel features such as the KubernetesPodOperator that was not available on all instances.

Now that the Dumps_v1 processes are in production, we should migrate the DAGs from the test-k8s instance to another instance.
The purpose of the test-k8s instance is for running integration test DAGs and for testing new features, so it should not really be running production pipelines.

There will be certain required changes:

Details

Related Changes in Gerrit:
Show related patches Customize query in gerrit
Related Changes in GitLab:
TitleReferenceAuthorSource BranchDest Branch
Move all dumps DAGs from test_k8s to dumpsrepos/data-engineering/airflow-dags!2410brouberolT404084main
Customize query in GitLab

Related Objects
Search...

Event Timeline

BTullis created this task.Sep 9 2025, 1:29 PM
Restricted Application added a subscriber: Aklapper. · View Herald TranscriptSep 9 2025, 1:29 PM
Ottomata moved this task from Incoming (new tickets) to Tag with Radar on the Data-Engineering board.Sep 17 2025, 6:41 PM
Ottomata added a project: Data-Engineering-Radar.
Gehel triaged this task as Medium priority.Sep 23 2025, 1:20 PM
Gehel edited projects, added: Data-Platform-SRE (2025.09.05 - 2025.09.26); removed: Data-Platform-SRE.
Gehel moved this task from Backlog - project to Backlog - operations on the Data-Platform-SRE (2025.09.05 - 2025.09.26) board.
Gehel subscribed.

Do we want to have a dedicated Airflow instance for Dumps?

Gehel edited projects, added: Data-Platform-SRE (2025.09.26 - 2025.10.17); removed: Data-Platform-SRE (2025.09.05 - 2025.09.26).Sep 26 2025, 1:51 PM
Gehel moved this task from Backlog - project to Backlog - operations on the Data-Platform-SRE (2025.09.26 - 2025.10.17) board.
Gehel added a project: Essential-Work.Sep 30 2025, 1:03 PM
xcollazo mentioned this in T406371: Bump max mapped task for all Airflow instances to 1200.Oct 3 2025, 6:28 PM
Gehel edited projects, added: Data-Platform-SRE (2025.10.17 - 2025.11.07); removed: Data-Platform-SRE (2025.09.26 - 2025.10.17).Oct 17 2025, 8:50 AM
Gehel moved this task from Backlog - project to Backlog - operations on the Data-Platform-SRE (2025.10.17 - 2025.11.07) board.Oct 17 2025, 8:52 AM
WMDE-leszek subscribed.Oct 20 2025, 2:10 PM
Gehel edited projects, added: Data-Platform-SRE (2025.11.07 - 2025.11.28); removed: Data-Platform-SRE (2025.10.17 - 2025.11.07).Nov 7 2025, 1:32 PM
Gehel moved this task from Backlog - project to Backlog - operations on the Data-Platform-SRE (2025.11.07 - 2025.11.28) board.
Gehel edited projects, added: Data-Platform-SRE (2026.01.05 - 2026.01.23); removed: Data-Platform-SRE (2025.11.07 - 2025.11.28).Jan 8 2026, 1:15 PM
Gehel moved this task from Backlog - project to Backlog - operations on the Data-Platform-SRE (2026.01.05 - 2026.01.23) board.
Gehel edited projects, added: Data-Platform-SRE (2026.01.23 - 2026.02.13); removed: Data-Platform-SRE (2026.01.05 - 2026.01.23).Jan 23 2026, 10:04 AM
Gehel moved this task from Backlog - project to Backlog - operations on the Data-Platform-SRE (2026.01.23 - 2026.02.13) board.
Gehel edited projects, added: Data-Platform-SRE (2026-02-13 - 2026-03-06); removed: Data-Platform-SRE (2026.01.23 - 2026.02.13).Feb 16 2026, 3:36 PM
Gehel moved this task from Backlog - project to Backlog - operations on the Data-Platform-SRE (2026-02-13 - 2026-03-06) board.
Gehel edited projects, added: Data-Platform-SRE (2026-03-06 - 2026-03-27); removed: Data-Platform-SRE (2026-02-13 - 2026-03-06).Mar 6 2026, 2:03 PM
Gehel moved this task from Backlog - project to Backlog - operations on the Data-Platform-SRE (2026-03-06 - 2026-03-27) board.
Gehel edited projects, added: Data-Platform-SRE (2026-03-27 - 2026-04-17); removed: Data-Platform-SRE (2026-03-06 - 2026-03-27).Mar 27 2026, 9:49 AM
Gehel moved this task from Backlog - project to Backlog - operations on the Data-Platform-SRE (2026-03-27 - 2026-04-17) board.
Gehel edited projects, added: Data-Platform-SRE (2026-04-24 - 2026-05-15); removed: Data-Platform-SRE (2026-03-27 - 2026-04-17).Apr 24 2026, 9:58 AM
Gehel moved this task from Backlog - project to Backlog - operations on the Data-Platform-SRE (2026-04-24 - 2026-05-15) board.
Gehel edited projects, added: Data-Platform-SRE (2026-06-05 - 2026-06-26); removed: Data-Platform-SRE (2026-04-24 - 2026-05-15).Jun 5 2026, 1:25 PM
Gehel moved this task from Backlog - project to Backlog - operations on the Data-Platform-SRE (2026-06-05 - 2026-06-26) board.
Gehel edited projects, added: Data-Platform-SRE (2026-07-03 - 2026-07-31); removed: Data-Platform-SRE (2026-06-05 - 2026-06-26).Jul 3 2026, 9:50 AM
Gehel moved this task from Backlog - project to Backlog - operations on the Data-Platform-SRE (2026-07-03 - 2026-07-31) board.
brouberol added a comment.Tue, Jul 21, 7:33 AM

@Gehel as discussed in previous team meetings, yes, we do. I think airlfow-main is already pretty crowded, especially so at the start of the month, which lines up with when the dumps v1 DAGs start.

brouberol renamed this task from Move the dumps_v1 DAGs from the Airflow test_k8s instance to the main instance to Move the dumps_v1 DAGs from the Airflow test_k8s instance to a dedicated instance.Tue, Jul 21, 7:33 AM
brouberol claimed this task.Tue, Jul 21, 7:33 AM
gerritbot added a comment.Tue, Jul 21, 7:36 AM

Change #1312978 had a related patch set uploaded (by Brouberol; author: Brouberol):

[operations/dns@master] Define the airflow-dumps public and internal domains

https://gerrit.wikimedia.org/r/1312978

gerritbot added a project: Patch-For-Review.Tue, Jul 21, 7:36 AM
gerritbot added a comment.Tue, Jul 21, 7:41 AM

Change #1312981 had a related patch set uploaded (by Brouberol; author: Brouberol):

[operations/puppet@production] deployment_server: define the airflow-dumps kubeconfigs in dse-k8s-eqiad

https://gerrit.wikimedia.org/r/1312981

gerritbot added a comment.Tue, Jul 21, 7:41 AM

Change #1312982 had a related patch set uploaded (by Brouberol; author: Brouberol):

[operations/puppet@production] idp: define the airflow-dumps service

https://gerrit.wikimedia.org/r/1312982

gerritbot added a comment.Tue, Jul 21, 7:41 AM

Change #1312983 had a related patch set uploaded (by Brouberol; author: Brouberol):

[operations/puppet@production] trafficserver: define the airflow-dumps public->internal mapping

https://gerrit.wikimedia.org/r/1312983

gerritbot added a comment.Tue, Jul 21, 7:41 AM

Change #1312984 had a related patch set uploaded (by Brouberol; author: Brouberol):

[operations/puppet@production] cache/text: enable caching for airflow-dumps.w.o

https://gerrit.wikimedia.org/r/1312984

brouberol changed the task status from Open to In Progress.Tue, Jul 21, 7:41 AM
brouberol moved this task from Backlog - operations to In Progress on the Data-Platform-SRE (2026-07-03 - 2026-07-31) board.
gerritbot added a comment.Tue, Jul 21, 7:54 AM

Change #1312989 had a related patch set uploaded (by Brouberol; author: Brouberol):

[operations/deployment-charts@master] dse-k8s-eqiad: define the airflow-dumps namespace

https://gerrit.wikimedia.org/r/1312989

gerritbot added a comment.Tue, Jul 21, 7:54 AM

Change #1312990 had a related patch set uploaded (by Brouberol; author: Brouberol):

[operations/deployment-charts@master] dse-k8s-eqiad: add airflow-dumps to the required operator tenant namespaces

https://gerrit.wikimedia.org/r/1312990

gerritbot added a comment.Tue, Jul 21, 7:54 AM

Change #1312991 had a related patch set uploaded (by Brouberol; author: Brouberol):

[operations/deployment-charts@master] dse-k8s-eqiad/mediawiki-dumps-legacy: enable pod creation RBAC from airflow-dumps

https://gerrit.wikimedia.org/r/1312991

gerritbot added a comment.Tue, Jul 21, 7:54 AM

Change #1312992 had a related patch set uploaded (by Brouberol; author: Brouberol):

[operations/deployment-charts@master] dse-k8s-eqiad: define the airflow-test-k8s helmfile and values

https://gerrit.wikimedia.org/r/1312992

gerritbot added a comment.Tue, Jul 21, 7:54 AM

Change #1312993 had a related patch set uploaded (by Brouberol; author: Brouberol):

[operations/deployment-charts@master] dse-k8s-eqiad/airflow-test-k8s: remove any configuration override related to dumps

https://gerrit.wikimedia.org/r/1312993

gerritbot added a comment.Tue, Jul 21, 8:28 AM

Change #1313051 had a related patch set uploaded (by Brouberol; author: Brouberol):

[operations/deployment-charts@master] dse-k8s-eqiad: define the postgreql-airflow-dumps helmfile

https://gerrit.wikimedia.org/r/1313051

CodeReviewBot added a comment.Tue, Jul 21, 8:55 AM

brouberol opened https://gitlab.wikimedia.org/repos/data-engineering/airflow-dags/-/merge_requests/2410

Move all dumps DAGs from test_k8s to dumps

gerritbot added a comment.Tue, Jul 21, 1:15 PM

Change #1312978 merged by Brouberol:

[operations/dns@master] Define the airflow-dumps public and internal domains

https://gerrit.wikimedia.org/r/1312978

gerritbot added a comment.Tue, Jul 21, 1:29 PM

Change #1312981 merged by Brouberol:

[operations/puppet@production] deployment_server: define the airflow-dumps kubeconfigs in dse-k8s-eqiad

https://gerrit.wikimedia.org/r/1312981

gerritbot added a comment.Tue, Jul 21, 1:40 PM

Change #1312989 merged by jenkins-bot:

[operations/deployment-charts@master] dse-k8s-eqiad: define the airflow-dumps namespace

https://gerrit.wikimedia.org/r/1312989

gerritbot added a comment.Tue, Jul 21, 1:41 PM

Change #1312990 merged by jenkins-bot:

[operations/deployment-charts@master] dse-k8s-eqiad: add airflow-dumps to the required operator tenant namespaces

https://gerrit.wikimedia.org/r/1312990

brouberol moved this task from In Progress to To Be Deployed on the Data-Platform-SRE (2026-07-03 - 2026-07-31) board.Tue, Jul 21, 1:42 PM
gerritbot added a comment.Tue, Jul 21, 1:49 PM

Change #1312991 merged by jenkins-bot:

[operations/deployment-charts@master] dse-k8s-eqiad/mediawiki-dumps-legacy: enable pod creation RBAC from airflow-dumps

https://gerrit.wikimedia.org/r/1312991

gerritbot added a comment.Tue, Jul 21, 1:49 PM

Change #1313051 merged by jenkins-bot:

[operations/deployment-charts@master] dse-k8s-eqiad: define the postgreql-airflow-dumps helmfile

https://gerrit.wikimedia.org/r/1313051

gerritbot added a comment.Tue, Jul 21, 1:49 PM

Change #1312992 merged by jenkins-bot:

[operations/deployment-charts@master] dse-k8s-eqiad: define the airflow-test-k8s helmfile and values

https://gerrit.wikimedia.org/r/1312992

brouberol added a comment.Tue, Jul 21, 2:25 PM

Screenshot 2026-07-21 at 16.25.05.png (2,820×592 px, 166 KB)
We see that both namespaces have the same quotas for now. The airflow-test-k8s quotas will be reduced after the migration.

brouberol updated the task description. (Show Details)Tue, Jul 21, 2:29 PM
brouberol added a comment.Tue, Jul 21, 2:39 PM

image.png (2,350×628 px, 119 KB)

As part of the migration of the dump DAGs to their own instance, I'm freezing the git ref airflow-test-k8s uses to https://gitlab.wikimedia.org/repos/data-engineering/airflow-dags/-/tree/1e2fcc4e5740b9c3480b6cd07d9046613ea7aae2, aka right before we removed the dump DAGs from the test_k8s folder.

I'm doing this because we still have running DAGs at the moment, and I'd like for them to finish before we can unpause DAGs over in airflow-dumps.

brouberol added a comment.Tue, Jul 21, 2:41 PM

Reminder to leave mediawiki_cirrussearch_dump paused over in airflow-dumps.

brouberol added a comment.Tue, Jul 21, 2:43 PM

I'm leaving our wiki monitoring DAG, the running ones, as well as the DAGs triggered when in need of syncing the ongoing wikibase dumps.

Screenshot 2026-07-21 at 16.42.55.png (2,942×1,808 px, 644 KB)
All the other dump DAGs are paused in test-k8s.

gerritbot added a comment.Tue, Jul 21, 3:01 PM

Change #1312982 merged by Brouberol:

[operations/puppet@production] idp: define the airflow-dumps service

https://gerrit.wikimedia.org/r/1312982

CodeReviewBot added a comment.Tue, Jul 21, 3:09 PM

brouberol merged https://gitlab.wikimedia.org/repos/data-engineering/airflow-dags/-/merge_requests/2410

Move all dumps DAGs from test_k8s to dumps

brouberol added a comment.Tue, Jul 21, 3:22 PM
NAME                                                 READY   STATUS    RESTARTS   AGE
airflow-envoy-797cd57b94-pjcm4                       1/1     Running   0          35s
airflow-gitsync-7f6987fd5f-tc95d                     1/1     Running   0          35s
airflow-hadoop-shell-86988fc89f-g5c8g                1/1     Running   0          35s
airflow-kerberos-7b97bb9745-8fb9n                    1/1     Running   0          35s
airflow-scheduler-58df9ffc65-hnpk6                   1/1     Running   0          36s
airflow-statsd-568fcc976c-pp62w                      1/1     Running   0          8m40s
airflow-task-shell-65fd6cd9dd-rj7f9                  1/1     Running   0          35s
airflow-webserver-8565466ff-sxdsk                    2/2     Running   0          36s
postgresql-airflow-dumps-1                           1/1     Running   0          77m
postgresql-airflow-dumps-2                           1/1     Running   0          77m
postgresql-airflow-dumps-pooler-rw-d747c5fcd-6qxmb   1/1     Running   0          78m
postgresql-airflow-dumps-pooler-rw-d747c5fcd-rdg84   1/1     Running   0          78m
postgresql-airflow-dumps-pooler-rw-d747c5fcd-ttq92   1/1     Running   0          78m
gerritbot added a comment.Tue, Jul 21, 3:23 PM

Change #1312983 merged by Brouberol:

[operations/puppet@production] trafficserver: define the airflow-dumps public->internal mapping

https://gerrit.wikimedia.org/r/1312983

gerritbot added a comment.Tue, Jul 21, 4:00 PM

Change #1312984 merged by Brouberol:

[operations/puppet@production] cache/text: enable caching for airflow-dumps.w.o

https://gerrit.wikimedia.org/r/1312984

brouberol added a comment.Tue, Jul 21, 4:05 PM

Screenshot 2026-07-21 at 18.05.52.png (1,388×1,405 px, 368 KB)

brouberol added a comment.Tue, Jul 21, 6:08 PM

https://airflow-dumps.wikimedia.org is now live! I will unpause mediawiki_commons_mediainfo_rdf_dump, mediawiki_wikidata_all_json_dump and mediawiki_wikidata_all_rdf_dump once they complete in test-k8s, so probably tomorrow. After which, I'll remove all dump-related DAGs from test-k8s altogether.

BTullis added a parent task: T398436: Future improvements to the Dumps on Airflow configuration - tracking task.Wed, Jul 22, 5:59 AM
brouberol added a comment.Wed, Jul 22, 7:33 AM

I updated https://wikitech.wikimedia.org/wiki/Dumps/Airflow and https://wikitech.wikimedia.org/wiki/Dumps/Airflow/Operations

brouberol updated the task description. (Show Details)Wed, Jul 22, 3:22 PM
gerritbot added a comment.Thu, Jul 23, 3:27 PM

Change #1312993 merged by Brouberol:

[operations/deployment-charts@master] dse-k8s-eqiad/airflow-test-k8s: remove any configuration override related to dumps

https://gerrit.wikimedia.org/r/1312993

brouberol updated the task description. (Show Details)Thu, Jul 23, 3:30 PM
brouberol closed this task as Resolved.Thu, Jul 23, 3:31 PM
brouberol moved this task from To Be Deployed to Done on the Data-Platform-SRE (2026-07-03 - 2026-07-31) board.
Maintenance_bot removed a project: Patch-For-Review.Thu, Jul 23, 3:31 PM
Gehel moved this task from Done to Reported on the Data-Platform-SRE (2026-07-03 - 2026-07-31) board.Fri, Aug 7, 7:53 AM
Content licensed under Creative Commons Attribution-ShareAlike (CC BY-SA) 4.0 unless otherwise noted; code licensed under GNU General Public License (GPL) 2.0 or later and other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL · Credits