Page MenuHomePhabricator
Create Task
Maniphest T394389

Migrate the additional dump types from snapshot1016 to Airflow
Closed, ResolvedPublic
Actions

Description

So far, as part of T352650 we have concentrated mainly on the SQL/XML dumps due to their size and complexity.

Now that we have the framework in place to carry out dumps using Mediawiki using Airflow and Kubernetes, then publish the results, we can move on to the other types of dumps.

This ticket will track the work on migrating these types, with possible sub-tickets.

Details

Related Changes in Gerrit:
SubjectRepoBranchLines +/-
Dumps: bump the mediawiki image deployed to the toolbox podoperations/deployment-chartsmaster+3 -4
dse-k8s: Configure limitranges for mediawiki-dumps-legacyoperations/deployment-chartsmaster+21 -0
Bump the mediawiki-dumps-legacy toolbox imageoperations/deployment-chartsmaster+1 -1
dumps: Bump toolbox mediawiki imageoperations/deployment-chartsmaster+1 -1
Adapt dump scripts for running in containersoperations/dumpsmaster+22 -43
mediawiki-dumps-legacy: Bump dumps toolbox image tagoperations/deployment-chartsmaster+1 -1
Add a copy of the dump scripts that are in puppetoperations/dumpsmaster+1 K -0
dumps: Add the addschanges.conf fileoperations/deployment-chartsmaster+36 -1
Customize query in gerrit
Related Changes in GitLab:
TitleReferenceAuthorSource BranchDest Branch
Dumps_v1: Add missing path element in the wikibase sync destinationrepos/data-engineering/airflow-dags!1535btullisfix_wikibase_sync_pathmain
Dumps_v1: Fix the command for the shorturls dumprepos/data-engineering/airflow-dags!1502btullisfix_shorturls_dump_cmdmain
Dumps_v1: Convert the pagetitles dump into two DAGs.repos/data-engineering/airflow-dags!1499btullisfix_mediatitles_dumpmain
(fix) common: differentiate dev instance script environments with k8s devenvsrepos/data-engineering/airflow-dags!1485brouberolhotfix-common-util-root-folder-locationmain
test_k8s/dumps: deprecate schedule_interval in favor for schedulerepos/data-engineering/airflow-dags!1484brouberolT394389main
Add links to the dumps on airflow/k8s documentation in the DAGs documentationrepos/data-engineering/airflow-dags!1483brouberolT394389main
test_k8s/dumps/wikibase/lexemes: fix dump command argumentsrepos/data-engineering/airflow-dags!1470brouberolT394389main
Dumps_v1: Add the shorturls dumprepos/data-engineering/airflow-dags!1462btullisadd_shorturls_dumpmain
Dumps_v1: Fix some sync task parametersrepos/data-engineering/airflow-dags!1449btullisfix_sync_targetsmain
Dumps_v1: Add the cirrussearch dumpsrepos/data-engineering/airflow-dags!1445btullisadd_cirrussearch_dumpsmain
Dumps_v1: Add a DAG to execute the media info dumprepos/data-engineering/airflow-dags!1438btullisdumps_imageinfomain
Dumps_v1: Fix the categories dump job spec fetcherrepos/data-engineering/airflow-dags!1417btullisfix_categories_dumpsmain
Dumps_v1: Add the wikibase dumpsrepos/data-engineering/airflow-dags!1413btulliswikibase_dumpsmain
Dumps_v1: Add the categories RDF dumps DAG.repos/data-engineering/airflow-dags!1408btullisadd_categories_dumpmain
Add the mediawiki content translation dumprepos/data-engineering/airflow-dags!1362btullisadd_more_dumps_dagsmain
Fix the paths of some of the dumps scripts and config filesrepos/releng/release!181btullisfix_dump_script_symlinksmain
Add a DAG to run the pagetitles dumpsrepos/data-engineering/airflow-dags!1337btullisdumps_pagetitlesmain
mediawiki-cli: Add the dcat repository and enable some dumps scriptsrepos/releng/release!171btullisadd_dumps_scriptsmain
dumps: Add the addschanges dumprepos/data-engineering/airflow-dags!1325btullisadds_changes_dagmain
Show related patches Customize query in GitLab

Related Objects
Search...

Event Timeline

There are a very large number of changes, so older changes are hidden. Show Older Changes
BTullis moved this task from Backlog - project to In Progress on the Data-Platform-SRE (2025.05.02 - 2025.05.23) board.May 15 2025, 12:04 PM
BTullis updated the task description. (Show Details)May 15 2025, 2:19 PM

There are some warnings coming from the draft addschanges dump of the following nature:

[2025-05-15, 14:19:42 UTC] {pod_manager.py:477} INFO - [base] [WARNING]: Warning: Undefined array key "x3" in /srv/mediawiki/src/etcd.php on line 136
[2025-05-15, 14:19:43 UTC] {pod_manager.py:477} INFO - [base] Warning: foreach() argument must be of type array|object, null given in /srv/mediawiki/src/etcd.php on line 136

But it doesn't seem to stop the dumps from completing.

gerritbot added a comment.May 16 2025, 4:24 PM

Change #1147028 had a related patch set uploaded (by Btullis; author: Btullis):

[operations/dumps@master] Add a copy of the dump scripts that are in puppet

https://gerrit.wikimedia.org/r/1147028

CodeReviewBot added a comment.May 16 2025, 5:04 PM

btullis opened https://gitlab.wikimedia.org/repos/releng/release/-/merge_requests/171

mediawiki-cli: Add the dcat repository and enable some dumps scripts

CodeReviewBot added a comment.May 16 2025, 5:52 PM

btullis merged https://gitlab.wikimedia.org/repos/data-engineering/airflow-dags/-/merge_requests/1325

dumps: Add the addschanges dump

BTullis updated the task description. (Show Details)May 16 2025, 5:53 PM
gerritbot added a comment.May 19 2025, 9:41 AM

Change #1147028 merged by Btullis:

[operations/dumps@master] Add a copy of the dump scripts that are in puppet

https://gerrit.wikimedia.org/r/1147028

CodeReviewBot added a comment.May 19 2025, 2:04 PM

btullis opened https://gitlab.wikimedia.org/repos/data-engineering/airflow-dags/-/merge_requests/1337

Add a DAG to run the pagetitles dumps

CodeReviewBot added a comment.May 19 2025, 2:52 PM

dancy merged https://gitlab.wikimedia.org/repos/releng/release/-/merge_requests/171

mediawiki-cli: Add the dcat repository and enable some dumps scripts

Stashbot added a comment.May 19 2025, 2:54 PM

Mentioned in SAL (#wikimedia-operations) [2025-05-19T14:54:24Z] <dancy@deploy1003> Started scap sync-world: Updating images for T394389

Stashbot added a comment.May 19 2025, 2:59 PM

Mentioned in SAL (#wikimedia-operations) [2025-05-19T14:59:48Z] <dancy@deploy1003> dancy: Updating images for T394389 synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.

Stashbot added a comment.May 19 2025, 3:07 PM

Mentioned in SAL (#wikimedia-operations) [2025-05-19T15:07:19Z] <dancy@deploy1003> Finished scap sync-world: Updating images for T394389 (duration: 12m 55s)

gerritbot added a comment.May 20 2025, 2:50 PM

Change #1148365 had a related patch set uploaded (by Btullis; author: Btullis):

[operations/deployment-charts@master] mediawiki-dumps-legacy: Bump dumps toolbox image tag

https://gerrit.wikimedia.org/r/1148365

gerritbot added a comment.May 20 2025, 2:55 PM

Change #1148365 merged by jenkins-bot:

[operations/deployment-charts@master] mediawiki-dumps-legacy: Bump dumps toolbox image tag

https://gerrit.wikimedia.org/r/1148365

gerritbot added a comment.May 21 2025, 1:16 PM

Change #1148863 had a related patch set uploaded (by Btullis; author: Btullis):

[operations/dumps@master] Adapt dump scripts for running in containers

https://gerrit.wikimedia.org/r/1148863

gerritbot added a comment.May 21 2025, 3:13 PM

Change #1148863 merged by Btullis:

[operations/dumps@master] Adapt dump scripts for running in containers

https://gerrit.wikimedia.org/r/1148863

Gehel edited projects, added Data-Platform-SRE (2025.05.24 - 2025.06.13); removed Data-Platform-SRE (2025.05.02 - 2025.05.23).May 23 2025, 1:08 PM
Gehel moved this task from Backlog - project to In Progress on the Data-Platform-SRE (2025.05.24 - 2025.06.13) board.
gerritbot added a comment.May 29 2025, 12:38 PM

Change #1152058 had a related patch set uploaded (by Btullis; author: Btullis):

[operations/deployment-charts@master] dumps: Bump toolbox mediawiki image

https://gerrit.wikimedia.org/r/1152058

gerritbot added a comment.May 29 2025, 12:43 PM

Change #1152058 merged by jenkins-bot:

[operations/deployment-charts@master] dumps: Bump toolbox mediawiki image

https://gerrit.wikimedia.org/r/1152058

BTullis added a comment.EditedMay 29 2025, 12:54 PM

Currently running the first manual dump of a cirrussearch section dump in the toolbox pod.

btullis@deploy1003:/srv/deployment-charts/helmfile.d/dse-k8s-services/mediawiki-dumps-legacy$ kubectl exec -it mediawiki-dumps-legacy-toolbox-5475d4b969-2d28p -- bash
Defaulted container "toolbox" out of: toolbox, mediawiki-dumps-legacy-resources-tls-proxy

www-data@mediawiki-dumps-legacy-toolbox-5475d4b969-2d28p:/$ /usr/local/bin/dumpcirrussearch.sh --config /etc/dumps/confs/wikidump.conf.other --dblist /srv/mediawiki/dblists/s1.dblist
Dumping 7000834 documents (7000834 in the index)
BTullis added a comment.May 29 2025, 2:27 PM

It seems to have been killed, possibly with an oom error.
The output was:

Dumping 7000834 documents (7000834 in the index)
	2% done...
	4% done...
	6% done...
	8% done...
	10% done...
	12% done...

We can also see that it used up to 8GB of RAM.

image.png (468×1 px, 104 KB)

However, apart from a directory with today's date and an updated current symlink, no files seem to have been created.

www-data@mediawiki-dumps-legacy-toolbox-5475d4b969-2d28p:/mnt/dumpsdata/otherdumps/cirrussearch$ ls -l
total 1
drwxrwsr-x 2 www-data www-data 0 May 20 15:53 20250520
drwxrwsr-x 2 www-data www-data 0 May 21 08:40 20250521
drwxrwsr-x 2 www-data www-data 0 May 29 12:35 20250529
lrwxrwxrwx 1 www-data www-data 8 May 29 12:35 current -> 20250529
www-data@mediawiki-dumps-legacy-toolbox-5475d4b969-2d28p:/mnt/dumpsdata/otherdumps/cirrussearch$ find
.
./20250520
./20250521
./current
./20250529

I'll have to keep investigating this.

BTullis added a comment.May 29 2025, 2:32 PM

Oh, it seems that it did produce a 14 GB temp files after all. I ran it with the --dryrun option to see the commands that would be executed.

www-data@mediawiki-dumps-legacy-toolbox-5475d4b969-2d28p:/$ /usr/local/bin/dumpcirrussearch.sh --config /etc/dumps/confs/wikidump.conf.other --dblist /srv/mediawiki/dblists/s1.dblist --dryrun
mkdir -p '/mnt/dumpsdata/otherdumps/cirrussearch/20250529'
/mnt/dumpsdata/otherdumps/cirrussearch/20250529/enwiki-20250529-cirrussearch-content.json.gz or /mnt/dumpsdata/xmldatadumps/temp/enwiki-20250529-cirrussearch-content.json.gz already exists, skipping...
/usr/bin/php8.1 '/srv/mediawiki/multiversion/MWScript.php' extensions/CirrusSearch/maintenance/DumpIndex.php --wiki='enwiki' --indexSuffix='general' | /bin/gzip > '/mnt/dumpsdata/xmldatadumps/temp/enwiki-20250529-cirrussearch-general.json.gz'
mv '/mnt/dumpsdata/xmldatadumps/temp/enwiki-20250529-cirrussearch-general.json.gz' '/mnt/dumpsdata/otherdumps/cirrussearch/20250529/enwiki-20250529-cirrussearch-general.json.gz'
www-data@mediawiki-dumps-legacy-toolbox-5475d4b969-2d28p:/$ ls -l /mnt/dumpsdata/xmldatadumps/temp/
total 13962480
drwxrwsr-x 79 www-data www-data          77 Apr 30 09:21 a
drwxrwsr-x 71 www-data www-data          69 Apr 30 09:48 b
-rw-rw-r--  1 www-data www-data 14297579520 May 29 14:29 enwiki-20250529-cirrussearch-content.json.gz
www-data@mediawiki-dumps-legacy-toolbox-5475d4b969-2d28p:/$

So maybe these cirrussearch dumps will just need more RAM.

CodeReviewBot added a comment.May 29 2025, 3:08 PM

btullis opened https://gitlab.wikimedia.org/repos/releng/release/-/merge_requests/181

Fix the paths of some of the dumps scripts and config files

BTullis added a comment.May 29 2025, 3:48 PM

I executed a manual dump run for the content translation dump and it seems to be progressing well.

www-data@mediawiki-dumps-legacy-toolbox-5475d4b969-2d28p:/$ /usr/local/bin/dumpcontentxlation.sh

The files for this look good.

www-data@mediawiki-dumps-legacy-toolbox-5475d4b969-2d28p:/mnt/dumpsdata/otherdumps/contenttranslation/20250529$ pstree -a 105
dumpcontentxlat /usr/local/bin/dumpcontentxlation.sh
  └─php8.1 /srv/mediawiki/multiversion/MWScript.php extensions/ContentTranslation/scripts/dump-corpora.php --wiki enwiki -q --split-at 500 --outputdir /mnt/dumpsdata/otherdumps/contenttranslation/20250529-
      └─sh -c gzip > '/mnt/dumpsdata/otherdumps/contenttranslation/20250529/cx-corpora.en2ar.html.json.gz'
          └─gzip
www-data@mediawiki-dumps-legacy-toolbox-5475d4b969-2d28p:/mnt/dumpsdata/otherdumps/contenttranslation/20250529$

www-data@mediawiki-dumps-legacy-toolbox-5475d4b969-2d28p:/mnt/dumpsdata/otherdumps/contenttranslation/20250529$ ls -lh
total 577M
-rw-rw-r-- 1 www-data www-data 4.0M May 29 15:39 cx-corpora._2af.html.json.gz
-rw-rw-r-- 1 www-data www-data  61M May 29 15:39 cx-corpora.en2af.html.json.gz
-rw-rw-r-- 1 www-data www-data 2.8M May 29 15:39 cx-corpora.en2ak.html.json.gz
-rw-rw-r-- 1 www-data www-data 510M May 29 15:45 cx-corpora.en2ar.html.json.gz
www-data@mediawiki-dumps-legacy-toolbox-5475d4b969-2d28p:/mnt/dumpsdata/otherdumps/contenttranslation/20250529$

Similarly, I did a manual dump of the growth-mentorship dump and this finished quickly, with no errors.

www-data@mediawiki-dumps-legacy-toolbox-5475d4b969-2d28p:~$ /usr/local/bin/dump-growth-mentorship.sh --config /etc/dumps/confs/wikidump.conf.other

The output files in /mnt/dumpsdata/otherdumps/growthmentorship/20250529 look good.

I can start creating DAG files for these two dump types now.

BTullis added a comment.May 29 2025, 3:56 PM

The /usr/local/bin/create-media-per-project-lists.sh dump fails due to a missing /etc/dumps/dblists/globalusage.dblist file.

The shortURLs dumps seems to work with the command:

/usr/bin/python3 /srv/deployment/dumps/xmldumps-backup/onallwikis.py --wiki metawiki --configfile /etc/dumps/confs/wikidump.conf.dumps:monitor  --filenameformat 'shorturls-{d}.gz' --outdir '/mnt/dumpsdata/otherdumps/shorturls' --script extensions/UrlShortener/maintenance/dumpURLs.php 'compress.zlib://{DIR}'

The generated file is 79 MB in size.

www-data@mediawiki-dumps-legacy-toolbox-5475d4b969-2d28p:/mnt/dumpsdata/otherdumps/shorturls$ ls -lh
total 79M
-rw-r--r-- 1 www-data www-data 79M May 29 15:53 shorturls-20250529.gz
www-data@mediawiki-dumps-legacy-toolbox-5475d4b969-2d28p:/mnt/dumpsdata/otherdumps/shorturls$
CodeReviewBot added a comment.May 29 2025, 5:16 PM

btullis opened https://gitlab.wikimedia.org/repos/data-engineering/airflow-dags/-/merge_requests/1362

Add the mediawiki content translation dump

CodeReviewBot added a comment.Jun 2 2025, 1:34 PM

btullis merged https://gitlab.wikimedia.org/repos/data-engineering/airflow-dags/-/merge_requests/1362

Add the mediawiki content translation dump

BTullis updated the task description. (Show Details)Jun 2 2025, 1:34 PM
CodeReviewBot added a comment.Jun 10 2025, 11:34 AM

cgoubert merged https://gitlab.wikimedia.org/repos/releng/release/-/merge_requests/181

Fix the paths of some of the dumps scripts and config files

Stashbot added a comment.Jun 10 2025, 11:35 AM

Mentioned in SAL (#wikimedia-operations) [2025-06-10T11:35:32Z] <cgoubert@deploy1003> Started scap sync-world: mediawiki-cli: Fix the paths of some of the dumps scripts and config files - T394389

Stashbot added a comment.Jun 10 2025, 11:44 AM

Mentioned in SAL (#wikimedia-operations) [2025-06-10T11:44:22Z] <cgoubert@deploy1003> Finished scap sync-world: mediawiki-cli: Fix the paths of some of the dumps scripts and config files - T394389 (duration: 08m 49s)

Gehel edited projects, added Data-Platform-SRE (2025.06.13 - 2025.07.04); removed Data-Platform-SRE (2025.05.24 - 2025.06.13).Jun 13 2025, 8:49 AM
Gehel moved this task from Backlog - project to In Progress on the Data-Platform-SRE (2025.06.13 - 2025.07.04) board.
gerritbot added a comment.Jun 13 2025, 3:51 PM

Change #1156862 had a related patch set uploaded (by Btullis; author: Btullis):

[operations/deployment-charts@master] Bump the mediawiki-dumps-legacy toolbox image

https://gerrit.wikimedia.org/r/1156862

gerritbot added a comment.Jun 13 2025, 3:56 PM

Change #1156862 merged by jenkins-bot:

[operations/deployment-charts@master] Bump the mediawiki-dumps-legacy toolbox image

https://gerrit.wikimedia.org/r/1156862

BTullis added a comment.Jun 13 2025, 4:06 PM

I'm now able to start a test run of the CategoriesRDF dump in the toolbox, like this:

/usr/local/bin/dumpcategoriesrdf.sh --config /etc/dumps/confs/wikidump.conf.other --list /srv/mediawiki/dblists/categories-rdf.dblist

The process tree looks OK.

www-data@mediawiki-dumps-legacy-toolbox-5fc88c9c76-cxk7t:/$ pstree -a 7
bash
  └─dumpcategoriesr /usr/local/bin/dumpcategoriesrdf.sh --config /etc/dumps/confs/wikidump.conf.other --list /srv/mediawiki/dblists/categories-rdf.dblist
      └─dumpcategoriesr /usr/local/bin/dumpcategoriesrdf.sh --config /etc/dumps/confs/wikidump.conf.other --list /srv/mediawiki/dblists/categories-rdf.dblist
          ├─gzip
          └─php8.1 /srv/mediawiki/multiversion/MWScript.php maintenance/dumpCategoriesAsRdf.php --wiki=arwiki --format=ttl

Files are being generated.

www-data@mediawiki-dumps-legacy-toolbox-5fc88c9c76-cxk7t:/$ find /mnt/dumpsdata/otherdumps/categoriesrdf/
/mnt/dumpsdata/otherdumps/categoriesrdf/
/mnt/dumpsdata/otherdumps/categoriesrdf/lastdump
/mnt/dumpsdata/otherdumps/categoriesrdf/lastdump/alswiki-categories.last
/mnt/dumpsdata/otherdumps/categoriesrdf/lastdump/acewiki-categories.last
/mnt/dumpsdata/otherdumps/categoriesrdf/lastdump/amwiktionary-categories.last
/mnt/dumpsdata/otherdumps/categoriesrdf/lastdump/adywiki-categories.last
/mnt/dumpsdata/otherdumps/categoriesrdf/lastdump/abwiki-categories.last
/mnt/dumpsdata/otherdumps/categoriesrdf/lastdump/anwiki-categories.last
/mnt/dumpsdata/otherdumps/categoriesrdf/lastdump/afwiki-categories.last
/mnt/dumpsdata/otherdumps/categoriesrdf/lastdump/anwiktionary-categories.last
/mnt/dumpsdata/otherdumps/categoriesrdf/lastdump/afwikiquote-categories.last
/mnt/dumpsdata/otherdumps/categoriesrdf/lastdump/amwikimedia-categories.last
/mnt/dumpsdata/otherdumps/categoriesrdf/lastdump/afwikibooks-categories.last
/mnt/dumpsdata/otherdumps/categoriesrdf/lastdump/akwiki-categories.last
/mnt/dumpsdata/otherdumps/categoriesrdf/lastdump/afwiktionary-categories.last
/mnt/dumpsdata/otherdumps/categoriesrdf/lastdump/angwiktionary-categories.last
/mnt/dumpsdata/otherdumps/categoriesrdf/lastdump/amwiki-categories.last
/mnt/dumpsdata/otherdumps/categoriesrdf/lastdump/arcwiki-categories.last
/mnt/dumpsdata/otherdumps/categoriesrdf/lastdump/angwiki-categories.last
/mnt/dumpsdata/otherdumps/categoriesrdf/20250613
/mnt/dumpsdata/otherdumps/categoriesrdf/20250613/adywiki-20250613-categories.ttl.gz
/mnt/dumpsdata/otherdumps/categoriesrdf/20250613/acewiki-20250613-categories.ttl.gz
/mnt/dumpsdata/otherdumps/categoriesrdf/20250613/angwiktionary-20250613-categories.ttl.gz
/mnt/dumpsdata/otherdumps/categoriesrdf/20250613/afwikibooks-20250613-categories.ttl.gz
/mnt/dumpsdata/otherdumps/categoriesrdf/20250613/afwiktionary-20250613-categories.ttl.gz
/mnt/dumpsdata/otherdumps/categoriesrdf/20250613/anwiki-20250613-categories.ttl.gz
/mnt/dumpsdata/otherdumps/categoriesrdf/20250613/akwiki-20250613-categories.ttl.gz
/mnt/dumpsdata/otherdumps/categoriesrdf/20250613/afwikiquote-20250613-categories.ttl.gz
/mnt/dumpsdata/otherdumps/categoriesrdf/20250613/arcwiki-20250613-categories.ttl.gz
/mnt/dumpsdata/otherdumps/categoriesrdf/20250613/anwiktionary-20250613-categories.ttl.gz
/mnt/dumpsdata/otherdumps/categoriesrdf/20250613/amwikimedia-20250613-categories.ttl.gz
/mnt/dumpsdata/otherdumps/categoriesrdf/20250613/amwiki-20250613-categories.ttl.gz
/mnt/dumpsdata/otherdumps/categoriesrdf/20250613/alswiki-20250613-categories.ttl.gz
/mnt/dumpsdata/otherdumps/categoriesrdf/20250613/afwiki-20250613-categories.ttl.gz
/mnt/dumpsdata/otherdumps/categoriesrdf/20250613/arwiki-20250613-categories.ttl.gz
/mnt/dumpsdata/otherdumps/categoriesrdf/20250613/angwiki-20250613-categories.ttl.gz
/mnt/dumpsdata/otherdumps/categoriesrdf/20250613/abwiki-20250613-categories.ttl.gz
/mnt/dumpsdata/otherdumps/categoriesrdf/20250613/amwiktionary-20250613-categories.ttl.gz
CodeReviewBot added a comment.Jun 16 2025, 8:32 AM

btullis opened https://gitlab.wikimedia.org/repos/data-engineering/airflow-dags/-/merge_requests/1408

Dumps_v1: Add the categories RDF dumps DAG.

BTullis updated the task description. (Show Details)Jun 16 2025, 8:54 AM
BTullis updated the task description. (Show Details)Jun 16 2025, 11:29 AM
CodeReviewBot added a comment.Jun 16 2025, 3:35 PM

btullis opened https://gitlab.wikimedia.org/repos/data-engineering/airflow-dags/-/merge_requests/1413

Dumps_v1: Add the wikibase dumps

BTullis updated the task description. (Show Details)Jun 16 2025, 11:41 PM
CodeReviewBot added a comment.Jun 17 2025, 12:22 AM

btullis merged https://gitlab.wikimedia.org/repos/data-engineering/airflow-dags/-/merge_requests/1408

Dumps_v1: Add the categories RDF dumps DAG.

CodeReviewBot added a comment.Jun 17 2025, 10:30 AM

btullis opened https://gitlab.wikimedia.org/repos/data-engineering/airflow-dags/-/merge_requests/1417

Dumps_v1: Fix the categories dump job spec fetcher

CodeReviewBot added a comment.Jun 17 2025, 10:38 AM

btullis merged https://gitlab.wikimedia.org/repos/data-engineering/airflow-dags/-/merge_requests/1413

Dumps_v1: Add the wikibase dumps

CodeReviewBot added a comment.Jun 17 2025, 10:39 AM

btullis merged https://gitlab.wikimedia.org/repos/data-engineering/airflow-dags/-/merge_requests/1417

Dumps_v1: Fix the categories dump job spec fetcher

BTullis updated the task description. (Show Details)Jun 17 2025, 10:41 AM
BTullis updated the task description. (Show Details)Jun 23 2025, 10:43 AM
CodeReviewBot added a comment.Jun 23 2025, 11:22 AM

btullis opened https://gitlab.wikimedia.org/repos/data-engineering/airflow-dags/-/merge_requests/1438

Dumps_v1: Add a DAG to execute the mediawiki dump

BTullis updated the task description. (Show Details)Jun 23 2025, 11:22 AM
CodeReviewBot added a comment.Jun 23 2025, 12:53 PM

btullis merged https://gitlab.wikimedia.org/repos/data-engineering/airflow-dags/-/merge_requests/1337

Add a DAG to run the pagetitles dumps

CodeReviewBot added a comment.Jun 23 2025, 1:43 PM

btullis opened https://gitlab.wikimedia.org/repos/data-engineering/airflow-dags/-/merge_requests/1445

Draft: Dumps_v1: Add the cirrussearch dumps

BTullis updated the task description. (Show Details)Jun 23 2025, 1:43 PM
CodeReviewBot added a comment.Jun 23 2025, 6:41 PM

btullis opened https://gitlab.wikimedia.org/repos/data-engineering/airflow-dags/-/merge_requests/1449

Dumps_v1: Fix some sync task parameters

BTullis updated the task description. (Show Details)Jun 23 2025, 9:33 PM
CodeReviewBot added a comment.Jun 24 2025, 10:30 AM

btullis merged https://gitlab.wikimedia.org/repos/data-engineering/airflow-dags/-/merge_requests/1449

Dumps_v1: Fix some sync task parameters

CodeReviewBot added a comment.Jun 24 2025, 10:32 AM

btullis closed https://gitlab.wikimedia.org/repos/data-engineering/airflow-dags/-/merge_requests/1438

Dumps_v1: Add a DAG to execute the media info dump

gerritbot added a comment.Jun 24 2025, 11:33 AM

Change #1163341 had a related patch set uploaded (by Btullis; author: Btullis):

[operations/deployment-charts@master] dse-k8s: Increase maximum container/pod size for mediawiki-dumps-legacy

https://gerrit.wikimedia.org/r/1163341

gerritbot added a comment.Jun 24 2025, 1:41 PM

Change #1163341 merged by jenkins-bot:

[operations/deployment-charts@master] dse-k8s: Configure limitranges for mediawiki-dumps-legacy

https://gerrit.wikimedia.org/r/1163341

BTullis updated the task description. (Show Details)Jun 24 2025, 2:34 PM
CodeReviewBot added a comment.Jun 24 2025, 3:38 PM

btullis merged https://gitlab.wikimedia.org/repos/data-engineering/airflow-dags/-/merge_requests/1445

Dumps_v1: Add the cirrussearch dumps

Maintenance_bot removed a project: Patch-For-Review.Jun 24 2025, 4:31 PM
CodeReviewBot added a project: Patch-For-Review.Jun 25 2025, 1:38 PM

btullis opened https://gitlab.wikimedia.org/repos/data-engineering/airflow-dags/-/merge_requests/1462

Dumps_v1: Add the shorturls dump

BTullis updated the task description. (Show Details)Jun 25 2025, 2:23 PM
CodeReviewBot added a comment.Jun 25 2025, 2:43 PM

btullis opened https://gitlab.wikimedia.org/repos/data-engineering/airflow-dags/-/merge_requests/1465

Dumps_v1: Miscellaneous schedule_interval fixes

CodeReviewBot added a comment.Jun 26 2025, 8:02 AM

brouberol opened https://gitlab.wikimedia.org/repos/data-engineering/airflow-dags/-/merge_requests/1470

test_k8s/dumps/wikibase/lexemes: fix dump command

CodeReviewBot added a comment.Jun 26 2025, 8:31 AM

brouberol merged https://gitlab.wikimedia.org/repos/data-engineering/airflow-dags/-/merge_requests/1470

test_k8s/dumps/wikibase/lexemes: fix dump command arguments

CodeReviewBot added a comment.Jun 26 2025, 8:32 AM

btullis merged https://gitlab.wikimedia.org/repos/data-engineering/airflow-dags/-/merge_requests/1462

Dumps_v1: Add the shorturls dump

CodeReviewBot added a comment.Jun 26 2025, 8:32 AM

btullis merged https://gitlab.wikimedia.org/repos/data-engineering/airflow-dags/-/merge_requests/1465

Dumps_v1: Miscellaneous schedule_interval fixes

Maintenance_bot removed a project: Patch-For-Review.Jun 26 2025, 9:31 AM
BTullis closed this task as Resolved.Jun 26 2025, 10:52 AM
BTullis updated the task description. (Show Details)
BTullis moved this task from In Progress to Done on the Data-Platform-SRE (2025.06.13 - 2025.07.04) board.
CodeReviewBot added a project: Patch-For-Review.Jun 27 2025, 8:23 AM

brouberol opened https://gitlab.wikimedia.org/repos/data-engineering/airflow-dags/-/merge_requests/1483

Add links to the dumps on airflow/k8s documentation in the DAGs documentation

CodeReviewBot added a comment.Jun 27 2025, 8:35 AM

brouberol merged https://gitlab.wikimedia.org/repos/data-engineering/airflow-dags/-/merge_requests/1483

Add links to the dumps on airflow/k8s documentation in the DAGs documentation

CodeReviewBot added a comment.Jun 27 2025, 8:41 AM

brouberol opened https://gitlab.wikimedia.org/repos/data-engineering/airflow-dags/-/merge_requests/1484

test_k8s/dumps: deprecate schedule_interval in favor for schedule

CodeReviewBot added a comment.Jun 27 2025, 8:51 AM

brouberol opened https://gitlab.wikimedia.org/repos/data-engineering/airflow-dags/-/merge_requests/1485

(fix) common: differentiate dev instance script environments with k8s devenvs

CodeReviewBot added a comment.Jun 27 2025, 8:56 AM

brouberol merged https://gitlab.wikimedia.org/repos/data-engineering/airflow-dags/-/merge_requests/1485

(fix) common: differentiate dev instance script environments with k8s devenvs

CodeReviewBot added a comment.Jun 27 2025, 2:24 PM

brouberol merged https://gitlab.wikimedia.org/repos/data-engineering/airflow-dags/-/merge_requests/1484

test_k8s/dumps: deprecate schedule_interval in favor for schedule

Maintenance_bot removed a project: Patch-For-Review.Jun 27 2025, 2:31 PM
Gehel moved this task from Done to Reported on the Data-Platform-SRE (2025.06.13 - 2025.07.04) board.Jun 27 2025, 2:46 PM
CodeReviewBot added a project: Patch-For-Review.Jun 30 2025, 10:15 PM

btullis opened https://gitlab.wikimedia.org/repos/data-engineering/airflow-dags/-/merge_requests/1499

Dumps_v1: Convert the pagetitles dump into two DAGs.

CodeReviewBot added a comment.Jul 1 2025, 9:19 AM

btullis merged https://gitlab.wikimedia.org/repos/data-engineering/airflow-dags/-/merge_requests/1499

Dumps_v1: Convert the pagetitles dump into two DAGs.

Maintenance_bot removed a project: Patch-For-Review.Jul 1 2025, 9:31 AM
CodeReviewBot added a project: Patch-For-Review.Jul 1 2025, 12:24 PM

btullis opened https://gitlab.wikimedia.org/repos/data-engineering/airflow-dags/-/merge_requests/1502

Dumps_v1: Fix the command for the shorturls dump

BTullis mentioned this in T398756: No wikidata dumps this week (20250630).Jul 7 2025, 9:37 AM
CodeReviewBot added a comment.Jul 7 2025, 9:54 AM

btullis opened https://gitlab.wikimedia.org/repos/data-engineering/airflow-dags/-/merge_requests/1535

Dumps_v1: Add missing path element in the wikibase sync destination

CodeReviewBot added a comment.Jul 7 2025, 10:35 AM

btullis merged https://gitlab.wikimedia.org/repos/data-engineering/airflow-dags/-/merge_requests/1535

Dumps_v1: Add missing path element in the wikibase sync destination

CodeReviewBot added a comment.Jul 8 2025, 8:25 AM

btullis closed https://gitlab.wikimedia.org/repos/data-engineering/airflow-dags/-/merge_requests/1502

Dumps_v1: Fix the command for the shorturls dump

Maintenance_bot removed a project: Patch-For-Review.Jul 8 2025, 8:30 AM
BTullis mentioned this in T399866: Migrate growth-mentorship dump to Airflow.Jul 17 2025, 3:47 PM
gerritbot added a comment.Jul 24 2025, 11:59 AM

Change #1172285 had a related patch set uploaded (by Btullis; author: Btullis):

[operations/deployment-charts@master] Dumps: bump the mediawiki image deployed to the toolbox pod

https://gerrit.wikimedia.org/r/1172285

gerritbot added a project: Patch-For-Review.Jul 24 2025, 11:59 AM
gerritbot added a comment.Jul 24 2025, 12:09 PM

Change #1172285 merged by jenkins-bot:

[operations/deployment-charts@master] Dumps: bump the mediawiki image deployed to the toolbox pod

https://gerrit.wikimedia.org/r/1172285

Content licensed under Creative Commons Attribution-ShareAlike (CC BY-SA) 4.0 unless otherwise noted; code licensed under GNU General Public License (GPL) 2.0 or later and other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL · Credits