| BTullis | |
| May 8 2025, 11:19 AM |
| F59773081: image.png | |
| May 8 2025, 11:19 AM |
Our current method of running the dumps v1 DAGs always creates sync tasks for every wiki.
However, in the case of private wikis, there is no /mnt/dumpsdata/xmldatadumps/${wiki} directory, so the sync tasks fail.
We should determine how to handle this. Options include:
| Title | Reference | Author | Source Branch | Dest Branch | |
|---|---|---|---|---|---|
| Skip sync tasks for private wikis | repos/data-engineering/airflow-dags!1299 | btullis | skip_private_wiki_dump_syncs | main |
| Status | Subtype | Assigned | Task | ||
|---|---|---|---|---|---|
| Open | None | T88728 Improve Wikimedia dumping infrastructure | |||
| Resolved | BTullis | T352650 WE 5.4 KR - Hypothesis 5.4.4 - Q3 FY24/25 - Migrate current-generation dumps to run on kubernetes | |||
| Resolved | brouberol | T388378 Orchestrate dumps v1 from an airflow instance | |||
| Resolved | BTullis | T393687 Skip sync tasks for private wikis, or allow them to fail gracefully |
My first thought is that we have a managed list of private wikis available.
Perhaps we could add a new task to cat this list and save it as an xcom, then check against this list before creating a sync task.
www-data@mediawiki-dumps-legacy-toolbox-c465d7598-96d97:~$ cat /srv/mediawiki/dblists/private.dblist # NOTE: This file is automatically generated. Do not edit it directly, run 'composer manage-dblist' instead. advisorswiki arbcom_cswiki arbcom_dewiki arbcom_enwiki arbcom_fiwiki arbcom_itwiki arbcom_nlwiki arbcom_ruwiki arbcom_zhwiki auditcomwiki boardgovcomwiki boardwiki chairwiki chapcomwiki checkuserwiki collabwiki ecwikimedia electcomwiki execwiki fdcwiki grantswiki id_internalwikimedia iegcomwiki ilwikimedia internalwiki legalteamwiki movementroleswiki noboard_chapterswikimedia officewiki ombudsmenwiki otrs_wikiwiki projectcomwiki searchcomwiki spcomwiki stewardwiki sysop_itwiki sysop_plwiki techconductwiki transitionteamwiki u4cwiki wg_enwiki wikimaniateamwiki www-data@mediawiki-dumps-legacy-toolbox-c465d7598-96d97:~$
That's a good question. I suppose it's because it's would represent a change in behaviour.
These private wikis are currently dumped, but the files just live on the dumpsdata servers and do not get synced to the clouddumps servers.
Perhaps changing the behaviour so that they are not dumped at all would be a good idea anyway.
I've asled the question in Slack, too.
I'll start by just modifying the DAG to skip the sync jobs, since that represents a no-op to the current dumps behaviour.
If we want to skip the dumping part as well, then I can come back to that.
btullis opened https://gitlab.wikimedia.org/repos/data-engineering/airflow-dags/-/merge_requests/1299
Skip sync tasks for private wikis
btullis merged https://gitlab.wikimedia.org/repos/data-engineering/airflow-dags/-/merge_requests/1299
Skip sync tasks for private wikis
I don't know of any product reasons to create dumps of the private wikis. We could always be surprised, but for now I agree that they should be skipped.