Publish Dumps 2 to dumps.wikimedia.org and provide only monthly dumps
Closed, ResolvedPublic
Actions

Assigned To

Authored By

	Ahoelzl
	Mon, Jan 12, 10:25 PM

Description

On 1/12/2026 in a WMF-wide group meeting it has been decided to move forward with the publication of the Dumps 2 XML artifacts on dumps.wikimedia.org

The XML artifacts from the new Dumps 2 process will be published on dumps.wikimedia.org next to the existing Dumps 1 artifacts.
The publication cadence of both Dumps 1 and Dumps 2 artifacts will be reduced to monthly, i.e. mid-month partial updates will no longer be provided.
The Dumps 1 XML artifacts will be marked as deprecated. Note that this deprecation affects only the XML content artifacts. All other SQL dumps of various database tables will continue, but on a monthly basis.

The changes will be accompanied with a communication to the community that the new Dumps 2 XML files are more stable and hence should be preferred for downloading.

Tasks

publish Dumps 2 XML artifacts to dump.wikimedia.org monthly
Turn off mid-month Dumps 1 artifacts (Note that this include all dump assets that we typically trigger on the 20th of the month.)
Post changes to dumps mailing list - https://lists.wikimedia.org/hyperkitty/list/xmldatadumps-l@lists.wikimedia.org/thread/E6D5EU4PMSTSOI2J7A46HJ3YW2W554CS/

Details

Related Changes in Gerrit:

Subject	Repo	Branch	Lines +/-
analytics: refinery: add data purge for File Export.	operations/puppet	production	+25 -3
dumps: Update index.html file to reflect XML dumps deprecation	operations/deployment-charts	master	+6 -16
dumps: Fix MW Content File Export. Remove already absented file def.	operations/puppet	production	+1 -11
dumps: Release the new MW Content File Export. Deprecate legacy XML dumps.	operations/puppet	production	+86 -182

Customize query in gerrit

Related Objects
Search...

Status	Assigned	Task
Open	None	T366752 Dumps 2.0 Phase III: Production level dumps (SDS 1.2)
In Progress	xcollazo	T384382 Production-level file export (aka dump) of MW Content in XML
Resolved	xcollazo	T414389 Publish Dumps 2 to dumps.wikimedia.org and provide only monthly dumps

Event Timeline

Ahoelzl created this task.Mon, Jan 12, 10:25 PM

Restricted Application added a subscriber: Aklapper. · View Herald TranscriptMon, Jan 12, 10:25 PM

Ahoelzl added a parent task: T384382: Production-level file export (aka dump) of MW Content in XML.Mon, Jan 12, 10:26 PM

xcollazo opened https://gitlab.wikimedia.org/repos/data-engineering/airflow-dags/-/merge_requests/1924

mw file export: remove mid-month file export DAG

xcollazo changed the task status from Open to In Progress.Fri, Jan 16, 3:50 PM

xcollazo claimed this task.

xcollazo triaged this task as High priority.

xcollazo merged https://gitlab.wikimedia.org/repos/data-engineering/airflow-dags/-/merge_requests/1924

mw file export: remove mid-month file export DAG

Maintenance_bot removed a project: Patch-For-Review.Fri, Jan 16, 4:32 PM

In T414389#11529535, @CodeReviewBot wrote:

xcollazo merged https://gitlab.wikimedia.org/repos/data-engineering/airflow-dags/-/merge_requests/1924

mw file export: remove mid-month file export DAG

I first paused the mid-month File Expor DAG (aka Dumps2) and then merged in these changes ^^^^.

I've now paused all Dumpsv1 DAGs tagged as partial-dump from the test-k8s Airflow instance that runs them.

Following up with a MR for the test-k8s instance shortly.

xcollazo opened https://gitlab.wikimedia.org/repos/data-engineering/airflow-dags/-/merge_requests/1928

test_k8s: dumps: remove partial dump DAGs.

xcollazo moved this task from Next Up to In progress on the Data-Engineering (Q3 FY25/26 January 1st - March 31th) board.Fri, Jan 16, 7:51 PM

BTullis mentioned this in T401048: Dump indexes do not agree.Mon, Jan 19, 9:38 AM

xcollazo updated the task description. (Show Details)Tue, Jan 20, 3:43 PM

xcollazo merged https://gitlab.wikimedia.org/repos/data-engineering/airflow-dags/-/merge_requests/1928

test_k8s: dumps: remove partial dump DAGs.

xcollazo updated the task description. (Show Details)Tue, Jan 20, 3:45 PM

Maintenance_bot removed a project: Patch-For-Review.Tue, Jan 20, 4:32 PM

Hi, is this why the mid-month dump run (20260120) has not started?

In T414389#11540482, @Poslovitch wrote:

Hi, is this why the mid-month dump run (20260120) has not started?

Yes. We will make an announcement shortly.

Change #1199783 had a related patch set uploaded (by Xcollazo; author: Xcollazo):

[operations/puppet@production] dumps: Release the new MW Content File Export. Deprecate legacy XML dumps.

https://gerrit.wikimedia.org/r/1199783

gerritbot added a project: Patch-For-Review.Thu, Jan 22, 7:08 PM

Change #1199783 merged by Brouberol:

[operations/puppet@production] dumps: Release the new MW Content File Export. Deprecate legacy XML dumps.

https://gerrit.wikimedia.org/r/1199783

Change #1230956 had a related patch set uploaded (by Xcollazo; author: Xcollazo):

[operations/puppet@production] dumps: Fix MW Content File Export. Remove already absented file def.

https://gerrit.wikimedia.org/r/1230956

Change #1230956 merged by Brouberol:

[operations/puppet@production] dumps: Fix MW Content File Export. Remove already absented file def.

https://gerrit.wikimedia.org/r/1230956

Maintenance_bot removed a project: Patch-For-Review.Fri, Jan 23, 4:31 PM

Change #1230965 had a related patch set uploaded (by Xcollazo; author: Xcollazo):

[operations/deployment-charts@master] dumps: Update index.html file to reflect XML dumps deprecation

https://gerrit.wikimedia.org/r/1230965

Change #1230965 merged by Brouberol:

[operations/deployment-charts@master] dumps: Update index.html file to reflect XML dumps deprecation

https://gerrit.wikimedia.org/r/1230965

Maintenance_bot removed a project: Patch-For-Review.Fri, Jan 23, 5:32 PM

Two comments:

I'm a little perplexed as to why the fact that "mid-month partial updates will no longer be provided" wasn't announced before 20 January. It isn't even in Tech News 2026-05 (the 26 Jan edition). Sure, perhaps you want to announce the deprecation of Dumps 1 in favour of Dumps 2 "when the time is right" - no problem with that. But the cessation of mid-month dumps broke people's workflows (including mine). This specific aspect should have been announced in advance of it occurring. I'm boldly adding the User-notice project so that the Tech News editors can decide what to do with this.
Dumps 1 also contains SQL dumps of various database tables. At this time, Dumps 2 doesn't contain an equivalent. Can we assume this aspect of Dumps 1 is not deprecated? If so, could this please be made clearer in the messaging?

In T414389#11551519, @TTO wrote:

Two comments:

I'm a little perplexed as to why the fact that "mid-month partial updates will no longer be provided" wasn't announced before 20 January. It isn't even in Tech News 2026-05 (the 26 Jan edition). Sure, perhaps you want to announce the deprecation of Dumps 1 in favour of Dumps 2 "when the time is right" - no problem with that. But the cessation of mid-month dumps broke people's workflows (including mine). This specific aspect should have been announced in advance of it occurring. I'm boldly adding the User-notice project so that the Tech News editors can decide what to do with this.

We should have announced it before removing, you are right.

Dumps 1 also contains SQL dumps of various database tables. At this time, Dumps 2 doesn't contain an equivalent. Can we assume this aspect of Dumps 1 is not deprecated?

Correct, this decision only affects the XML content artifacts. All other SQL dumps of various database tables will continue to be done on a monthly basis.

If so, could this please be made clearer in the messaging?

I will now modify the description of this ticket to make this clear. If you have suggestions for how to communicate this more clearly at https://dumps.wikimedia.org/ we'll take them as well.

xcollazo updated the task description. (Show Details)Tue, Jan 27, 7:31 PM

Change #1233836 had a related patch set uploaded (by Xcollazo; author: Xcollazo):

[operations/puppet@production] analytics: refinery: add data purge for File Export.

https://gerrit.wikimedia.org/r/1233836

gerritbot added a project: Patch-For-Review.Tue, Jan 27, 9:35 PM

xcollazo updated the task description. (Show Details)Tue, Jan 27, 9:36 PM

STei-WMF moved this task from To Triage to In current Tech/News draft on the User-notice board.Thu, Jan 29, 2:01 PM

xcollazo updated the task description. (Show Details)Fri, Jan 30, 11:30 PM

Do Dumps 2 files, which correspond to a revision range of a single page, have correct names?

I can see that, for example, on https://dumps.wikimedia.org/other/mediawiki_content_history/enwiki/2026-01-01/xml/bzip2/ there's a file enwiki-2026-01-01-p10701605r123221892r123221892.xml.bz2 which apparently contains revisions in range 123221892 - 123221892 (both ends are the same). The same can be found in the documentation, which I think is counterintuitive and likely suggests an issue with interpretation of the file name.

I don't see a combined file for mediawiki_content_current like pages-meta-current.xml.bz2 that exists in the Dumps 1 directory. I have a workflow that utilizes the combined file: TemplateParameterBot (enwiki)
I also don't see an equivalent to pages-articles.xml.bz2 for "Articles, templates, media/file descriptions, and primary meta-pages". I have multiple workflows that utilize this file for multiple wikis: TemplateParameterBot (several wikis), WikidataClassBrowser (wikidata), CheckWiki (several wikis). These could be converted to use the full content dumps, they will take longer to run though.

In T414389#11571338, @Msz2001 wrote:

Do Dumps 2 files, which correspond to a revision range of a single page, have correct names?

I can see that, for example, on https://dumps.wikimedia.org/other/mediawiki_content_history/enwiki/2026-01-01/xml/bzip2/ there's a file enwiki-2026-01-01-p10701605r123221892r123221892.xml.bz2 which apparently contains revisions in range 123221892 - 123221892 (both ends are the same). The same can be found in the documentation, which I think is counterintuitive and likely suggests an issue with interpretation of the file name.

Thanks for the report, opened T416176: File Export files with revision ranges do not show end revision in filename to track this.

In T414389#11572089, @Bamyers99 wrote:

I don't see a combined file for mediawiki_content_current like pages-meta-current.xml.bz2 that exists in the Dumps 1 directory. I have a workflow that utilizes the combined file: TemplateParameterBot (enwiki)

Right, the new system is indeed simpler. There is no combined files. There is also no recompression into .7z.

I also don't see an equivalent to pages-articles.xml.bz2 for "Articles, templates, media/file descriptions, and primary meta-pages". I have multiple workflows that utilize this file for multiple wikis: TemplateParameterBot (several wikis), WikidataClassBrowser (wikidata), CheckWiki (several wikis). These could be converted to use the full content dumps, they will take longer to run though.

There is no plan to remove the pages-articles job from the legacy system. The only XML artifacts that are deprecated are the jobs related to pages-meta-current and pages-meta-history. I do see the need to be more specific about what is deprecated and what is not, will do so via T416180: Modify legacy dumps to specify what artifacts are deprecated.