Production-level file export (aka dump) of MW Content in XML
Open, In Progress, HighPublic
Actions

Assigned To

Authored By

	xcollazo
	Jan 21 2025, 9:15 PM

Description

On T335862: Implement job to generate Dump XML files, we developed PoC code to dump wiki content in XML form from data lake table wmf_content.mediawiki_content_history.

We now want to get this code to production level with a set of tasks aimed at hardening, testing, and integrating this mechanism.

This is done to support FY2025 Q1[[ https://app.asana.com/1/3758245663860/project/1210776716741007/overview/1210776805319899 | SDS 1.2.1 ]]:

If we migrate the XML Dumps process from the current 'Dumps 1' infrastructure to a data pipeline that leverages the MediaWiki Content Pipelines we will be able to guarantee SLOs and turn off the 'Dumps 1'-based XML export.

Related Objects
Search...

Status	Assigned	Task
Open	None	T366752 Dumps 2.0 Phase III: Production level dumps (SDS 1.2)
In Progress	xcollazo	T384382 Production-level file export (aka dump) of MW Content in XML
Resolved	pfischer	T381016 Optimize XML Dump code to be able to handle wikis from simplewiki to enwiki
Resolved	pfischer	T384383 Modify XML dumping code to be able to do 'partial' dumps
Resolved	pfischer	T384385 Refactor code to use new table and column names
Open	None	T384392 Put together a DPE Deep Dive session on learnings from Dumps 2 XML generation code
Resolved	xcollazo	T384945 Modify code to dump all slots
Declined	BTullis	T384500 Consider writing Spark files to Ceph (S3) instead of Hadoop
Duplicate	None	T386211 Modify wmf_content.mediawiki_content_history_v1 to include slot origin
Resolved	Antoine_Quhen	T388793 When doing ADD COLUMN to a struct under a map, Iceberg fails to SELECT it
Resolved	xcollazo	T396031 Remove code and tables related to deprecated mediawiki_wikitext_history and mediawiki_wikitext_current
		Restricted Task
Open	None	T379410 Go over tasks in #dumps-generation and figure what makes sense to fix in Dumps 2.0 (DPE-Mediawiki-Content)
		Restricted Task
Open	None	T400632 Missing/inconsistent page_redirect_target field for redirects in Mediawiki content current v1 dumps
Resolved	xcollazo	T384381 Airflow jobs to do monthly XML dumps
Open	xcollazo	T395139 MediaWiki Content History alerts too much for minor reconcile issues
Resolved	xcollazo	T401022 Implement the data layout, UI, and documentation for the XML file export
Resolved	xcollazo	T402209 Modify XML dumping code to support multiple compression algorithms
Resolved	xcollazo	T402229 Compare outputs between XML File Export and MediaWiki's Special:Export
Resolved	xcollazo	T405641 Adapt MW Content pipelines to the removal of upstream revision.rev_sha1
Resolved	Antoine_Quhen	T405944 Rewrite wmf_content.mediawiki_content_*_v1 tables with a new column for origin_rev_id
Resolved	xcollazo	T407237 Wait till november wmf_raw.mediawiki_slots sqoop table is available, and apply origin_rev_id fix to mw_content tables
Resolved	xcollazo	T407649 Compare the exported content between File Export and DumpV1
Resolved	xcollazo	T414389 Publish Dumps 2 to dumps.wikimedia.org and provide only monthly dumps

Event Timeline

xcollazo created this task.Jan 21 2025, 9:15 PM

Restricted Application added a subscriber: Aklapper. · View Herald TranscriptJan 21 2025, 9:15 PM

xcollazo added a subtask: T381016: Optimize XML Dump code to be able to handle wikis from simplewiki to enwiki.Jan 21 2025, 9:16 PM

xcollazo added a parent task: T366752: Dumps 2.0 Phase III: Production level dumps (SDS 1.2).

pfischer changed the status of subtask T384383: Modify XML dumping code to be able to do 'partial' dumps from Open to In Progress.Jan 30 2025, 2:31 PM

pfischer changed the status of subtask T384385: Refactor code to use new table and column names from Open to In Progress.

pfischer changed the status of subtask T384945: Modify code to dump all slots from Open to In Progress.Jan 30 2025, 4:17 PM

pfischer changed the status of subtask T384945: Modify code to dump all slots from In Progress to Stalled.Feb 7 2025, 8:43 AM

xcollazo added a subtask: T384500: Consider writing Spark files to Ceph (S3) instead of Hadoop.Feb 11 2025, 7:11 PM

Ottomata added a project: Epic.Feb 18 2025, 6:53 PM

Ahoelzl moved this task from Incoming (new tickets) to Tag with Roadmap on the Data-Engineering board.Feb 18 2025, 6:53 PM

Ahoelzl edited projects, added Data-Engineering-Roadmap; removed Data-Engineering.

Ahoelzl edited projects, added DPE-Mediawiki-Content; removed Dumps 2.0.Feb 20 2025, 7:16 PM

xcollazo closed subtask T381016: Optimize XML Dump code to be able to handle wikis from simplewiki to enwiki as Resolved.Mar 10 2025, 5:26 PM

xcollazo closed subtask T384385: Refactor code to use new table and column names as Resolved.

xcollazo closed subtask T384383: Modify XML dumping code to be able to do 'partial' dumps as Resolved.Mar 10 2025, 5:30 PM

BTullis closed subtask T384500: Consider writing Spark files to Ceph (S3) instead of Hadoop as Resolved.Mar 19 2025, 2:24 PM

Gehel changed the status of subtask T384500: Consider writing Spark files to Ceph (S3) instead of Hadoop from Resolved to Declined.Mar 21 2025, 9:46 AM

xcollazo renamed this task from Productionization of code to dump in XML to Production-level file export (aka dump) of MW Content in XML.Jul 25 2025, 5:51 PM

xcollazo changed the task status from Open to In Progress.

xcollazo claimed this task.

xcollazo triaged this task as High priority.

xcollazo edited projects, added Data-Engineering; removed Data-Engineering-Roadmap.

xcollazo updated the task description. (Show Details)

xcollazo removed a subscriber: Aklapper.

xcollazo added a subtask: T396031: Remove code and tables related to deprecated mediawiki_wikitext_history and mediawiki_wikitext_current.Jul 25 2025, 5:55 PM

xcollazo added a subtask: Restricted Task.

xcollazo added a subtask: T379410: Go over tasks in #dumps-generation and figure what makes sense to fix in Dumps 2.0 (DPE-Mediawiki-Content).Jul 25 2025, 6:01 PM

xcollazo added a subtask: Restricted Task.Jul 25 2025, 7:09 PM

xcollazo changed the status of subtask Restricted Task from Open to In Progress.Jul 29 2025, 1:24 PM

xcollazo added a subtask: T400632: Missing/inconsistent page_redirect_target field for redirects in Mediawiki content current v1 dumps.Aug 1 2025, 3:37 PM