Page MenuHomePhabricator
Create Task
Maniphest T366752

Dumps 2.0 Phase III: Production level dumps (SDS 1.2)
Open, HighPublic
Actions

Description

(Previous phase: T358877: Dumps 2.0 Phase II: Production intermediate table milestone)

Achieving this milestone would mean that we are in a position to consider decommissioning the XML Dumps from Dumps 1.0 in favor of Dumps 2.0.

Related Objects
Search...

StatusSubtypeAssignedTask
 OpenNoneT366752 Dumps 2.0 Phase III: Production level dumps (SDS 1.2)
 ResolvedBTullisT374952 Figure why we can't dump labswiki, aka Wikitech
 InvalidNoneT363941 dumps_publish_wikitext_raw_to_xml DAG fails sporadically. Needs sensors to wait before running.
 ResolvedBTullisT371720 Bump Presto to latest pickup Iceberg fixes and VIEWs support
 OpenNoneT379365 [SPIKE] Benchmark the run time of batch processing
OpenNoneT379921 Implement a canary against MariaDB schema changes
 OpenNoneT380773 Time partitioning for mediawiki_content_history
ResolvedMilimetricT381615 mediawiki-jdbc, our custom spark data source, doesn't fully hit the indexes
 ResolvedxcollazoT371509 Consider whether we want to dump private and/or closed wikis
 In ProgressxcollazoT384382 Production-level file export (aka dump) of MW Content in XML
 ResolvedpfischerT381016 Optimize XML Dump code to be able to handle wikis from simplewiki to enwiki
 ResolvedpfischerT384383 Modify XML dumping code to be able to do 'partial' dumps
 ResolvedpfischerT384385 Refactor code to use new table and column names
 OpenNoneT384392 Put together a DPE Deep Dive session on learnings from Dumps 2 XML generation code
 ResolvedxcollazoT384945 Modify code to dump all slots
 DeclinedBTullisT384500 Consider writing Spark files to Ceph (S3) instead of Hadoop
 DuplicateNoneT386211 Modify wmf_content.mediawiki_content_history_v1 to include slot origin
 ResolvedAntoine_QuhenT388793 When doing ADD COLUMN to a struct under a map, Iceberg fails to SELECT it
 ResolvedxcollazoT396031 Remove code and tables related to deprecated mediawiki_wikitext_history and mediawiki_wikitext_current
Restricted Task
 OpenNoneT379410 Go over tasks in #dumps-generation and figure what makes sense to fix in Dumps 2.0 (DPE-Mediawiki-Content)
Restricted Task
 OpenNoneT400632 Missing/inconsistent page_redirect_target field for redirects in Mediawiki content current v1 dumps
 ResolvedxcollazoT384381 Airflow jobs to do monthly XML dumps
 OpenxcollazoT395139 MediaWiki Content History alerts too much for minor reconcile issues
 ResolvedxcollazoT401022 Implement the data layout, UI, and documentation for the XML file export
 ResolvedxcollazoT402209 Modify XML dumping code to support multiple compression algorithms
 ResolvedxcollazoT402229 Compare outputs between XML File Export and MediaWiki's Special:Export
 ResolvedxcollazoT405641 Adapt MW Content pipelines to the removal of upstream revision.rev_sha1
 ResolvedAntoine_QuhenT405944 Rewrite wmf_content.mediawiki_content_*_v1 tables with a new column for origin_rev_id
 ResolvedxcollazoT407237 Wait till november wmf_raw.mediawiki_slots sqoop table is available, and apply origin_rev_id fix to mw_content tables
 ResolvedxcollazoT407649 Compare the exported content between File Export and DumpV1
 ResolvedxcollazoT414389 Publish Dumps 2 to dumps.wikimedia.org and provide only monthly dumps
 ResolvedxcollazoT384397 HDFS capacity needs for XML Dumps temporary storage
 OpenNoneT384723 [Dumps 2] Pay down technical debt
 OpenNoneT384724 Investigate Flink app errors
 OpenNoneT380197 Create a fat jar to embed wmf-event-stream Spark data source
 OpenNoneT380648 Address post merge review comments on event emit code
 OpenNoneT375298 Stop using spark.jars.packages
 OpenNoneT380101 Figure out how to unit test Iceberg tables
 OpenNoneT380104 Figure out how to send in file:// URIs to wmf-event-stream
 ResolvedxcollazoT382703 refinery-python should be moved to the data-engineering namespace
 ResolvedbrouberolT379676 Add relevant kafka clusters to defined airflow connections in puppet
 ResolvedxcollazoT384744 Add table maintenance for wmf_data_ops.data_quality_metrics
 OpenxcollazoT388612 Airflow table maintenance auto-create execution pools
 OpenamastilovicT384726 Implement full parity between HiveSensor and RESTExternalTaskSensor
 OpenNoneT384871 Usability improvements to mediawiki-content-history dashboards
 ResolvedxcollazoT385112 Investigate reasons for remaining inconsistencies
 ResolvedxcollazoT387021 Support downstream users in adopting mediawiki_content_history_v1
 ResolvedamastilovicT386973 A dataset sensor should work independent of airflow instance
ResolvedfkaelinT386933 Research airflow instance can not reach out to gitlab.wikimedia.org
 ResolvedxcollazoT387033 Figure root cause of silent failures when computing metrics for mediawiki_content_history_v1
 ResolvedxcollazoT388040 Estimate effort for migrating wmf.wikidata_entity to the new mediawiki content pipelines
 ResolvedxcollazoT390727 Canary failure on airflow platform_eng intsance after migrating to Kubernetes
 ResolvedtchinT388439 Add metrics for monthly reconciles
 ResolvedxcollazoT388715 Investigate and fix duplicate data on wmf_content.mediawiki_content_history_v1 for muswiki
 ResolvedtchinT389162 [Data Quality] Add ability to add tags to alerts
 ResolvedtchinT384962 Implement alerting for wmf_content.mediawiki_content_history_v1
 ResolvedxcollazoT391279 Daily updated wmf_content.mediawiki_content_current_v1
 ResolvedxcollazoT391280 Modify table maintenance mechanism to support Iceberg's rewrite_position_delete_files()
 ResolvedxcollazoT391282 Create table and pyspark job to produce wmf_content.mediawiki_content_current_v1
 ResolvedxcollazoT391283 Create Airflow pipeline to produce wmf_content.mediawiki_content_current_v1
 ResolvedxcollazoT366544 Spike: Figure how best to produce wmf_content.mediawiki_content_current_v1
 ResolvedxcollazoT392494 Add data quality metrics to mediawiki_content_current_v1
 ResolvedxcollazoT392948 Document wmf_content.mediawiki_content_current_v1
 ResolvedxcollazoT393012 Enable merge-on-read for wmf_content.mediawiki_content_history_v1
 ResolvedfkaelinT385999 [FY25-SDS1.4.2] Update research pipelines to use Dumps2
 ResolvedfkaelinT386000 Wikidiff dataset using content history
 ResolvedfkaelinT386001 Update Article quality pipeline to use Dumps2
 ResolvedfkaelinT386003 Update knowledge gaps pipeline to use article quality features V2
 ResolvedfkaelinT388144 Update reference risk pipelines for dumps2
 ResolvedfkaelinT388146 Update addalink pipeline for dumps2
 ResolvedfkaelinT390704 Update article embeddings pipeline for dumps2
 ResolvedxcollazoT396882 DPE Deep Dive on MW Content Tables
 ResolvedxcollazoT397333 Create Airflow pool for bursty MW Content Pipelines tasks
 ResolvedbrouberolT406371 Bump max mapped task for all Airflow instances to 1200
 ResolvedxcollazoT406864 Optimize table maintenance for wmf_content.mediawiki_content_current_v1
 OpenNoneT401010 Optimize metrics computation for the MW Content Pipeline
 In ProgressxcollazoT408819 When wikis cannot be exported due to SiteInfo, don't fail them
 ResolvedJMonton-WMFT408850 mediawiki_event_enrichment should enrich all events for the page_content_change stream
 OpenNoneT412461 On reconcile, consider what happens when a restore and a delete happen on the same revision

Event Timeline

There are a very large number of changes, so older changes are hidden. Show Older Changes
xcollazo added a subtask: T381615: mediawiki-jdbc, our custom spark data source, doesn't fully hit the indexes.Dec 5 2024, 8:47 PM
xcollazo added a subtask: T371509: Consider whether we want to dump private and/or closed wikis.Dec 6 2024, 7:12 PM
pfischer changed the status of subtask T381016: Optimize XML Dump code to be able to handle wikis from simplewiki to enwiki from Open to In Progress.Dec 16 2024, 3:15 PM
xcollazo changed the status of subtask T381615: mediawiki-jdbc, our custom spark data source, doesn't fully hit the indexes from Open to In Progress.Dec 19 2024, 4:44 PM
tchin subscribed.Dec 23 2024, 12:12 PM
xcollazo closed subtask T381615: mediawiki-jdbc, our custom spark data source, doesn't fully hit the indexes as Resolved.Jan 8 2025, 6:24 PM
xcollazo added a subtask: T380104: Figure out how to send in file:// URIs to wmf-event-stream.Jan 8 2025, 7:38 PM
xcollazo added a subtask: T380101: Figure out how to unit test Iceberg tables.
xcollazo added a subtask: T375298: Stop using spark.jars.packages.Jan 8 2025, 7:41 PM
xcollazo added a subtask: T380648: Address post merge review comments on event emit code.Jan 8 2025, 7:44 PM
xcollazo removed a subtask: T381016: Optimize XML Dump code to be able to handle wikis from simplewiki to enwiki.Jan 21 2025, 9:16 PM
xcollazo added a subtask: T384382: Production-level file export (aka dump) of MW Content in XML.
xcollazo mentioned this in T384397: HDFS capacity needs for XML Dumps temporary storage.Jan 21 2025, 10:07 PM
xcollazo added a subtask: T384397: HDFS capacity needs for XML Dumps temporary storage.
xcollazo closed subtask T384397: HDFS capacity needs for XML Dumps temporary storage as Resolved.Jan 22 2025, 9:35 PM
xcollazo removed a subtask: T380648: Address post merge review comments on event emit code.Jan 24 2025, 4:32 PM
xcollazo removed a subtask: T375298: Stop using spark.jars.packages.
xcollazo removed a subtask: T380101: Figure out how to unit test Iceberg tables.
xcollazo removed a subtask: T380104: Figure out how to send in file:// URIs to wmf-event-stream.
xcollazo removed a subtask: T382703: refinery-python should be moved to the data-engineering namespace.
xcollazo removed a subtask: T379676: Add relevant kafka clusters to defined airflow connections in puppet.Jan 24 2025, 4:35 PM
xcollazo added a subtask: T384726: Implement full parity between HiveSensor and RESTExternalTaskSensor.Jan 24 2025, 4:54 PM
xcollazo added a subtask: T384871: Usability improvements to mediawiki-content-history dashboards.Jan 27 2025, 8:59 PM
xcollazo added a subtask: T384962: Implement alerting for wmf_content.mediawiki_content_history_v1.Jan 28 2025, 7:51 PM
Ahoelzl edited projects, added Data-Engineering, DPE-Mediawiki-Content; removed Dumps 2.0.Feb 20 2025, 7:21 PM
Ahoelzl moved this task from Incoming (new tickets) to Tag with Roadmap on the Data-Engineering board.Mar 3 2025, 9:28 PM
Ahoelzl edited projects, added Data-Engineering-Roadmap; removed Data-Engineering.
xcollazo added a subtask: T388793: When doing ADD COLUMN to a struct under a map, Iceberg fails to SELECT it.Mar 13 2025, 2:02 PM
xcollazo removed a subtask: T388793: When doing ADD COLUMN to a struct under a map, Iceberg fails to SELECT it.
xcollazo changed the status of subtask T366544: Spike: Figure how best to produce wmf_content.mediawiki_content_current_v1 from Open to In Progress.Mar 20 2025, 3:16 PM
xcollazo closed subtask T363941: dumps_publish_wikitext_raw_to_xml DAG fails sporadically. Needs sensors to wait before running. as Invalid.Mar 20 2025, 4:13 PM
xcollazo closed subtask T388715: Investigate and fix duplicate data on wmf_content.mediawiki_content_history_v1 for muswiki as Resolved.Apr 2 2025, 4:08 PM
xcollazo added a subtask: T391279: Daily updated wmf_content.mediawiki_content_current_v1.Apr 7 2025, 6:34 PM
xcollazo removed a subtask: T366544: Spike: Figure how best to produce wmf_content.mediawiki_content_current_v1.Apr 7 2025, 6:50 PM
Ahoelzl changed the status of subtask T391279: Daily updated wmf_content.mediawiki_content_current_v1 from Open to In Progress.Apr 25 2025, 4:35 PM
Ahoelzl moved this task from Backlog to Next on the Data-Engineering-Roadmap board.
xcollazo added a subtask: T393012: Enable merge-on-read for wmf_content.mediawiki_content_history_v1.Apr 30 2025, 1:55 PM
xcollazo changed the status of subtask T393012: Enable merge-on-read for wmf_content.mediawiki_content_history_v1 from Open to In Progress.Apr 30 2025, 7:59 PM
xcollazo removed a subtask: T393012: Enable merge-on-read for wmf_content.mediawiki_content_history_v1.May 8 2025, 2:36 PM
xcollazo changed the status of subtask T385112: Investigate reasons for remaining inconsistencies from Open to In Progress.May 8 2025, 7:56 PM
xcollazo added a subtask: Restricted Task.May 14 2025, 2:09 PM
BTullis mentioned this in T237361: Discuss common needs in a job manager/scheduler.May 14 2025, 6:05 PM
BTullis mentioned this in T149894: do proof of concept dump of one wiki via analytics infra.May 14 2025, 6:09 PM
BTullis mentioned this in T143205: Draft usage scenarios for job/workflow manager.
BTullis mentioned this in T143206: List requirements needed for task/job/workflow manager..May 14 2025, 6:11 PM
BTullis mentioned this in T143207: Evaluate software packages for job/task/workflow management.
BTullis mentioned this in T146070: Review progress of dumps rewrite.
BTullis mentioned this in T128520: What already available software can we build on for a job scheduler?.
BTullis mentioned this in T128755: What monitoring of dumps can we provide to users?.May 14 2025, 6:14 PM
BTullis mentioned this in T128752: What sort of internal storage do we want for the new dumps sytem?.May 14 2025, 6:16 PM
BTullis mentioned this in T128754: Can we use a stream of events for facilitating incrementals of the project content dumps?.May 14 2025, 6:18 PM
nshahquinn-wmf mentioned this in T128513: Dumps 2.0 Platform design questions.May 14 2025, 9:49 PM
xcollazo added a subtask: T395139: MediaWiki Content History alerts too much for minor reconcile issues.May 23 2025, 4:27 PM
xcollazo closed subtask T391279: Daily updated wmf_content.mediawiki_content_current_v1 as Resolved.May 23 2025, 5:07 PM
xcollazo closed subtask T384962: Implement alerting for wmf_content.mediawiki_content_history_v1 as Resolved.May 28 2025, 1:20 AM
xcollazo added a subtask: T396031: Remove code and tables related to deprecated mediawiki_wikitext_history and mediawiki_wikitext_current.Jun 4 2025, 2:46 PM
xcollazo closed subtask T371509: Consider whether we want to dump private and/or closed wikis as Resolved.Jun 6 2025, 7:36 PM
xcollazo closed subtask T387021: Support downstream users in adopting mediawiki_content_history_v1 as Resolved.Jun 6 2025, 7:45 PM
xcollazo closed subtask T389162: [Data Quality] Add ability to add tags to alerts as Resolved.
xcollazo added a subtask: T396882: DPE Deep Dive on MW Content Tables.Jun 13 2025, 3:29 PM
xcollazo added a subtask: T397333: Create Airflow pool for bursty MW Content Pipelines tasks.Jun 18 2025, 1:39 PM
xcollazo changed the status of subtask T397333: Create Airflow pool for bursty MW Content Pipelines tasks from Open to In Progress.Jun 20 2025, 2:41 PM
xcollazo closed subtask T397333: Create Airflow pool for bursty MW Content Pipelines tasks as Resolved.Jun 25 2025, 2:20 PM
xcollazo closed subtask T396882: DPE Deep Dive on MW Content Tables as Resolved.
xcollazo mentioned this in T338057: Upgrade Spark to a version with long term Iceberg support, and with fixes to support Dumps 2.0.Jun 25 2025, 2:24 PM
xcollazo removed a subtask: T338057: Upgrade Spark to a version with long term Iceberg support, and with fixes to support Dumps 2.0.
xcollazo changed the status of subtask T396031: Remove code and tables related to deprecated mediawiki_wikitext_history and mediawiki_wikitext_current from Open to In Progress.Jul 8 2025, 3:54 PM
xcollazo changed the status of subtask T384382: Production-level file export (aka dump) of MW Content in XML from Open to In Progress.Jul 25 2025, 5:51 PM
xcollazo removed a subtask: T396031: Remove code and tables related to deprecated mediawiki_wikitext_history and mediawiki_wikitext_current.Jul 25 2025, 5:55 PM
xcollazo removed a subtask: Restricted Task.
xcollazo removed a subtask: T379410: Go over tasks in #dumps-generation and figure what makes sense to fix in Dumps 2.0 (DPE-Mediawiki-Content).Jul 25 2025, 6:01 PM
tchin closed subtask T388439: Add metrics for monthly reconciles as Resolved.Jul 25 2025, 6:34 PM
xcollazo removed a subtask: T384381: Airflow jobs to do monthly XML dumps.Aug 1 2025, 6:22 PM
xcollazo removed a subtask: T395139: MediaWiki Content History alerts too much for minor reconcile issues.Aug 1 2025, 6:24 PM
BTullis mentioned this in T126722: Spike: How will we migrate to Dumps 2.0?.Aug 5 2025, 3:31 PM
BTullis mentioned this in T403793: 20250901 enwiki dump is triple the normal size.Sep 12 2025, 1:02 PM
xcollazo added a subtask: T406371: Bump max mapped task for all Airflow instances to 1200.Oct 3 2025, 6:28 PM
xcollazo added a subtask: T406864: Optimize table maintenance for wmf_content.mediawiki_content_current_v1.Oct 9 2025, 1:26 PM
brouberol changed the status of subtask T406371: Bump max mapped task for all Airflow instances to 1200 from Open to In Progress.Oct 14 2025, 8:58 AM
brouberol closed subtask T406371: Bump max mapped task for all Airflow instances to 1200 as Resolved.Oct 14 2025, 1:36 PM
xcollazo added a subtask: T407237: Wait till november wmf_raw.mediawiki_slots sqoop table is available, and apply origin_rev_id fix to mw_content tables.Oct 14 2025, 1:40 PM
xcollazo added a subtask: T401010: Optimize metrics computation for the MW Content Pipeline.Oct 17 2025, 5:06 PM
xcollazo removed a subtask: T407237: Wait till november wmf_raw.mediawiki_slots sqoop table is available, and apply origin_rev_id fix to mw_content tables.Oct 17 2025, 5:09 PM
Ahoelzl renamed this task from Dumps 2.0 Phase III: Production level dumps to Dumps 2.0 Phase III: Production level dumps (SDS 1.2).Oct 23 2025, 11:56 PM
Ahoelzl triaged this task as High priority.
Ahoelzl removed subscribers: • VirginiaPoundstone, • WDoranWMF.
Ahoelzl moved this task from Next to Q2 FY25-26 on the Data-Engineering-Roadmap board.Oct 29 2025, 9:38 PM
xcollazo added a subtask: T408819: When wikis cannot be exported due to SiteInfo, don't fail them.Oct 30 2025, 3:54 PM
xcollazo added a subtask: T408850: mediawiki_event_enrichment should enrich all events for the page_content_change stream.Oct 30 2025, 6:24 PM
Ahoelzl closed subtask T408850: mediawiki_event_enrichment should enrich all events for the page_content_change stream as Resolved.Nov 18 2025, 10:26 PM
Restricted Application edited projects, added Data-Engineering; removed Data-Engineering-Roadmap. · View Herald TranscriptNov 18 2025, 10:26 PM
A_smart_kitten mentioned this in T410248: Data Engineering's Herald rule (H126) might not currently react to any tasks.Nov 19 2025, 2:15 PM
Aklapper edited projects, added Data-Engineering-Roadmap; removed Data-Engineering.Nov 19 2025, 2:20 PM
Restricted Application edited projects, added Data-Engineering; removed Data-Engineering-Roadmap. · View Herald TranscriptNov 19 2025, 2:20 PM
Aklapper edited projects, added Data-Engineering-Roadmap; removed Data-Engineering.Nov 19 2025, 3:06 PM
brouberol changed the status of subtask T408819: When wikis cannot be exported due to SiteInfo, don't fail them from Open to In Progress.Nov 27 2025, 8:14 AM
xcollazo added a subtask: T412461: On reconcile, consider what happens when a restore and a delete happen on the same revision.Dec 11 2025, 9:42 PM
xcollazo changed the status of subtask T406864: Optimize table maintenance for wmf_content.mediawiki_content_current_v1 from Open to In Progress.Tue, Jan 13, 12:42 PM
xcollazo closed subtask T385112: Investigate reasons for remaining inconsistencies as Resolved.Thu, Jan 15, 5:24 PM
Ahoelzl closed subtask T406864: Optimize table maintenance for wmf_content.mediawiki_content_current_v1 as Resolved.Mon, Feb 2, 6:32 PM
Content licensed under Creative Commons Attribution-ShareAlike (CC BY-SA) 4.0 unless otherwise noted; code licensed under GNU General Public License (GPL) 2.0 or later and other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL · Credits