Update the research pipelines to use the new content history dataset (aka dumps2).
Context: DE has stopped producing dumps1 wikitext history datasets due to recurring issues, until the affected research pipelines are updated they will not produce any data (e.g. the knowledge gaps / content gap metrics).
| Status | Subtype | Assigned | Task | ||
|---|---|---|---|---|---|
| Open | None | T366752 Dumps 2.0 Phase III: Production level dumps (SDS 1.2) | |||
| Resolved | xcollazo | T391279 Daily updated wmf_content.mediawiki_content_current_v1 | |||
| Resolved | fkaelin | T385999 [FY25-SDS1.4.2] Update research pipelines to use Dumps2 | |||
| Resolved | fkaelin | T386000 Wikidiff dataset using content history | |||
| Resolved | fkaelin | T386001 Update Article quality pipeline to use Dumps2 | |||
| Resolved | fkaelin | T386003 Update knowledge gaps pipeline to use article quality features V2 | |||
| Resolved | fkaelin | T388144 Update reference risk pipelines for dumps2 | |||
| Resolved | fkaelin | T388146 Update addalink pipeline for dumps2 | |||
| Resolved | fkaelin | T390704 Update article embeddings pipeline for dumps2 |
Weekly updates
Weekly updates
Weekly updates
Weekly updates
Weekly updates
Final asana report:
Hypothesis: The research team will adopt the wmf_content.mediawiki_content_history_v1 on all existing use cases in which they currently use the deprecated wmf.mediawiki_wikitext_history.
Confirm whether the hypothesis was supported or contradicted
The hypothesis is supported.
Briefly describe what was accomplished over the course of the hypothesis work (list of deliverables, links to documents, etc.)
Over the course of this hypothesis work we delivered the following:
Major lessons
Migrating the research pipelines to using an incremental content history has been a goal for a while, it is great that we were able to accomplish this. This represented a substantial effort, and illustrated the cost of maintenance for a small team like research (one swe, no sre resources). The benefits of this investment are two-fold: the immediate benefit is that implementing and maintaining pipelines using mediawiki content is much cleaner and cheaper that it was previously. The main goal for research is to be able to run incremental daily pipelines (same cadence as the content history), but in order for that we also need an incremental mediawiki history dataset (currently generated as monthly snapshot). Until such a data source is available, the output generated by most of the pipelines updated with this hypothesis will remain unchanged.