Context

This task is the result of a discussion in T362849 where it was realized that historical statuses of Wikidata are currently not possible to work with within the Data Lake. Specifically, wmf.mediawiki_wikitext_history does not have a partition for Wikidata since the 2024-02 dump when work that was done in T357859 came into effect.

Further details:

• The partition was removed to improve WMF Movement Insights' metrics delivery rate
  • Removing Wikidata cut the full wmf.mediawiki_wikitext_history process from ~25 to ~13 days
• Wikimedia Deutschland currently needs the historical states of Wikidata to rebuild data segments over time
• We expect to need this data for other tasks in the future

Request

• Based on discussions we believe that a separate job to make the partition would be best
• We do not want the Wikidata partition being added back in to negatively affect other teams (i.e. please do not revert changes made in T357859)

Alternatives

• To my knowledge this would require working directly with the XML dumps via PySpark
  • My assumption is that tasks like creating historical item segments on the full history of Wikidata would be prohibitively expensive if done locally, but please let us know if there are other alternatives!
• We could also discuss the created job being something that can be triggered and shut off in cases when there is a need for the Wikidata partition of wmf.mediawiki_wikitext_history
  • Thoughts on this would be appreciated!