For Data Products Sprint 00, we want to work on having a fully working dump process for one of the smaller wikis. We chose simplewki.
In this task we want to make sure we can backfill for simplewiki.
Done is:
| Title | Reference | Author | Source Branch | Dest Branch | |
|---|---|---|---|---|---|
| Fix ivy issue on mediawiki dumps, and some cosmetic fixes. | repos/data-engineering/airflow-dags!495 | xcollazo | fix-ivy-issue | main | |
| Add DAG to backfill wmf_dumps.wikitext_raw. | repos/data-engineering/airflow-dags!484 | xcollazo | add-mediawiki-dumps-backfill | main |
| Status | Subtype | Assigned | Task | ||
|---|---|---|---|---|---|
| Open | VirginiaPoundstone | T345988 [Epic] XML MediaWiki data dumps for right to fork | |||
| Resolved | Milimetric | T330296 Dumps 2.0 Phase I: Proof of concept for MediaWiki XML content dump via Event Platform, Iceberg and Spark | |||
| Resolved | xcollazo | T340861 Implement a backfill job for the dumps hourly table | |||
| Resolved | xcollazo | T344709 Implement a backfill job for the dumps hourly table that can handle simplewiki |
wmf_dumps.wikitext_raw_rc1 has been backfilled with all of simplewiki:
spark-sql (default)> select count(1) as count from wikitext_raw_rc1 where wiki_db = 'simplewiki'; count 8087771 Time taken: 38.38 seconds, Fetched 1 row(s)
simplewiki scale does not trigger the issues we had seen on T340861.
xcollazo updated https://gitlab.wikimedia.org/repos/data-engineering/airflow-dags/-/merge_requests/484
Draft: Add DAG to backfill wmf_dumps.wikitext_raw.
T340861#9135613 should take care of delivering the Airflow job to production. Thus, marking this task as ready for code review.
xcollazo merged https://gitlab.wikimedia.org/repos/data-engineering/airflow-dags/-/merge_requests/484
Add DAG to backfill wmf_dumps.wikitext_raw.
With https://gitlab.wikimedia.org/repos/data-engineering/airflow-dags/-/merge_requests/484 and T341383 in production, we now have a full backfill that can definitely handle simplewiki. We're done here.
xcollazo updated https://gitlab.wikimedia.org/repos/data-engineering/airflow-dags/-/merge_requests/495
Fix ivy issue on mediawiki dumps, and some cosmetic fixes.
milimetric merged https://gitlab.wikimedia.org/repos/data-engineering/airflow-dags/-/merge_requests/495
Fix ivy issue on mediawiki dumps, and some cosmetic fixes.