Change Details

As part of [SDS 2.6.2](https://docs.google.com/document/d/1iTgL8V7FNb1VG_mWp6U_2SmWN4F2BEm789BSvvYq3PY/edit), I've been investigating the data dependencies of the [movement metrics](https://meta.wikimedia.org/wiki/Movement_Insights/Movement_metrics). Our critical path takes around 25 days and goes: * XML dumps generation * loading XML dumps to HDFS * mediawiki_wikitext_history * research_article_quality * knowledge_gaps By far the longest portion (~19 days) is waiting for the XML dumps to be generated. But after the first 7 days (when the English Wikipedia dump arrives), we're waiting only on the Wikidata dump. I doubt that anyone is regularly using the Wikidata XML dump since `wmf.wikidata_entity` (which comes from the JSON dump) is much better and faster. The XML dump is apparently the only one that contains non-current data, but that's probably a very rare need.

As part of [SDS 2.6.2](https://docs.google.com/document/d/1iTgL8V7FNb1VG_mWp6U_2SmWN4F2BEm789BSvvYq3PY/edit), I've been investigating the data dependencies of the [movement metrics](https://meta.wikimedia.org/wiki/Movement_Insights/Movement_metrics). Our critical path takes around 25 days and goes: * **[XML dumps generation](https://meta.wikimedia.org/wiki/Data_dumps)** * **loading XML dumps to HDFS** ([Python script](https://github.com/wikimedia/analytics-refinery/blob/master/bin/import-mediawiki-dumps), [template for running script](https://github.com/wikimedia/operations-puppet/blob/production/modules/profile/templates/analytics/refinery/job/refinery-import-mediawiki-dumps.sh.erb), [Puppet management of SystemD timers running script](https://github.com/wikimedia/operations-puppet/blob/production/modules/profile/manifests/analytics/refinery/job/import_mediawiki_dumps.pp)) * **[mediawiki_wikitext_history](https://wikitech.wikimedia.org/wiki/Analytics/Data_Lake/Content/Mediawiki_wikitext_history)** * **research_article_quality** ([Airflow DAG](https://gitlab.wikimedia.org/repos/data-engineering/airflow-dags/-/blob/main/research/dags/article_quality_dag.py), [code](https://gitlab.wikimedia.org/repos/research/article-quality/)) * **knowledge_gaps** ([Airflow DAG](https://gitlab.wikimedia.org/repos/data-engineering/airflow-dags/-/blob/main/research/dags/knowledge_gaps_dag.py), [code](https://gitlab.wikimedia.org/repos/research/knowledge-gaps/)) By far the longest portion (~19 days) is waiting for the XML dumps to be generated. But after the first 7 days (when the English Wikipedia dump arrives), we're waiting only on the Wikidata dump. I doubt that anyone is regularly using the Wikidata XML dump since `wmf.wikidata_entity` (which comes from the JSON dump) is much better and faster. The XML dump is apparently the only one that contains non-current data, but that's probably a very rare need. Can we skip loading the Wikidata XML altogether? Other strategies like splitting it out as a separate job would be fine too, but just skipping it would be much easier and likely fine, with no one using the data.

As part of [SDS 2.6.2](https://docs.google.com/document/d/1iTgL8V7FNb1VG_mWp6U_2SmWN4F2BEm789BSvvYq3PY/edit), I've been investigating the data dependencies of the [movement metrics](https://meta.wikimedia.org/wiki/Movement_Insights/Movement_metrics). Our critical path takes around 25 days and goes: * * **[XML dumps generation](https://meta.wikimedia.org/wiki/Data_dumps)** * loading XML dumps to HDFS**loading XML dumps to HDFS** ([Python script](https://github.com/wikimedia/analytics-refinery/blob/master/bin/import-mediawiki-dumps), [template for running script](https://github.com/wikimedia/operations-puppet/blob/production/modules/profile/templates/analytics/refinery/job/refinery-import-mediawiki-dumps.sh.erb), [Puppet management of SystemD timers running script](https://github.com/wikimedia/operations-puppet/blob/production/modules/profile/manifests/analytics/refinery/job/import_mediawiki_dumps.pp)) * m* **[mediawiki_wikitext_history](https://wikitech.wikimedia.org/wiki/Analytics/Data_Lake/Content/Mediawiki_wikitext_history)** * research_article_quality**research_article_quality** ([Airflow DAG](https://gitlab.wikimedia.org/repos/data-engineering/airflow-dags/-/blob/main/research/dags/article_quality_dag.py), [code](https://gitlab.wikimedia.org/repos/research/article-quality/)) * knowledge_gaps**knowledge_gaps** ([Airflow DAG](https://gitlab.wikimedia.org/repos/data-engineering/airflow-dags/-/blob/main/research/dags/knowledge_gaps_dag.py), [code](https://gitlab.wikimedia.org/repos/research/knowledge-gaps/)) By far the longest portion (~19 days) is waiting for the XML dumps to be generated. But after the first 7 days (when the English Wikipedia dump arrives), we're waiting only on the Wikidata dump. I doubt that anyone is regularly using the Wikidata XML dump since `wmf.wikidata_entity` (which comes from the JSON dump) is much better and faster. The XML dump is apparently the only one that contains non-current data, but that's probably a very rare need. Can we skip loading the Wikidata XML altogether? Other strategies like splitting it out as a separate job would be fine too, but just skipping it would be much easier and likely fine, with no one using the data.