Import the manual scraper run per-wiki outputs into our new Hive tables.
- Identify the scraper runs that ended successfully
- https://analytics.wikimedia.org/published/datasets/one-off/html-dump-scraper-refs/ contains outputs of our previous runs. The earlier data in https://analytics.wikimedia.org/published/datasets/one-off/html-dump-scraper-refs-and-maps/ were un-published because of bad quality.
Read the *.log files to verify that the run was successful. eg. https://analytics.wikimedia.org/published/datasets/one-off/html-dump-scraper-refs/20250320/scrape-wiki-dump.logEdit: this is overkill, and difficult because of scraper software changes leaving different artifacts in the log.
- Decide on which subset of the data is important.
- Import summary data for all wikis.
- Ignore the lists of template names, since our current strategy is to only look at the latest snapshot of template statistics.
- Write a one-off script to load the data, either directly (eg. Spark insert from Elixir or Python) or by preprocessing the data and importing as CSV or JSON using an HQL script.
- Backfill from manual February 2026 run, which is in the new format but only available as page-summary .ndjson.
- Verify that the time series is linear for each column, ie. double-check that the data looks reasonable and null out any bad values.
- Document this historical backfilling in Datahub if possible.
Code to review:
