Import the manual scraper run per-wiki outputs into our new Hive tables.
- [ ] Identify the scraper runs that ended successfully
- https://analytics.wikimedia.org/published/datasets/one-off/html-dump-scraper-refs/ contains outputs of our previous runs. The earlier data in https://analytics.wikimedia.org/published/datasets/one-off/html-dump-scraper-refs-and-maps/ were un-published because of bad quality.
- ~~Read the *.log files to verify that the run was successful. eg. https://analytics.wikimedia.org/published/datasets/one-off/html-dump-scraper-refs/20250320/scrape-wiki-dump.log ~~ Edit: this is overkill, and difficult because of scraper software changes leaving different artifacts in the log.
- [x] Decide on which subset of the data is important.
- Import summary data for all wikis.
- Ignore the lists of template names, since our current strategy is to only look at the latest snapshot of template statistics.
- [ ] Write a one-off script to load the data, either directly (eg. Spark insert from Elixir or Python) or by preprocessing the data and importing as CSV or JSON using an HQL script.
- [ ] Verify that the time series is linear for each column, ie. double-check that the data looks reasonable and null out any bad values.
- [ ] Document this historical backfilling in Datahub if possible.