Before we destroy the instance, let's take the opportunity to check for errors:
Implementation: https://gitlab.com/wmde/technical-wishes/scrape-wiki-html-dump/-/tree/validate-outputs?ref_type=heads
| Status | Subtype | Assigned | Task | ||
|---|---|---|---|---|---|
| Resolved | None | T345411 Scraper: destroy Cloud VPS runner instance | |||
| Resolved | None | T345417 Scraper: scan outputs for inconsistencies |
The validator has been run on outputs: we found no JSON errors, but several metric tons of duplicate errors. A tally of duplicates in each file will be included in reports, in a file "duplicates.txt".
As far as I can tell the only new file in that branch is https://gitlab.com/wmde/technical-wishes/scrape-wiki-html-dump/-/blob/validate-outputs/validate-outputs.exs. Is that correct?
Can we see the output anywhere online? Specifically the duplicates file?
Unfortunately not yet, this is tracked in T341751. The outputs and duplicates file are currently on a closed WMCS instance but members of our team have login access:
ssh runner.dump-references-processor.eqiad1.wikimedia.cloud ls /srv/reports/duplicates.txt ls /srv/reports/
Outputs and duplicates.txt are publicly available now: https://analytics.wikimedia.org/published/datasets/one-off/html-dump-scraper-refs-and-maps/2023-06-01/