The checkpointing is incompatible with the gzip format, something is not finalized in the output--or maybe we're concatenating on top of a truncated file without ending a block safely.
Code to review:
https://gitlab.com/wmde/technical-wishes/scrape-wiki-html-dump/-/merge_requests/66
| Status | Subtype | Assigned | Task | ||
|---|---|---|---|---|---|
| Resolved | None | T345411 Scraper: destroy Cloud VPS runner instance | |||
| Resolved | None | T341751 Publish dump scraper reports | |||
| Resolved | None | T335411 Scraper: produce spreadsheet of scraped statistics for comparing wikis | |||
| Resolved | awight | T332032 Create baseline statistics for reference usage (2023) | |||
| Resolved | None | T337450 Scraper: page-summary .gz can become corrupted after crash |
Find corrupted outputs:
(for i in reports/*.gz; do zcat $i > /dev/null || echo $i; done) > bad.txt