Our generated reports will take up a lot of space, something in the range of 20-40 GB. Create an external storage volume, attach, and point the output directory there. Sync the existing reports over. Resume processing.
Description
Description
| Status | Subtype | Assigned | Task | ||
|---|---|---|---|---|---|
| Resolved | None | T345411 Scraper: destroy Cloud VPS runner instance | |||
| Resolved | None | T341751 Publish dump scraper reports | |||
| Resolved | None | T335411 Scraper: produce spreadsheet of scraped statistics for comparing wikis | |||
| Resolved | awight | T332032 Create baseline statistics for reference usage (2023) | |||
| Resolved | awight | T338496 Attach external volume to the scraper instance, store reports there |
Event Timeline
Comment Actions
New 80GB volume attached to the runner as /dev/sdb: https://horizon.wikimedia.org/project/volumes/ec65493b-e06c-446a-8aa8-7e4df54ee7fd/
# Oops, this wasn't necessary: sudo mkfs.ext4 /dev/sdb sudo wmcs-prepare-cinder-volume
It generates this /etc/fstab entry:
UUID=782b67f8-d021-41af-b18a-f529b10310b1 /srv/reports ext4 discard,nofail,x-systemd.device-timeout=2s 0 2
To move partial reports from the root partition:
mv reports reports-slash sudo chgrp wikidev /srv/reports/ sudo chmod g+ws /srv/reports/ ln -s /srv/reports /srv/scrape-wiki-html-dump/ rsync -av --info=progress2 reports-slash/ reports/