Icinga is alerting:
wikitech-static CRIT - wikitech and wikitech-static out of sync (9089711s > 200000s)
Logging in reveals that the host has ran out disk:
root@wikitech-static:/var# df -h / Filesystem Size Used Avail Use% Mounted on /dev/xvda1 79G 77G 0 100% /
| Status | Subtype | Assigned | Task | ||
|---|---|---|---|---|---|
| Resolved | Clement_Goubert | T348503 wikitech-static is out of disk | |||
| Open | None | T378512 wikitech-static: clarify alert destinations |
The archived images directory seems to be taking most of the space:
root@wikitech-static:/srv/mediawiki/images/wikitech# du -sh archive/ 41G archive/
Mentioned in SAL (#wikimedia-operations) [2023-10-10T08:24:41Z] <taavi> wikitech-static: cleanup image archive directory: T348503
For documentation purpose, the maintenance script deleteArchivedFiles.php does not work and return 0 file deleted.
Running find /srv/mediawiki/images/wikitech/archive -type f | xargs rm is needed to clear disk space.
Mentioned in SAL (#wikimedia-operations) [2024-10-29T15:08:33Z] <claime> Running find /srv/mediawiki/images/wikitech/archive -type f | xargs rm on wikitech-static - T374114 T348503
Mentioned in SAL (#wikimedia-operations) [2024-10-29T15:10:33Z] <claime> Running /usr/bin/systemd-cat -t "import-wikitech.sh" /wikitech-static/wikitechsync/import-wikitech.sh & on wikitech-static - T348503