| • taavi | |
| Mar 27 2024, 9:02 PM |
| F43610868: Screenshot 2024-03-28 at 14-16-51 Cloud VPS project board - WMCS - Cloud VPS projects - Dashboards - Grafana.png | |
| Mar 28 2024, 6:19 AM |
| F43554732: image.png | |
| Mar 27 2024, 9:07 PM |
Hey. The /ws-export partition on wsexport-prod02.wikisource.eqiad1.wikimedia.cloud seems to have filled up, and this is now flooding your mailboxes with error messages it seems.
For now I've blocked tools.wsexport@tools.wmflabs.org from sending mail on the Toolforge mail server to prevent any further spam - I'll remove that once this is resolved (or you can update it to toolforge.org which'll do it too).
Looking at metrics we collect this doesn't seem to have been a sudden explosion in used disk space:
root@wsexport-prod02:/ws-export/var# du -sh * 243M cache 32G calibre-temp 837M file-cache 59M log 2.5M sessions 4.8G www-data
There's a cronjob that's meant to clean up those Calibre files.
First up, it looks like it's not actually executable, oddly, so I fixed that:
$ ls -l /etc/cron.daily -rw-r--r-- 1 root root 85 Feb 12 09:20 calibre-cleanup $ sudo chmod +x /etc/cron.daily/calibre-cleanup $ ls -l /etc/cron.daily -rwxr-xr-x 1 root root 85 Feb 12 09:20 calibre-cleanup
But it's also not looking at /ws-export/var/calibre-temp, so I've added that and run it manually. Now there's lots of space:
$ df -h Filesystem Size Used Avail Use% Mounted on /dev/sda 40G 6.9G 31G 19% /ws-export
