Showing as an icinga warning
root@install2002:/# df -hT Filesystem Type Size Used Avail Use% Mounted on udev devtmpfs 10M 0 10M 0% /dev tmpfs tmpfs 401M 41M 361M 11% /run /dev/vda1 ext4 78G 69G 5.0G 94% / tmpfs tmpfs 1003M 0 1003M 0% /dev/shm tmpfs tmpfs 5.0M 0 5.0M 0% /run/lock tmpfs tmpfs 1003M 0 1003M 0% /sys/fs/cgroup tmpfs tmpfs 1.0G 0 1.0G 0% /var/lib/nginx
1.4G home 63G srv 2.5G var
| Status | Subtype | Assigned | Task | ||
|---|---|---|---|---|---|
| Resolved | MoritzMuehlenhoff | T211850 install2002 94% disk usage on "/" | |||
| Resolved | colewhite | T229997 install2002 short on disk space |
apt-get clean -> 93%
So it's /srv and there the notable ones by size are: ~ 20G of Junos stuff for netops , ~ 35G /srv/wikimedia/pool the APT repo of which are 20G main and 12G thirdparty. /srv/wikimedia/incoming seems like a potential candidate for cleanup but that's under 1.5G.
It's notable here that install1002 has more space than install2002 does. (should have picked the same size at VM creation?) but also install1002 uses more space (at least the aptrepo part should be automatically synced between the 2, shouldn't it? and everything else is not synced?)
install2002: /dev/vda1 78G 69G 5.8G 93% /
install1002: /dev/vda1 177G 123G 46G 73% /
Mentioned in SAL (#wikimedia-operations) [2018-12-13T21:55:47Z] <mutante> Ganeti - creating new 120G virtual disk on install2002 (T211850)
Mentioned in SAL (#wikimedia-operations) [2018-12-14T00:43:14Z] <mutante> install2002 (T211850) restarted instance, created ext4 filesystem on new /dev/vdb, mounted on /mnt/vdb, rsyncing /srv/ to /mnt/vdb/
Mentioned in SAL (#wikimedia-operations) [2018-12-14T01:39:09Z] <mutante> install2002 deleted /srv/ contents,then mounted /mnt/vdb on /srv so same content but now / is used only 7% and /srv 57% (T211850)
root@install2002:/srv# df -hT Filesystem Type Size Used Avail Use% Mounted on udev devtmpfs 10M 0 10M 0% /dev tmpfs tmpfs 401M 5.4M 396M 2% /run /dev/vda1 ext4 78G 5.2G 69G 7% / tmpfs tmpfs 1003M 0 1003M 0% /dev/shm tmpfs tmpfs 5.0M 0 5.0M 0% /run/lock tmpfs tmpfs 1003M 0 1003M 0% /sys/fs/cgroup tmpfs tmpfs 1.0G 0 1.0G 0% /var/lib/nginx /dev/vdb ext4 118G 63G 49G 57% /srv
Lowering prio because there is space now. Not quite closed yet because we may want to do the same thing on 1001 for consistency, not sure. We are also not using the existing space on /srv there.
Ack, in the current scheme of things, the apt repo now lives on apt1001, a dedicated machine.