Page MenuHomePhabricator

install2002 94% disk usage on "/"
Closed, ResolvedPublic

Description

Showing as an icinga warning

root@install2002:/# df -hT
Filesystem     Type      Size  Used Avail Use% Mounted on
udev           devtmpfs   10M     0   10M   0% /dev
tmpfs          tmpfs     401M   41M  361M  11% /run
/dev/vda1      ext4       78G   69G  5.0G  94% /
tmpfs          tmpfs    1003M     0 1003M   0% /dev/shm
tmpfs          tmpfs     5.0M     0  5.0M   0% /run/lock
tmpfs          tmpfs    1003M     0 1003M   0% /sys/fs/cgroup
tmpfs          tmpfs     1.0G     0  1.0G   0% /var/lib/nginx
1.4G	home
63G	srv
2.5G	var

Related Objects

Event Timeline

Restricted Application added a subscriber: Aklapper. · View Herald TranscriptDec 13 2018, 6:19 AM
Dzahn added a subscriber: Dzahn.Dec 13 2018, 8:38 PM

apt-get clean -> 93%

So it's /srv and there the notable ones by size are: ~ 20G of Junos stuff for netops , ~ 35G /srv/wikimedia/pool the APT repo of which are 20G main and 12G thirdparty. /srv/wikimedia/incoming seems like a potential candidate for cleanup but that's under 1.5G.

Dzahn added a comment.Dec 13 2018, 8:42 PM

It's notable here that install1002 has more space than install2002 does. (should have picked the same size at VM creation?) but also install1002 uses more space (at least the aptrepo part should be automatically synced between the 2, shouldn't it? and everything else is not synced?)

install2002: /dev/vda1 78G 69G 5.8G 93% /

install1002: /dev/vda1 177G 123G 46G 73% /

Mentioned in SAL (#wikimedia-operations) [2018-12-13T21:55:47Z] <mutante> Ganeti - creating new 120G virtual disk on install2002 (T211850)

Dzahn triaged this task as High priority.Dec 13 2018, 10:21 PM

Mentioned in SAL (#wikimedia-operations) [2018-12-14T00:43:14Z] <mutante> install2002 (T211850) restarted instance, created ext4 filesystem on new /dev/vdb, mounted on /mnt/vdb, rsyncing /srv/ to /mnt/vdb/

Mentioned in SAL (#wikimedia-operations) [2018-12-14T01:39:09Z] <mutante> install2002 deleted /srv/ contents,then mounted /mnt/vdb on /srv so same content but now / is used only 7% and /srv 57% (T211850)

Dzahn lowered the priority of this task from High to Medium.Dec 14 2018, 1:41 AM
root@install2002:/srv# df -hT
Filesystem     Type      Size  Used Avail Use% Mounted on
udev           devtmpfs   10M     0   10M   0% /dev
tmpfs          tmpfs     401M  5.4M  396M   2% /run
/dev/vda1      ext4       78G  5.2G   69G   7% /
tmpfs          tmpfs    1003M     0 1003M   0% /dev/shm
tmpfs          tmpfs     5.0M     0  5.0M   0% /run/lock
tmpfs          tmpfs    1003M     0 1003M   0% /sys/fs/cgroup
tmpfs          tmpfs     1.0G     0  1.0G   0% /var/lib/nginx
/dev/vdb       ext4      118G   63G   49G  57% /srv

Lowering prio because there is space now. Not quite closed yet because we may want to do the same thing on 1001 for consistency, not sure. We are also not using the existing space on /srv there.

As 1001 and 2002 are gone this task might be good to close?

MoritzMuehlenhoff closed this task as Resolved.May 11 2020, 12:43 PM
MoritzMuehlenhoff claimed this task.

Ack, in the current scheme of things, the apt repo now lives on apt1001, a dedicated machine.