Page MenuHomePhabricator

maps-test* hosts running low on space
Closed, ResolvedPublic

Description

[18:32:20]  <icinga-wm_> PROBLEM - Disk space on maps-test2003 is CRITICAL: DISK CRITICAL - free space: /srv 39008 MB (3% inode=99%)

Other test hosts aren't feeling any better:

maxsem@maps-test2001:~$ rm -rf water-polygons-split-3857* dump.txt
maxsem@maps-test2001:~$ df -h
Filesystem                           Size  Used Avail Use% Mounted on
udev                                  10M     0   10M   0% /dev
tmpfs                                 13G  772M   12G   6% /run
/dev/dm-0                             37G   34G  1.3G  97% /
tmpfs                                 32G  4.0K   32G   1% /dev/shm
tmpfs                                5.0M     0  5.0M   0% /run/lock
tmpfs                                 32G     0   32G   0% /sys/fs/cgroup
/dev/sda1                            268M  100M  155M  40% /boot
/dev/mapper/maps--test2001--vg-data 1008G  957G   76M 100% /srv

Related Objects

Event Timeline

Note that we can't increase the number of inodes on these boxes on the fly (ext4), there's no room left for expansion of the lvms, and I've no idea what could be tossed from the cassandra dirs, maybe nothing.

At this point, the best is probably to trash them and re-image, which we should do anyway to validate once more that re-imaging works and to make sure that the maps-test servers are well aligned with other maps clusters.

Meanwhile, the hosts are at zero / space too and therefore services are down.

Gehel claimed this task.

Resovled by reimaging the maps-test servers