Page MenuHomePhabricator

maps-test* hosts running low on space
Closed, ResolvedPublic

Description

[18:32:20]  <icinga-wm_> PROBLEM - Disk space on maps-test2003 is CRITICAL: DISK CRITICAL - free space: /srv 39008 MB (3% inode=99%)

Other test hosts aren't feeling any better:

maxsem@maps-test2001:~$ rm -rf water-polygons-split-3857* dump.txt
maxsem@maps-test2001:~$ df -h
Filesystem                           Size  Used Avail Use% Mounted on
udev                                  10M     0   10M   0% /dev
tmpfs                                 13G  772M   12G   6% /run
/dev/dm-0                             37G   34G  1.3G  97% /
tmpfs                                 32G  4.0K   32G   1% /dev/shm
tmpfs                                5.0M     0  5.0M   0% /run/lock
tmpfs                                 32G     0   32G   0% /sys/fs/cgroup
/dev/sda1                            268M  100M  155M  40% /boot
/dev/mapper/maps--test2001--vg-data 1008G  957G   76M 100% /srv

Related Objects

Event Timeline

MaxSem created this task.Sep 28 2016, 1:47 AM
Restricted Application added a subscriber: Aklapper. · View Herald TranscriptSep 28 2016, 1:47 AM

Note that we can't increase the number of inodes on these boxes on the fly (ext4), there's no room left for expansion of the lvms, and I've no idea what could be tossed from the cassandra dirs, maybe nothing.

Gehel added a comment.Sep 29 2016, 8:47 AM

At this point, the best is probably to trash them and re-image, which we should do anyway to validate once more that re-imaging works and to make sure that the maps-test servers are well aligned with other maps clusters.

Meanwhile, the hosts are at zero / space too and therefore services are down.

Gehel closed this task as Resolved.Oct 12 2016, 6:35 PM
Gehel claimed this task.

Resovled by reimaging the maps-test servers