Found while looking into T170521: deployment-logstash2 out of disk space
$ ssh deployment-eventlogging03.eqiad.wmflabs df -h Filesystem Size Used Avail Use% Mounted on udev 2.0G 12K 2.0G 1% /dev tmpfs 396M 424K 395M 1% /run /dev/vda1 18G 18G 0 100% / none 4.0K 0 4.0K 0% /sys/fs/cgroup none 5.0M 0 5.0M 0% /run/lock none 2.0G 0 2.0G 0% /run/shm none 100M 0 100M 0% /run/user /dev/mapper/vd-second--local--disk 21G 1.4G 19G 7% /srv
I have dropped logs in var/log and also srv/eventlogging but still there is a lot of space consumed by db that i have not been able to drop
Thanks to help from some releng folks looks like this are getting better:
root@deployment-eventlogging03:~# df -h
Filesystem Size Used Avail Use% Mounted on
udev 2.0G 12K 2.0G 1% /dev
tmpfs 396M 404K 395M 1% /run
/dev/vda1 18G 14G 3.3G 81% /
none 4.0K 0 4.0K 0% /sys/fs/cgroup
none 5.0M 0 5.0M 0% /run/lock
none 2.0G 0 2.0G 0% /run/shm
none 100M 0 100M 0% /run/user
/dev/mapper/vd-second--local--disk 21G 1.6G 19G 8% /srv
overflow 1.0M 4.0K 1020K 1% /tmp
There's around 700MB in /var/lib/mysql.5.5.46-0ubuntu0.14.04.2 that hasn't been touched since 2015
Dropped tables and logs and run apt-get clean and apt-get autoclean, we are now at 80% space. Restarted eventloging again.