The instances provided on labs eqiad (such as deployment-bastion.eqiad.wmflabs) have a /var/ of only 2GB. That tends to fill up pretty quickly with log being written there often causing issues.
This is a tracking bug to list all /var/ related issues in labs, such as daemon not compressing or logrotate keeping too much data...
Version: unspecified
Severity: normal
There are two things that can be done to alleviate/fix this issue entirely:
(a) /var/log itself can be made much bigger at need (there is a puppet class for that very purpose).
(b) you can substitute the default /var with a LVM instead. This is harder to do after the fact but not ridiculously so (it does require a reboot to have daemons use the new one though).
Either of those could be used - please ping me on IRC if you want to discuss either in detail and for help deploying them.
This is "fixed" only insofar as there is now more /var to fill before things break; but that provides no guard against unbounded log growth.
Apr 12 20:39:13 <bd808> Krenair: on deployment-mediawiki01 it looks like the biggest disk hog is /var/cache/hhvm/fcgi.hhbc.sq3 (2.3G). Apr 12 20:39:47 <bd808> Krenair: judging by the output of `sqlite3 fcgi.hhbc.sq3 .tables` we aren't cleaning that file up when we deploy new HHVM builds Apr 12 20:40:48 <bd808> a "feature" of hhbc files is that they version the cache to match the exact HHVM build Apr 12 20:41:10 <bd808> so when we deploy a new hhvm binary it starts a fresh cache segment Apr 12 20:41:22 <bd808> but nothing automatically cleans up the old cache data Apr 12 20:43:09 <bd808> it looks like we have 9 versions of each table in that particular hhbc cache Apr 12 20:46:38 <bd808> !log Cleaned up large hhbc cache file on deployment-medaiwiki01 via `sudo service hhvm stop; sudo rm /var/cache/hhvm/fcgi.hhbc.sq3; sudo service hhvm start` Apr 12 20:47:18 <bd808> !log Cleaned up large hhbc cache file on deployment-medaiwiki02 via `sudo service hhvm stop; sudo rm /var/cache/hhvm/fcgi.hhbc.sq3; sudo service hhvm start` Apr 12 20:47:47 <bd808> !log Cleaned up large hhbc cache file on deployment-medaiwiki03 via `sudo service hhvm stop; sudo rm /var/cache/hhvm/fcgi.hhbc.sq3; sudo service hhvm start` Apr 12 20:48:13 <bd808> That got back about 2G on each MW server
That was a transient issue due to labs instances having a /var of just 2G bytes which is really not enough. Got solved by fine tuning log spam and I believe most instances now have a way larger /var.
There is still the diamond probe that fill in apache2 log but that is really a low issue ( T74175 ).