Thanks to ori that pointed out that cp3040 was finishing the space and the Nginx log was not rotated.
The logrotation of nginx and varnishkafka files on all (I guess) cpXXXX servers is broken since July 1st 2015 because the file permission on the logrotate files are wrong:
0 = root
root@cp1052:/var/log/nginx# ls -lna /etc/logrotate.d/ total 84 drwxr-xr-x 2 0 0 4096 Jan 29 15:48 . drwxr-xr-x 115 0 0 12288 Feb 15 13:33 .. -rw-r--r-- 1 0 0 173 Feb 23 2015 apt -rw-r--r-- 1 0 0 79 Nov 8 2014 aptitude -r--r--r-- 1 0 0 289 Jul 1 2015 confd -rw-r--r-- 1 0 0 232 Feb 22 2015 dpkg -rw-r--r-- 1 0 0 146 Feb 17 2015 exim4-base -rw-r--r-- 1 0 0 126 Feb 17 2015 exim4-paniclog -rw-r--r-- 1 0 0 117 Jan 31 2014 graphite-carbon -rw-r--r-- 1 0 0 129 Mar 9 2015 logster -rw-rw-r-- 1 998 998 289 Mar 11 2015 nginx -r--r--r-- 1 0 0 396 Mar 11 2015 puppet -rw-r--r-- 1 0 0 515 Oct 2 2014 rsyslog -r--r--r-- 1 0 0 394 Jun 18 2015 salt-common -rw-r--r-- 1 0 0 338 Feb 26 2015 varnish -rw-rw-r-- 1 998 998 176 Mar 11 2015 varnishkafka -rw-r--r-- 1 0 0 222 Aug 31 15:33 varnishkafka-eventlogging-stats -rw-r--r-- 1 0 0 210 May 12 2015 varnishkafka-statsv-stats -rw-r--r-- 1 0 0 218 Mar 11 2015 varnishkafka-webrequest-stats
root@cp3040:~# logrotate --debug /etc/logrotate.d/nginx Ignoring /etc/logrotate.d/nginx because of bad file mode. Handling 0 logs
The symptoms is that the files are not rotating, space is filling up on the root partition and the log file are quite big (~3GB on some server):
root@cp3040:/etc/nginx# ls -larth /var/log/nginx/unified.error.log -rw-r----- 1 www-data www-data 2.8G Feb 16 00:38 /var/log/nginx/unified.error.log root@cp3040:/etc/nginx# ls -lah /var/log/varnishkafka.log -rw-r----- 1 root adm 564M Feb 9 16:00 /var/log/varnishkafka.log
We should be careful with the fix, because just fixing the issue will cause ALL servers to rotate the log and start compressing a quite big existing log at the same time (cron.daily runs at 06:25) and might cause CPU overload across the fleet based on the usage around that time.