Logrotate on mw servers sometimes fails with errors like:
/etc/cron.daily/logrotate: error: error opening /var/log/hhvm/error.log-20160624: Permission denied run-parts: /etc/cron.daily/logrotate exited with return code 1
After a bit of digging with Joe:
- logfiles were owned by root:adm rather than www-data:www-data
- hhvm's logrotate settings contain rotate 90 (files older than 90 days are deleted)
- The root:adm logfiles' stat metadata suggest that they were created months ago during a known maintenance or that they are related to new reimaged hosts (the ones being re-installed with Debian Jessie).
I chowned the current wrong files across all the app/api servers but we'd need to put in place a permanent solution for new installs, like applying the HHVM Rsyslog config before installing the package.