We are running low of disk space on the root filesystem of multiple cache nodes in eqsin:
cp5009.eqsin.wmnet: /dev/md0 ext4 9.1G 7.5G 1.2G 87% / cp5008.eqsin.wmnet: /dev/md0 ext4 9.1G 7.6G 1.1G 88% / cp5011.eqsin.wmnet: /dev/md0 ext4 9.1G 7.8G 926M 90% /
A significant amount of such space is taken by logs (and the apt cache):
root@cp5008:/var# du -sh * | grep G 1.7G cache 1.7G log
The bulk of those logs is produced by trafficserver-tls, and is stored on disk under both /var/log/messages as well as /var/log/user.log by rsyslog.
root@cp5008:/var/log# awk '{print $5}' /var/log/user.log | sort | uniq -c | sort -n | tail -3 1285 puppet-agent-cronjob: 28652 prometheus-trafficserver-exporter[30565]: 1488736 trafficserver-tls[18826]:
Considering that trafficserver-tls logs are handled by the systemd journal already, we should not persist them to disk too (and surely not duplicate them to multiple files).