Page MenuHomePhabricator

Prometheus cronspam
Closed, ResolvedPublic

Description

From daily root@ cronspam:

Cron Daemon root@prometheus1001.eqiad.wmnet via wikimedia.org 
7:25 AM (6 hours ago)

to root 
/etc/cron.daily/logrotate:
error: prometheus-node-exporter:1 duplicate log entry for /var/log/prometheus/prometheus-node-exporter.log

On one prometheus host:

elukey@prometheus1001:/etc/logrotate.d$ cat prometheus
/var/log/prometheus/*.log
{
    rotate 4
    daily
    missingok
    notifempty
    delaycompress
    compress
    sharedscripts
    postrotate
        service rsyslog rotate >/dev/null 2>&1 || true
    endscript
}
elukey@prometheus1001:/etc/logrotate.d$ cat prometheus.dpkg-dist
/var/log/prometheus/prometheus.log {
    weekly
    rotate 10
    copytruncate
    compress
    delaycompress
    notifempty
    missingok
}
elukey@prometheus1001:/etc/logrotate.d$ cat prometheus-node-exporter
/var/log/prometheus/prometheus-node-exporter.log {
    weekly
    rotate 10
    copytruncate
    compress
    delaycompress
    notifempty
    missingok
}

So the .dpkg-dist should not be a problem and we can delete them, but the prometheus and prometheus-node-exporter are trying to handle the same files.

Event Timeline

Change 322724 had a related patch set uploaded (by Filippo Giunchedi):
prometheus: logrotate only server.log

https://gerrit.wikimedia.org/r/322724

Change 322724 merged by Filippo Giunchedi:
prometheus: logrotate only server.log

https://gerrit.wikimedia.org/r/322724

Should be fixed by the reviews above, waiting tomorrow to confirm, thanks!