Page MenuHomePhabricator

logrotate failing with $FILE.1.gz: File exists
Open, MediumPublic

Description

We had some hosts that started failing logrotate on many different files with the error $FILE.1.gz: File exists.
Last example from cp1008:

# Email subject:
Cron <root@cp1008> test -x /usr/sbin/anacron || ( cd / && run-parts --report /etc/cron.daily )

# Email content
/etc/cron.daily/logrotate:
error: error creating output file /var/log/syslog.1.gz: File exists
error: error creating output file /var/log/kern.log.1.gz: File exists
error: error creating output file /var/log/auth.log.1.gz: File exists
error: error creating output file /var/log/user.log.1.gz: File exists
error: error creating output file /var/log/debug.1.gz: File exists
error: error creating output file /var/log/messages.1.gz: File exists
error: error creating output file /var/log/update-ocsp-all.log.1.gz: File exists
error: error creating output file /var/log/varnishkafka.log.1.gz: File exists
error: error creating output file /var/cache/varnishkafka/eventlogging.stats.json.1.gz: File exists
error: error creating output file /var/cache/varnishkafka/statsv.stats.json.1.gz: File exists
error: error creating output file /var/cache/varnishkafka/webrequest.stats.json.1.gz: File exists
error: error creating output file /var/log/zerofetch.log.1.gz: File exists
run-parts: /etc/cron.daily/logrotate exited with return code 1

Looking at it those files actually exists and are of size 0. An empty file that is gzipped uses 24 bytes, so an empty .gz file doesn't make any sense, there was an error while creating it.
Those files where created during the previous logrotate run.

The way to fix it is to remove those empty files and run logrotate (or let it run the next day):

sudo find /var/log -name "*.gz" -size 0 -delete
sudo /usr/sbin/logrotate /etc/logrotate.conf

Is not clear why logrotate created empty files, one possibility is disk full, there is some related error in syslog on cp1008.

Event Timeline

Volans created this task.Nov 22 2016, 11:40 AM

Mentioned in SAL (#wikimedia-operations) [2016-11-22T11:42:09Z] <volans> fixed logrotate on cp1008, removed empty created .1.gz files T151314