Page MenuHomePhabricator

Setup an easy to use logrotate based system for rotating tools logs
Closed, DuplicatePublic


Currently they can go *really* large, and there's no easy way to fix this. Having a way for tool authors to easily setup logrotate would be nice.

Ideally, they should be able to create a rotatelogs.conf in their tools' homedir, and have things automatically be taken care of.

Version: unspecified
Severity: normal



Event Timeline

bzimport raised the priority of this task from to Needs Triage.Nov 22 2014, 3:11 AM
bzimport added a project: Toolforge.
bzimport set Reference to bz66623.
  • Bug 46471 has been marked as a duplicate of this bug. ***

I haven't tested it, but I assume SGE only lets go of the file descriptors when the job is not running. So any log rotation would require a job restart with at least a few seconds of downtime.

Also, as log rotater and job would probably not be on the same host, it might a good idea to compress only .2ff.s, i. e. let the rotation from .out to .out.1 be just a simple rename to avoid too much hassle.

scfc triaged this task as Lowest priority.Apr 7 2015, 5:04 AM

logrotate can truncate instead of move-and-replace, which SGE seems to be able to cope with. However, Yuvi is planning to work on logstash-like solutions soon(tm), so we might let this be anyway.

I've fiddled a bit more with this; the following works correctly:

valhallasw@tools-bastion-01:~$ cat testrotate
/home/valhallasw/echoforever.out {
    size 100
    maxage 7

where size should probably be ~10M (maybe 100M), not 100 bytes ;-)

valhallasw@tools-bastion-01:~$ logrotate -v -s ./statefile testrotate
reading config file testrotate

Handling 1 logs

rotating pattern: /home/valhallasw/echoforever.out  100 bytes (no old logs will be kept)
empty log files are rotated, old logs are removed
considering log /home/valhallasw/echoforever.out
  log needs rotating
rotating log /home/valhallasw/echoforever.out, log->rotateCount is 0
dateext suffix '-20160209'
glob pattern '-[0-9][0-9][0-9][0-9][0-9][0-9][0-9][0-9]'
glob finding old rotated logs failed
copying /home/valhallasw/echoforever.out to /home/valhallasw/echoforever.out-20160209
truncating /home/valhallasw/echoforever.out
compressing log with: /bin/gzip
valhallasw@tools-bastion-01:~$ ls -l echoforever.*
-rw------- 1 valhallasw wikidev 2581 Feb  9 20:13 echoforever.err
-rw------- 1 valhallasw wikidev  609 Feb  9 20:13 echoforever.out
-rw------- 1 valhallasw wikidev   96 Feb  9 20:12 echoforever.out-20160209.gz
-rwxr-xr-x 1 valhallasw wikidev   72 Feb  9 20:07

However, logrotate needs to run as the same user as the owner of the log files, which makes it slightly harder to implement in a tool labs setting.

@MusikAnimal was asking about log rotation for access.log today. Would that be easier to setup or the same sort of hassle?

We're using a simple bash script for xtools, which I've now added to pageviews:

for logfile in *.err *.out *.log
  tail -c 100000 $logfile > temp.$$; mv temp.$$ $logfile

Then calling via cron job with jlocal. Not sure if this helps at all.

@MusikAnimal: No, this will keep the old, huge files around (with no name) until the job that is holding them open terminates, and that job will continue to output its errors and warnings to these files with no names and not the new ones which will be empty forever.

@valhallasw's solution doesn't do that, but (cf. logrotate(8), emphasis mine):

       Truncate the original log file to zero size in place after creating a copy, instead of moving the old log file and optionally creating a new one.  It
       can be used when some program cannot be told to close its logfile and thus might continue writing (appending) to the previous log file forever.  Note
       that  there is a very small time slice between copying the file and truncating it, so some logging data might be lost.  When this option is used, the
       create option will have no effect, as the old log file stays in place.

Confirming my interest for tools.heritage, where our chatty logs grow and grow :)