Currently they can go *really* large, and there's no easy way to fix this. Having a way for tool authors to easily setup logrotate would be nice.
Ideally, they should be able to create a rotatelogs.conf in their tools' homedir, and have things automatically be taken care of.
Version: unspecified
Severity: normal
I haven't tested it, but I assume SGE only lets go of the file descriptors when the job is not running. So any log rotation would require a job restart with at least a few seconds of downtime.
Also, as log rotater and job would probably not be on the same host, it might a good idea to compress only .2ff.s, i. e. let the rotation from .out to .out.1 be just a simple rename to avoid too much hassle.
logrotate can truncate instead of move-and-replace, which SGE seems to be able to cope with. However, Yuvi is planning to work on logstash-like solutions soon(tm), so we might let this be anyway.
I've fiddled a bit more with this; the following works correctly:
valhallasw@tools-bastion-01:~$ cat testrotate
/home/valhallasw/echoforever.out {
compress
copytruncate
dateext
size 100
maxage 7
missingok
}where size should probably be ~10M (maybe 100M), not 100 bytes ;-)
valhallasw@tools-bastion-01:~$ logrotate -v -s ./statefile testrotate reading config file testrotate Handling 1 logs rotating pattern: /home/valhallasw/echoforever.out 100 bytes (no old logs will be kept) empty log files are rotated, old logs are removed considering log /home/valhallasw/echoforever.out log needs rotating rotating log /home/valhallasw/echoforever.out, log->rotateCount is 0 dateext suffix '-20160209' glob pattern '-[0-9][0-9][0-9][0-9][0-9][0-9][0-9][0-9]' glob finding old rotated logs failed copying /home/valhallasw/echoforever.out to /home/valhallasw/echoforever.out-20160209 truncating /home/valhallasw/echoforever.out compressing log with: /bin/gzip
valhallasw@tools-bastion-01:~$ ls -l echoforever.* -rw------- 1 valhallasw wikidev 2581 Feb 9 20:13 echoforever.err -rw------- 1 valhallasw wikidev 609 Feb 9 20:13 echoforever.out -rw------- 1 valhallasw wikidev 96 Feb 9 20:12 echoforever.out-20160209.gz -rwxr-xr-x 1 valhallasw wikidev 72 Feb 9 20:07 echoforever.sh
However, logrotate needs to run as the same user as the owner of the log files, which makes it slightly harder to implement in a tool labs setting.
@MusikAnimal was asking about log rotation for access.log today. Would that be easier to setup or the same sort of hassle?
We're using a simple bash script for xtools, which I've now added to pageviews:
for logfile in *.err *.out *.log do tail -c 100000 $logfile > temp.$$; mv temp.$$ $logfile done
Then calling via cron job with jlocal. Not sure if this helps at all.
@MusikAnimal: No, this will keep the old, huge files around (with no name) until the job that is holding them open terminates, and that job will continue to output its errors and warnings to these files with no names and not the new ones which will be empty forever.
@valhallasw's solution doesn't do that, but (cf. logrotate(8), emphasis mine):
copytruncate
Truncate the original log file to zero size in place after creating a copy, instead of moving the old log file and optionally creating a new one. It
can be used when some program cannot be told to close its logfile and thus might continue writing (appending) to the previous log file forever. Note
that there is a very small time slice between copying the file and truncating it, so some logging data might be lost. When this option is used, the
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
create option will have no effect, as the old log file stays in place.