Now that we have migrated the tools to the new cluster we have some overhead to do processing for something like log rotation. This is going to be relatively primitive but at the moment I have been forced to say we have no option for this in the case of many runaway logs. A small practical solution will allow us to focus on the actual intended central logging outcome instead of constantly auditing and mitigating logs that could easily be truncated on a schedule.
Basic premise:
- Create a 'logs' directory in every tools directly (by default and retroactively)
- Handle files ending in .log and .err (keep a months worth unless x size type of thing. Basically, allow us to keep the lights on and stop saturating the Tools share)
- Make jsub put .out and .err into 'logs' as .log and .err
- Make other services were we control the default output path put logs in this dir
- Be able to offload cases where a user has no ability to manage their own logs to this programattic mechanism