Page MenuHomePhabricator

Rotate /data/project/hay/crawler.log regularly to make it easier to check for crawler issues
Closed, ResolvedPublic

Description

The log file /data/project/hay/crawler.log is 65M. 65M is not enormous for a log file, but this particular log file is linked from https://tools.wmflabs.org/hay/directory/ as the way to see how the crawler is doing. The data in the file goes back to 2014 and generally the 'interesting' data is at the end of the file (results of the last run or two).

I think this file is the result of a periodic cron job, so it should be relatively easy to rotate or truncate the file periodically (once a month?) and make viewing it online much easier.