The log file /data/project/hay/crawler.log is 65M. 65M is not enormous for a log file, but this particular log file is linked from https://tools.wmflabs.org/hay/directory/ as the way to see how the crawler is doing. The data in the file goes back to 2014 and generally the 'interesting' data is at the end of the file (results of the last run or two).

I think this file is the result of a periodic cron job, so it should be relatively easy to rotate or truncate the file periodically (once a month?) and make viewing it online much easier.