Page MenuHomePhabricator

Hourly log rotation for large MW logs
Closed, ResolvedPublic

Description

The daily API log is about 211GB uncompressed, 43GB compressed, and as such it takes a while to find the log entry you're looking for, even with an exact timestamp. T275101 proposes to add a new API log, perhaps with some deduplication of fields between the two logs, which will increase the need for finding correlating log entries. To make this quicker, I propose that we move to hourly rotation for api.log.

In terms of configuration files on mwlog1001:

  • Duplicate the existing section in /etc/logrotate.d/udp2log-mw . Change the filename to /srv/mw-log/api.log, change daily to hourly. I suggest also datehourago -- really we should have had dateyesterday from the outset, and maybe that's a good idea for when we switch to mwlog1002 (T224565), but it's easy enough to change here. The specific section for api.log needs to come after the wildcard. That's simplest if they are in the same file.
  • Move /etc/cron.daily/logrotate to /etc/cron.hourly/logrotate

But I'm not sure how to puppetize that. /etc/logrotate.d/udp2log-mw comes from a shared template in the udp2log module, and I'm not sure of the correct way to rename a cron file. After T275101 there will be at least two affected log files, so it should be possible to specify a list of files that have hourly rotation.

Note that the date format in the archive filenames can be customised with dateformat. The default is -%Y%m%d%H.

Event Timeline

hi @tstarling we can help, how would you like to proceed?

If you propose a puppet change, I can review it and verify that it's working after deployment.

lmata moved this task from Inbox to Backlog on the observability board.

Change 919063 had a related patch set uploaded (by Herron; author: Herron):

[operations/puppet@production] mwlog: rotate api.log hourly

https://gerrit.wikimedia.org/r/919063

Change 919063 merged by Herron:

[operations/puppet@production] mwlog: rotate api.log hourly

https://gerrit.wikimedia.org/r/919063

Looking good, I see the first hourly rotated file on disk

-rw-r--r-- 1 udp2log udp2log  35G May 16 00:23 api.log-20230516
-rw-r--r-- 1 udp2log udp2log  22G May 16 19:00 api.log-2023051618

Change 927187 had a related patch set uploaded (by Herron; author: Herron):

[operations/puppet@production] mwlog: fix mw-log logrotate glob

https://gerrit.wikimedia.org/r/927187

Change 927187 merged by Herron:

[operations/puppet@production] mwlog: fix mw-log logrotate glob

https://gerrit.wikimedia.org/r/927187