Page MenuHomePhabricator

Making centrallog syslog easier and faster to work with
Open, MediumPublic

Description

Currently centrallog outputs syslogs from all hosts into a common set of flat files. These files become large, and searching through them for something specific can be time consuming.

To help with this, let's split syslogs off into hostname directories like this:

/srv/syslog/$hostname/syslog.log

Along with that, let's update logrotate to rotate and compress files in the hostname directory so they are easier/faster to find in the future. like this:

/srv/syslog/$hostname/syslog.log-$datestamp.gz

Along with this we'll need to update mtail to look at the new log paths, and automate cleanup of hosts as they leave the fleet.

Event Timeline

herron triaged this task as Medium priority.Jun 5 2020, 4:43 PM
herron created this task.
Restricted Application added a subscriber: Aklapper. · View Herald TranscriptJun 5 2020, 4:43 PM
herron added a comment.Jun 5 2020, 4:43 PM

Adding this patch retroactively (the per-hostname split is already done) https://gerrit.wikimedia.org/r/#/c/operations/puppet/+/601836/

Change 602470 had a related patch set uploaded (by Herron; owner: Herron):
[operations/puppet@production] centrallog: update mtail syslog file locations

https://gerrit.wikimedia.org/r/602470

Change 602470 merged by Herron:
[operations/puppet@production] centrallog: update mtail syslog file locations

https://gerrit.wikimedia.org/r/602470

Change 602764 had a related patch set uploaded (by Herron; owner: Herron):
[operations/puppet@production] centrallog: disable mtail fsnotify, increase fd limit & simplify glob

https://gerrit.wikimedia.org/r/602764

Change 602764 merged by Herron:
[operations/puppet@production] centrallog: disable mtail fsnotify, increase fd limit & simplify glob

https://gerrit.wikimedia.org/r/602764

fgiunchedi moved this task from Inbox to In progress on the observability board.Jun 8 2020, 2:17 PM