rsyslog disk spool files filled the filesystem on thanos-fe2001
Closed, ResolvedPublic
Actions

Assigned To

Authored By

	fgiunchedi
	May 16 2022, 1:41 PM

Description

Over the weekend thanos-fe2001 ran out of disk space on /. Turns out /var/spool/rsyslog/ was full of queue files such as:

-rw------- 1 root root 1049237 May 16 11:14 centrallog1001.eqiad.wmnet:6514.00093952
-rw------- 1 root root  606208 May 16 12:11 centrallog2002.codfw.wmnet:6514.00000977

The same thing was happening on thanos-fe2002, though the filesystem wasn't full yet. It looked like the transport to centrallog hosts was stuck somehow, I'm assuming post-reboot for Bullseye upgrades. I've restarted rsyslog on thanos-fe2002 and it looks like it is flushing its queue now.

Details

	Project	Branch	Lines +/-	Subject
	operations/puppet	production	+7 -0	rsyslog: bound disk-assisted queues

Customize query in gerrit

Event Timeline

fgiunchedi created this task.May 16 2022, 1:41 PM

Restricted Application added a subscriber: Aklapper. · View Herald TranscriptMay 16 2022, 1:41 PM

Also thanos-fe2003 doesn't seem to be affected (i.e. no spool files), for 2001 and 2002 at startup we also have this:

thanos-fe2001/syslog.log-20220512.gz:May 11 10:09:28 thanos-fe2001 rsyslogd: fwd_centrallog1001.eqiad.wmnet:6514 queue[DA]: queue files exist on disk, re-starting with 1673 messages. This will keep the disk queue file open, details: https://rainer.gerhards.net/2013/07/rsyslog-why-disk-assisted-queues-keep-a-file-open.html [v8.2102.0 try https://www.rsyslog.com/e/0 ]
thanos-fe2002/syslog.log:May 11 10:29:30 thanos-fe2002 rsyslogd: fwd_centrallog2002.codfw.wmnet:6514 queue[DA]: queue files exist on disk, re-starting with 384 messages. This will keep the disk queue file open, details: https://rainer.gerhards.net/2013/07/rsyslog-why-disk-assisted-queues-keep-a-file-open.html [v8.2102.0 try https://www.rsyslog.com/e/0 ]

herron added a subscriber: herron.May 18 2022, 2:12 PM

fgiunchedi moved this task from Backlog to Doing on the User-fgiunchedi board.May 20 2022, 6:56 AM

Change 799870 had a related patch set uploaded (by Filippo Giunchedi; author: Filippo Giunchedi):

[operations/puppet@production] rsyslog: bound disk-assisted queues

https://gerrit.wikimedia.org/r/799870

gerritbot added a project: Patch-For-Review.May 26 2022, 9:56 AM

Change 799870 merged by Filippo Giunchedi:

[operations/puppet@production] rsyslog: bound disk-assisted queues

https://gerrit.wikimedia.org/r/799870

Mentioned in SAL (#wikimedia-operations) [2022-06-07T08:26:32Z] <godog> set on-disk max queue size for rsyslog fleet wide - T308439

Maintenance_bot removed a project: Patch-For-Review.Jun 7 2022, 8:30 AM

Change is merged and deploying, optimistically resolving

lmata moved this task from Inbox to Done on the SRE Observability (FY2021/2022-Q4) board.Jun 9 2022, 7:39 PM

rsyslog disk spool files filled the filesystem on thanos-fe2001Closed, ResolvedPublicActions

Description

Details

Event Timeline

rsyslog disk spool files filled the filesystem on thanos-fe2001
Closed, ResolvedPublic
Actions