Fix paniclog alert to only sent mails once
Open, MediumPublic
Actions

Assigned To

None

Authored By

	MoritzMuehlenhoff
	Jul 3 2020, 6:34 AM

Description

Currently if there's a paniclog mail for any of our mail servers, the paniclog gets mailed to root@. Virtually all of those are harmless and the effect of day-to-day operation, e.g.

2020-07-02 23:34:59 1jr8jB-0000za-CO spam acl condition: all spamd servers failed

which apparently was caused by a spamassassin restart.

The alert however is re-sent daily until someone manually removes the paniclog file. We should add some systemd timer that removes paniclog files which haven't changed since > 24 hrs to reduce noise

Details

	Subject	Repo	Branch	Lines +/-
	exim4: move daily paniclog rotate from exim4-base to exim4-paniclog	operations/puppet	production	+18 -1
	mx: add paniclog to exim logrotate	operations/puppet	production	+1 -1

Customize query in gerrit

Related Objects
Search...

Status	Assigned	Task
Open	None	T294906 Puppet Improvements
Duplicate	jbond	T265138 Work required to prepare for puppet 7
Resolved	SLyngshede-WMF	T273673 replace all puppet crons with systemd timers
Open	None	T132324 Tracking and Reducing cron-spam to root@
Open	None	T257016 Fix paniclog alert to only sent mails once

Event Timeline

MoritzMuehlenhoff created this task.Jul 3 2020, 6:34 AM

Restricted Application added a subscriber: Aklapper. · View Herald TranscriptJul 3 2020, 6:34 AM

MoritzMuehlenhoff added a parent task: T132324: Tracking and Reducing cron-spam to root@ .Jul 3 2020, 6:34 AM

One idea to avoid causing alerts during spamassassin restarts is have the two MXes act as failback spamd servers for each other. This way when the local spamd goes down for restart exim will try the spamd of the other MX before logging 'all spamd servers failed'

fgiunchedi moved this task from Inbox to Backlog on the observability board.Jul 6 2020, 11:26 AM

fgiunchedi mentioned this in T224399: exim paniclog on $HOST has non-zero size.Jul 6 2020, 11:31 AM

Dzahn added a project: Mail.Jul 8 2020, 6:15 PM

Change 616524 had a related patch set uploaded (by Herron; owner: Herron):
[operations/puppet@production] mx: add paniclog to exim logrotate

https://gerrit.wikimedia.org/r/616524

gerritbot added a project: Patch-For-Review.Jul 27 2020, 2:05 PM

herron triaged this task as Medium priority.Jul 27 2020, 2:05 PM

Change 616524 merged by Herron:
[operations/puppet@production] mx: add paniclog to exim logrotate

https://gerrit.wikimedia.org/r/616524

Maintenance_bot removed a project: Patch-For-Review.Jul 28 2020, 3:10 PM

@herron Seems to work fine, didn't see a paniclog mail today \o/

In T257016#6344879, @MoritzMuehlenhoff wrote:

@herron Seems to work fine, didn't see a paniclog mail today \o/

Actually, the paniclog mail for kubernetes2002 was sent again.

Change 617529 had a related patch set uploaded (by Herron; owner: Herron):
[operations/puppet@production] exim4: move daily paniclog rotate from exim4-base to exim4-paniclog

https://gerrit.wikimedia.org/r/617529

gerritbot added a project: Patch-For-Review.Jul 30 2020, 7:46 PM

In T257016#6348642, @MoritzMuehlenhoff wrote:

In T257016#6344879, @MoritzMuehlenhoff wrote:

@herron Seems to work fine, didn't see a paniclog mail today \o/

Actually, the paniclog mail for kubernetes2002 was sent again.

Ah, yes, only was rotating paniclog on the MXes. Other hosts should be less noisy overall, but might as well rotate the paniclog daily across the fleet.

Change 617529 merged by Herron:
[operations/puppet@production] exim4: move daily paniclog rotate from exim4-base to exim4-paniclog

https://gerrit.wikimedia.org/r/617529

Maintenance_bot removed a project: Patch-For-Review.Aug 3 2020, 6:11 PM

Aklapper added a project: Infrastructure-Foundations.Jun 21 2021, 8:59 PM

lmata edited projects, added SRE Observability; removed observability.Jul 12 2021, 2:39 AM

lmata moved this task from Inbox to Backlog on the SRE Observability board.Jul 15 2021, 4:09 AM

lmata edited projects, added Observability-Alerting; removed SRE Observability.Aug 9 2021, 2:38 AM

lmata moved this task from Inbox to Backlog on the Observability-Alerting board.Aug 10 2021, 3:18 PM