Page MenuHomePhabricator

improve cron spam visibility
Open, LowPublic


right now cron spam is mostly unlooked at, however there might be real issues lurking

I think one of the problems is that there is no aggregation/deduplication so the same issue ends up with many emails and thus ignored, one way to tackle this would be to provide aggregation/deduplication so the real issue is easy to spot (and possibly fix) for example experimenting with would be a good start. In short cron wouldn't be reporting output via mail but rather via sentry which would then collate based on e.g. the command line

Event Timeline

fgiunchedi raised the priority of this task from to Needs Triage.
fgiunchedi updated the task description. (Show Details)
fgiunchedi added a project: ops-core.
fgiunchedi changed Security from none to None.
fgiunchedi added a subscriber: fgiunchedi.

also note that to test this easily we could either a dummy mailbox to receive cronspam and turn them into sentry messages, this way existing mails wouldn't be touched

@Volans This seems to be what you mentioned in last monitoring meeting when you suggested an Icinga alert for cronspam.

Volans added a project: observability.
Volans added a subscriber: faidon.

@Dzahn thanks for pointing this out, I've merged in as duplicate the other task I had opened.

An option we discussed recently was to ingest mail generated by the servers into Logstash by either pulling events from a mailbox or piping off events at the mail servers. Once in ES, queries could be run and aggregated emails generated as a daily report and/or alerts generated via log alerting.

@herron has asked OIT if a bot inbox is possible.

Ingesting events from a mailbox will also help T230835. If the mx pipe option is chosen, it wouldn't be too difficult to adapt to that paradigm.