Page MenuHomePhabricator

improve cron spam visibility
Open, LowPublic

Description

right now cron spam is mostly unlooked at, however there might be real issues lurking

I think one of the problems is that there is no aggregation/deduplication so the same issue ends up with many emails and thus ignored, one way to tackle this would be to provide aggregation/deduplication so the real issue is easy to spot (and possibly fix) for example experimenting with https://github.com/mediacore/raven-cron would be a good start. In short cron wouldn't be reporting output via mail but rather via sentry which would then collate based on e.g. the command line

Event Timeline

fgiunchedi updated the task description. (Show Details)
fgiunchedi raised the priority of this task from to Needs Triage.
fgiunchedi added a project: ops-core.
fgiunchedi changed Security from none to None.
fgiunchedi added a subscriber: fgiunchedi.
fgiunchedi edited projects, added acl*sre-team; removed ops-core.Dec 18 2014, 9:02 AM
fgiunchedi triaged this task as Low priority.Dec 22 2014, 10:02 AM

also note that to test this easily we could either a dummy mailbox to receive cronspam and turn them into sentry messages, this way existing mails wouldn't be touched

@Volans This seems to be what you mentioned in last monitoring meeting when you suggested an Icinga alert for cronspam.

Volans added a project: observability.
Volans added a subscriber: faidon.

@Dzahn thanks for pointing this out, I've merged in as duplicate the other task I had opened.