Page MenuHomePhabricator

Remove the recurring queue threshold alerts
Closed, DeclinedPublic

Description

I think there was a fairly good reason for monitoring this a while back, but now it just produces a bunch of spam on IRC and emails every time PayPal hits us with recurring IPNs, which we can see here.

@Elliott Eggleston what was the reason we set up this alert again?

Also, @Jeff Green @Dallas Wisehaupt, I tried acknowledging the alert in Icinga, but I didn't have permission to do it. Can we get permission for that?

Event Timeline

I raised the alert thresholds to 15K/warning, 20K/critical for now.

@Elliott Eggleston what was the reason we set up this alert again?

The main reason for the queue alerts is to remind us if we turn the consumers off and forget to turn them on again.

@Elliott Eggleston what was the reason we set up this alert again?

The main reason for the queue alerts is to remind us if we turn the consumers off and forget to turn them on again.

I'd suggest we keep the check but raise the thresholds to dire levels so we only get alerts for real problems. If 15/20k has silenced the noise, maybe we're already there?

Re. acking alerts, hopefully that was fixed in T348559.

Sure, sounds good to me @Jgreen, then we can close this ticket.