Change #1192183 had a related patch set uploaded (by Andrea Denisse; author: Andrea Denisse):

[operations/alerts@master] mediawiki-engineering: Add API Gateway alerts with thresholds

https://gerrit.wikimedia.org/r/1192183

Hi @Mooeypoo !

I've finished the patch and the tests, now measuring by all of the clusters meaning all the responses are accounted for when measuring the percentages and the alerts would trigger as soon as those conditions are met. https://gerrit.wikimedia.org/r/c/operations/alerts/+/1192183/

I added severity high to all of them but please let me know if it should be critical or page so I can adjust the patch accordingly. I can also add other thresholds and severity like 500 errors exceed 0.5% being high and 500 errors exceed 1% being critical. Thanks!

Thank you!!

I just consulted with @HCoplin-WMF about this -- Let's do "Critical" for the 50x errors, and "High" for the 4xx errors.

Change #1194321 had a related patch set uploaded (by Andrea Denisse; author: Andrea Denisse):

[operations/puppet@production] alertmanager: Add support for team mentions on the Slack template

https://gerrit.wikimedia.org/r/1194321

Change #1192183 abandoned by Andrea Denisse:

[operations/alerts@master] mediawiki-engineering: Add REST API alerts with thresholds

Reason:

Hi folks, I'm abandoning this as I've set-up these alerts from Grafana. https://grafana-rw.wikimedia.org/alerting/list?search=namespace%3A%22MediaWiki+Engineering+Team%22+group%3A%22ResourceLoader+Alerts%22

https://gerrit.wikimedia.org/r/1192183