Change 304263 had a related patch set uploaded (by Thcipriani):
Labs: Shinken alert for beta error rate

https://gerrit.wikimedia.org/r/304263

The log errors project is for actual errors, not infrastructure to deal with them.

Actually there's an "Improvements" column in Wikimedia-production-error for exactly this sort of thing :)

Mea culpa :)

Change 304263 merged by Dzahn:
Labs: Shinken alert for beta error rate

https://gerrit.wikimedia.org/r/304263

That is nice! https://gerrit.wikimedia.org/r/304263 managed to get an alarm overnight on beta-cluster alerts https://lists.wikimedia.org/pipermail/betacluster-alerts/2016-November/023912.html

Subject: [Betacluster-alerts] PROBLEM alert - Graphite Labs/Mediawiki Error Rate is CRITICAL
Notification Type: PROBLEM

Service: Mediawiki Error Rate
Host: Graphite Labs
Address: graphite-labs.wikimedia.org
State: CRITICAL

Date/Time: Thu 24 Nov 00:07:02 UTC 2016

Additional Info:

Graphite shows the error rate being 0 for the last 24 hours, so the alarm should NOT have triggered

https://graphite-labs.wikimedia.org/render/?width=586&height=308&_salt=1479978264.585&from=-1days&target=transformNull(logstash.rate.mediawiki.exception.ERROR.rate%2C0)&target=transformNull(logstash.rate.mediawiki.fatal.ERROR.rate%2C0)

I guess the check_graphite_threshold command fails on the Shinken host.

Change 325122 had a related patch set uploaded (by Alex Monk):
Follow-up I863367b8, Ic9db0829: These two commits conflicted

https://gerrit.wikimedia.org/r/325122

Change 325122 merged by Dzahn:
Follow-up I863367b8, Ic9db0829: These two commits conflicted

https://gerrit.wikimedia.org/r/325122