T119736 showed that we sometime fail to recognize the severity of bugs that have a substantial impact on users. To make sure nothing slips through the cracks, we should have monitoring and alerting of small set of key "business" metrics. Namely:
See also:
Related incident: https://wikitech.wikimedia.org/wiki/Incident_documentation/20160915-MediaWiki
| Project | Branch | Lines +/- | Subject | |
|---|---|---|---|---|
| operations/puppet | production | +8 -0 | Add alerting for MediaWiki exceptions and fatals |
| Status | Subtype | Assigned | Task | ||
|---|---|---|---|---|---|
| Resolved | akosiaris | T140942 Tracking: Monitoring and alerts for "business" metrics | |||
| Resolved | Joe | T146090 High failure rate of account creation should trigger an alarm / page people | |||
| Resolved | • Gilles | T146125 MediaWiki load time regression should trigger an alarm / page people | |||
| Resolved | • Gilles | T153166 Set up Grafana alerts for Web Performance metrics | |||
| Resolved | • Gilles | T153041 Alerts using WebPageTest | |||
| Resolved | Peter | T163420 Restructure WebPageTest Grafana alerts | |||
| Declined | • Gilles | T153167 Configure Grafana for sending e-mail | |||
| Resolved | Peter | T153168 Send Grafana alerts to IRC #wikimedia-perf | |||
| Resolved | • Gilles | T153169 Grafana alerts for Navigation Timing and related | |||
| Resolved | aaron | T153170 Grafana alerts for save timing | |||
| Resolved | • Gilles | T153171 Grafana alerts for ResourceLoader | |||
| Declined | None | T153978 Use WebDav for dashboard images in alerts |
There are some graphite metrics for authn things: https://grafana.wikimedia.org/dashboard/db/authentication-metrics
One thing we noticed about login failures especially during the SessionManager deployment is that this can skew upwards to crazy levels due to just one bot that is retrying to login in an infinite loop. I think the rate of login success is a bit more stable.
There is a filter for login interface (web / API). There is no guarantee some bot wouldn't use the web interface, but in practice filtering out API queries gives pretty predictable results.
In the case of T119736 the existing metrics would not have been helpful because an exception in the middle of the login process is not counted as failure. Filed that as T140943. (As for successful logins, there wasn't any visible dip. I suspect people tried to log in several times so 1K login failures per day translates to much less than 1K missing successful logins per day, and so isn't really visible over the background of ~10K successful logins.)
FWIW I think the obvious thing to alert on in this case would have been the thousands of exceptions per day in production.
It seems like T117470 would have helped as well (compare frequency of T119736 and frequency of CAS errors).
Change 300327 had a related patch set uploaded (by Ori.livneh):
Add alerting for MediaWiki exceptions and fatals
Thinking about this more, not sure if login/signup metrics are worth the effort. One of the strengths of Wikimedia is the strong connections between developers and power users so universal problems like login breaking completely would be reported very quickly. And less universal problems would not be detected, e.g. T119736 is not visible at all in the login stats.
What might be more helpful is marking certain pages or code sections as mission critical and alerting when exceptions are thrown from there.
Paging people if there are exceptions thrown on Special:Login or Special:Preferences might be worthwhile, yes.
Account creation got broken entirely for 18 hours last week despite metrics being available. I have filled T146090 to get it to send an alarm/page or whatever.
The alert feature introduced in the recent Grafana update (T152473) could be of interest for this. To quote some marketing copy from http://grafana.org/blog/2016/12/12/grafana-4.0-stable-release/ : "Alerting is a really revolutionary feature for Grafana. It transforms Grafana from a visualization tool into a truly mission critical monitoring tool. The alert rules are very easy to configure using your existing graph panels and threshold levels can be set simply by dragging handles to the right side of the graph. The rules will continually be evaluated by grafana-server and notifications will be sent out [e.g. via email] when the rule conditions are met." I understand the Performance team has started to try this out already.
I've been trying out alerts for a while, let me write down a summary the coming days.