during today's outage (T124804) we did not get any Icinga alerts and nothing changed on http://status.wikimedia.org because only wikis in .wikimedia.org were affected and we were still serving the generic portal page.
Since we check HTTP status but do not check for some specific content this went unnoticed by monitoring.
Can we check for specific strings that never change under normal circumstances but are specific to a project and would have caught this?
Checking for specific strings would make sense - standard HTTP tokens or headers perhaps? But beyond that, the user expectation of status.wikimedia.org is that it will report the status of the Wikimedia projects, regardless of whether that is automatically or manually detected. If it's not already possible for operations staff to manually trigger a service outage report on status.wikimedia.org (maybe through a big red button in the WMF offices? ;-) ), then that would be a good thing to add.
Additionally, are there presubmit/integration checks that would have caught this? The builds looked green on push.
Change 290606 had a related patch set uploaded (by Dzahn):
add icinga monitoring for content on commons
So the Icinga part is there now. What i don't know is: Should that be paging now? and todo is watchmouse
I added the same type of check to "watchmouse" too:
http://status.wikimedia.org/8777/438553/
it's a https check on https://commons.wikimedia.org/wiki/Main_Page but additionally for the string "Picture of the Day" and it's green .. all the settings like other things in core services
shows up as "commons.wikimedia.org main page content" on http://status.wikimedia.org/ right now but once refreshed will be "https content - commons" for more consistent naming.
Change 291347 had a related patch set uploaded (by Dzahn):
icinga: make commonts content check critical (paging)