We failed to track several auto-generated tasks for RAID errors, specially T215892: Degraded RAID on cloudvirt1024
Let's evaluate if we can increase the visibility of these auto-generated tasks by subscribing concrete people to the task, and adding the task to a concrete tag.
This is part of https://wikitech.wikimedia.org/wiki/Incident_documentation/20190213-cloudvps
| Status | Subtype | Assigned | Task | ||
|---|---|---|---|---|---|
| Unknown Object (Task) | |||||
| Resolved | Andrew | T216218 Cloud VPS outage on cloudvirt1024 and cloudvirt1018 due to storage failure | |||
| Resolved | Andrew | T216133 Increase visibility of auto-generated tasks for RAID errors |
We discussed this a little bit yesterday, and T216088 was filed to further discuss this. Help there is welcome :)
I wonder, until we sort all this out, could we perhaps address this underlying problem with a simple Herald rule?
For what is worth, I do have a Herald rule that automatically subscribes me to any degraded RAID ticket for the databases and that proved to be a good way to get my attention, as otherwise monitoring the Operations queue is hard and it is easy to miss things.
Because I had no clue what those are: https://www.mediawiki.org/wiki/Phabricator/Help/Herald_Rules
:)
Icinga alerts for these notifications on cloud/labs hosts were changed to notify the WMCS team in T246130