Scope: The cloud hosts in production and perhaps toolschecker, the internal cloud observability is happening on another task.
There's a template for onboarding into alertmanager, which currently has Phabricator integrations as well as IRC and email/VO. Shall we go try to set that up?
From there, we could use better definition around alerting, when we alert and what to do when an alert comes in (so this is also a documentation task).
@dcaro proposed this general structure
> Needs action now -> page + task
> Needs action -> task
> Does not need action -> to logging system (probably not an alert)
In general, icinga allows runbooks to be included with alerts, and a review of our current alerts to add the right runbooks would definitely help. The systemd alerts are also a thing, which is on the subtask.