It has been observed multiple times that Icinga alerts can be noisy, especially on IRC and during incidents it can be very distracting. In particular the following should help improving the signal to noise ratio:
Replace host-level IRC alerts with equivalent service-level.
Especially on IRC there's often no need to have notifications for single hosts, e.g: CPU alerts, dpkg broken, etc. These host-level alerts in some cases make sense aggregated (e.g. per cluster) and/or not to be sent on IRC but shown on icinga UI only.
[x] Alerts that page should say so
ATM it is impossible to tell whether a given alert has paged folks, a paging alert indicates a serious issue and a certain level of response expected. Thus explicitly paging alerts will help picking out serious issue (e.g. from IRC)
[stretch] Downtime hosts from IRC
It'll be useful if folks can downtime hosts from IRC in a similar fashion to how we !log for example, useful during incidents since we're on IRC anyways and the icinga ui can be clunky/slow, ditto for logging into icinga host and issuing downtime-host for each host.