I am not sure what's the current status for wikireplicas alerts. I do know they do alert on IRC, but I am not sure if they already page cloud-services-team engineers or not.

That aside i wondered if we should have some general policy on uses granted wmcs-roots i.e. should they be added to the batphone paging group in victorops (im not even sure if wmcs has a batphone group)?

WMCS does not currently have a batphone group, but we have a wmcs team in VictorOps with a multi-step escalation policy, so I believe it could be reasonable to ask members of wmcs-roots to be added to the wmcs group in VictorOps. This is not true at the moment though, as we have a few people in wmcs-roots that are NOT part of the wmcs group in VictorOps.

I am not sure what's the current status for wikireplicas alerts. I do know they do alert on IRC, but I am not sure if they already page cloud-services-team engineers or not.

I believe they do alert on IRC and email, but they are not tagged with wmcs in Prometheus/VictorOps, hence they do not page cloud-services-team engineers. If you can find a recent wikireplica alert I can double check that.

@fnegri keep in mind that wikireplicas do not page for SRE as well. The most recent alert I can think of are the ones related to the last outage T337446 which might have shown lag/replication broken, however I think I caught the first alert (sanitarium) and probably downtime the others to avoid noise. Sanitarium hosts should also be alerting, as they are part of the wiki replicas service (you can see the alert they triggered at https://wikitech.wikimedia.org/wiki/Incidents/2023-05-28_wikireplicas_lag)

I found that alert in Logstash, attached is the alert JSON data from Logstash.

"team": "sre" means that it didn't page WMCS, but shouldn't it have paged SREs as it has "alert_severity": "critical"?

There are two possible reasons:

1. I caught the alert too fast before it even paged.
2. The host is marked as non critical (aka irc-alert only) somewhere else.