In T256168#11960239, @taavi wrote:

In T256168#11960221, @bd808 wrote:

On my side we are still missing any notification if the sync jobs break. I found a good blog post on using OnFailure stanzas with systemd units over the weekend that actually seems like a promising direction. I would like email and irc yelling so we don't miss things getting messed up.

You don't need anything on the systemd unit side for that, as the node-exporter provided node_systemd_unit_state metric can be used already.

I moved on from this advice without comment before when @dancy suggested an email alerting method that was relatively easy to add to the systemd unit.

Using prometheus + alert manger would give us the ability to alert on IRC and other alertmanager sinks. I currently am in need of some hand holding by subject matter experts to understand a few things:

• Is there already a prometheus instance collecting the node_systemd_unit_state metric for the wmf-beta-update-all.service unit on deployment-deploy04.deployment-prep.eqiad1.wikimedia.cloud?
  • If not where do I start to make that part happen?

• Is there an example of a node_systemd_unit_state metric being used in a grafana dashboard to show job stability?

• Is there an example of a node_systemd_unit_state metric being used in an alertmanager alert?

• Is T315695: Add basic MediaWiki/web site up alerting to the Beta Cluster still the best documentation/example for adding alertmanager monitoring for a Cloud VPS project without its own alerting stack?

In T432364#12129373, @bd808 wrote:

• Is there already a prometheus instance collecting the node_systemd_unit_state metric for the wmf-beta-update-all.service unit on deployment-deploy04.deployment-prep.eqiad1.wikimedia.cloud?

Yes, the central metricsinfra one: https://prometheus.wmcloud.org/graph?g0.expr=node_systemd_unit_state%7Bproject%3D%22deployment-prep%22%2Cstate%3D%22failed%22%2Cname%3D%22wmf-beta-update-all.service%22%7D&g0.tab=0&g0.stacked=0&g0.range_input=1h&g0.max_source_resolution=0s&g0.deduplicate=1&g0.partial_response=1&g0.store_matches=%5B%5D&g0.engine=prometheus&g0.analyze=0&g0.tenant=

• Is there an example of a node_systemd_unit_state metric being used in a grafana dashboard to show job stability?

I am not immediately aware of one.

• Is there an example of a node_systemd_unit_state metric being used in an alertmanager alert?

MariaDB [prometheusconfig]> select *  from alerts where id = 31\G
*************************** 1. row ***************************
         id: 31
 project_id: 10
       name: MaintainProjectsBroken
       expr: node_systemd_unit_state{project="metricsinfra",name="metricsinfra-maintain-projects.service",state="failed"} == 1
   duration: 15m
   severity: warning
annotations: {"summary": "metricsinfra failing to update projects", "runbook": "https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/MetricsinfraMaintainProjectsBroken"}

• Is T315695: Add basic MediaWiki/web site up alerting to the Beta Cluster still the best documentation/example for adding alertmanager monitoring for a Cloud VPS project without its own alerting stack?

https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Monitoring#Monitoring_for_Cloud_VPS is the canonical documentation. No comment whether that's better or worse for alerts specifically than the examples in that task.