In the last Gerrit incident T423027: 2026-04-12 Gerrit Outage (was: DiskSpace) CI was stuck for several hours. This was noticed by people in #mediawiki_security and was resolved by restarting Gerrit manually.

There should be alerts when Zuul, Gearman, Jenkins, (?) lost it's connection and no more jobs are scheduled.

In the Zuul pipelines business dashboard a gap is visible.

Also the all projects zuul overview dashboard shows a clear gap.

Same with the Zuul: job All in pipeline All status All dashboard

So we might be able to use one of those metrics for an alert. I'm not sure if zuul exports any directly connected metrics liked failed/active ssh sessions.

Also it looks like a few jobs are scheduled even on weekends. However there are some periods with almost 10 hours without any job end of December for example. So we might need some kind of heartbeat job or find better metrics.