Page MenuHomePhabricator
Create Task
Maniphest T423035

Enable paging for Gerrit (was: Gerrit outage didn't page until 4.5 hours after the first alert)
Closed, ResolvedPublic
Actions

Description

T423027: 2026-04-12 Gerrit Outage (was: DiskSpace) was filed at 10:08 UTC and the first user report and became an outage at around 14:15 UTC. It didn't trigger a paging alert until

14:38:51 <+jinxer-wm> FIRING: [2x] ATSBackendErrorsHigh: ATS: elevated 5xx errors from gerrit.discovery.wmnet in eqiad #_page - https://wikitech.wikimedia.org/wiki/Apache_Traffic_Server#Debugging - https://alerts.wikimedia.org/?q=alertname%3DATSBackendErrorsHigh

Note: It was also rapidly rising in disk space from around midnight UTC so there was a 10 hour gap where something wrong could have been detected even earlier.

That's around 15 minutes after the first user report and 4.5 hours after automated monitoring detected a problem. It probably should have gone off a bit louder given Gerrit is a fairly critical part of Infrastructure and it caused secondary alerting from at least authdns-update failing and CI issues.

Details

Related Changes in Gerrit:
SubjectRepoBranchLines +/-
gerrit: add paging blackbox checkoperations/puppetproduction+14 -0
Customize query in gerrit

Related Objects
Search...

Event Timeline

RhinosF1 created this task.Apr 12 2026, 2:45 PM
IKhitron subscribed.Apr 12 2026, 2:46 PM
A_smart_kitten added a comment.Apr 12 2026, 2:48 PM

For the record, T423027 was originally automatically opened to report:

* **summary**: Disk space gerrit2003:9100:/ 3.822% free
RhinosF1 updated the task description. (Show Details)Apr 12 2026, 2:53 PM
Jdforrester-WMF subscribed.Apr 12 2026, 5:35 PM
Jelto subscribed.Apr 13 2026, 7:17 AM

thank you for opening this task. For additional context: paging alerts for Gerrit were also previously discussed in T365148.

Jelto mentioned this in T423027: 2026-04-12 Gerrit Outage (was: DiskSpace).Apr 13 2026, 9:20 AM
Novem_Linguae subscribed.Apr 13 2026, 9:22 AM
A_smart_kitten removed a parent task: T423027: 2026-04-12 Gerrit Outage (was: DiskSpace).Apr 13 2026, 9:50 AM

(rm'ing as a subtask, as iiuc this is a follow-up task rather than something that needs to be done before the incident can be resolved)

LSobanski closed this task as a duplicate of Restricted Task.Apr 13 2026, 3:52 PM
LSobanski subscribed.Thu, Apr 16, 4:20 PM

Reopening this task after all as it's more relevant at this time.

LSobanski reopened this task as Open.Thu, Apr 16, 4:20 PM
LSobanski merged a task: Restricted Task.
LSobanski added subscribers: ssingh, ABran-WMF, ArielGlenn and 7 others.
LSobanski triaged this task as Medium priority.Thu, Apr 16, 4:23 PM

Quoting @Jelto from T365148:

I'm adding the Sustainability (Incident Followup) tag because of the last incident T423027 and raising the priority. We should decide if we want to page for Gerrit (in general/outside of office hours).

Need technical work:
Moving the service behind the CDN (T365259)

Gerrit is behind the CDN now.

Also there is a built-in alert because of that for ATSBackendErrorsHigh cache_text sre (gerrit.discovery.wmnet). So we are kind-of paging for Gerrit.

Document a common set of remediation steps that can be safely done by a responder (together with RelEng)

This is also a follow-up of T423027. We need runbooks and update the existing documentation.

ABran-WMF added a parent task: T423027: 2026-04-12 Gerrit Outage (was: DiskSpace).Fri, Apr 17, 9:02 AM
LSobanski renamed this task from Gerrit outage didn't page until 4.5 hours after the first alert to Enable paging for Gerrit (was: Gerrit outage didn't page until 4.5 hours after the first alert).Mon, Apr 20, 3:58 PM
LSobanski added a comment.Mon, Apr 27, 3:51 PM

Next steps

  • Create a separate blackbox check with severity:page (in addition to the existing IRC one)
  • Create an alert to fire after the above check is firing for a set time (15 minutes?)
LSobanski assigned this task to ABran-WMF.Mon, Apr 27, 3:52 PM
LSobanski moved this task from Incoming to Work in Progress on the collaboration-services board.
gerritbot added a comment.Tue, Apr 28, 6:54 AM

Change #1278238 had a related patch set uploaded (by Arnaudb; author: Arnaudb):

[operations/puppet@production] gerrit: add paging blackbox check

https://gerrit.wikimedia.org/r/1278238

gerritbot added a project: Patch-For-Review.Tue, Apr 28, 6:54 AM
ABran-WMF changed the task status from Open to In Progress.Tue, Apr 28, 6:57 AM
ABran-WMF moved this task from Work in Progress to Awaiting Input on the collaboration-services board.
gerritbot added a comment.Tue, Apr 28, 9:01 AM

Change #1278238 merged by Arnaudb:

[operations/puppet@production] gerrit: add paging blackbox check

https://gerrit.wikimedia.org/r/1278238

ABran-WMF closed this task as Resolved.Tue, Apr 28, 9:03 AM

a page will be sent if the black box monitoring check fails for more than 15 minutes

Maintenance_bot removed a project: Patch-For-Review.Tue, Apr 28, 9:30 AM
Content licensed under Creative Commons Attribution-ShareAlike (CC BY-SA) 4.0 unless otherwise noted; code licensed under GNU General Public License (GPL) 2.0 or later and other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL · Credits