Update and improve operation runbooks and documentation for Gerrit
Closed, ResolvedPublic
Actions

Assigned To

Authored By

	Jelto
	Apr 16 2026, 2:19 PM

Project Tags

Referenced Files

	Restricted File
	Apr 24 2026, 9:23 AM

Subscribers

Description

In T423027: 2026-04-12 Gerrit Outage (was: DiskSpace) the lack of up-to-date runbooks was mentioned. As a follow-up we should improve the documentation for Gerrit so that SREs on-call also know how to perform basic Gerrit operations.

Additionally (if possible) we should also document how to temporarily escalate privileges, some incidents might require gerrit admin permissions.

non-existing runbooks and documented gerrit ssh commands are not working when Gerrit is stuck/down
how to
- restart CI T424196: Document how to restart CI when Gerrit had a production issue
- restart gerrit,
- fix disk issues on Gerrit https://gerrit.wikimedia.org/r/c/operations/alerts/+/1277048 - F77665655
- have gerrit ssh commands to elevate privileges

Details

Related Changes in Gerrit:

	Subject	Author	Repo	Branch	Lines +/-
	gerrit: fix AlertLintProblem for GerritDiskSpaceExhaustionIncoming	Arnaudb	operations/alerts	master	+1 -0
	gerrit: predict_linear alert for diskspace	Arnaudb	operations/alerts	master	+43 -1

Customize query in gerrit

Related Objects
Search...

Status	Assigned	Task
Resolved	Jelto	T423027 2026-04-12 Gerrit Outage (was: DiskSpace)
Resolved	ABran-WMF	T423601 Update and improve operation runbooks and documentation for Gerrit
Open	hashar	T424196 Document how to restart CI when Gerrit had a production issue

Event Timeline

Jelto created this task.Apr 16 2026, 2:19 PM

Restricted Application added a subscriber: Aklapper. · View Herald TranscriptApr 16 2026, 2:19 PM

ABran-WMF subscribed.Apr 16 2026, 2:20 PM

Jelto mentioned this in T423027: 2026-04-12 Gerrit Outage (was: DiskSpace).Apr 16 2026, 2:20 PM

Jelto updated the task description. (Show Details)

(swapping incident tag for incident-followup; also boldly removing as a subtask, as iiuc this is a follow-up task rather than something that needs to be done before the incident can be called resolved)

RhinosF1 subscribed.Apr 16 2026, 5:07 PM

ABran-WMF added a parent task: T423027: 2026-04-12 Gerrit Outage (was: DiskSpace).Apr 17 2026, 9:02 AM

I've added a bit of documentation on gerrit ssh commands: https://wikitech.wikimedia.org/wiki/Gerrit/Operations#Use_gerrit_ssh_commands

Unfortunately, the process is not trivial and requires updating ldap before being able to access ssh commands

I've also updated the "restart gerrit" part: https://wikitech.wikimedia.org/wiki/Gerrit/Operations#Restarting

ABran-WMF updated the task description. (Show Details)Apr 17 2026, 1:47 PM

LSobanski assigned this task to ABran-WMF.Apr 20 2026, 3:42 PM

LSobanski triaged this task as High priority.

LSobanski moved this task from Incoming to Work in Progress on the Collaboration-Services board.

ABran-WMF created subtask T424196: Document how to restart CI when Gerrit had a production issue.Apr 23 2026, 9:30 AM

ABran-WMF updated the task description. (Show Details)

Change #1277048 had a related patch set uploaded (by Arnaudb; author: Arnaudb):

[operations/alerts@master] gerrit: predict_linear alert for diskspace

https://gerrit.wikimedia.org/r/1277048

gerritbot added a project: Patch-For-Review.Apr 24 2026, 9:04 AM

ABran-WMF updated the task description. (Show Details)Apr 24 2026, 9:05 AM

ABran-WMF updated the task description. (Show Details)Apr 24 2026, 9:23 AM

Change #1277048 merged by jenkins-bot:

[operations/alerts@master] gerrit: predict_linear alert for diskspace

https://gerrit.wikimedia.org/r/1277048

Maintenance_bot removed a project: Patch-For-Review.Apr 27 2026, 1:32 PM

this can be considered done, the remaining bit will be tracked with T424196: Document how to restart CI when Gerrit had a production issue

Change #1277565 had a related patch set uploaded (by Arnaudb; author: Arnaudb):

[operations/alerts@master] gerrit: fix AlertLintProblem for GerritDiskSpaceExhaustionIncoming

https://gerrit.wikimedia.org/r/1277565

gerritbot added a project: Patch-For-Review.Apr 27 2026, 2:29 PM

Change #1277565 merged by jenkins-bot: