Gerrit switchover between secondary instances
Closed, ResolvedPublic
Actions

Assigned To

Authored By

	ABran-WMF
	Oct 3 2025, 1:35 PM

Project Tags

Referenced Files

None

Subscribers

Description

As in T387833: Gerrit switchover process we'll need to document and automate the process of switching secondary roles (replica/spare)

Details

Related Changes in Gerrit:

Subject	Author	Repo	Branch	Lines +/-
gerrit: fix discovery record	Arnaudb	operations/dns	master	+1 -1
gerrit: swap gerrit-spare and gerrit-replica	Arnaudb	operations/puppet	production	+4 -4
gerrit: swap gerrit-replica and gerrit-spare	Arnaudb	operations/dns	master	+9 -9

Customize query in gerrit

Related Objects
Search...

Status	Assigned	Task
Open	None	T387831 Standardize failover procedures for Collab services
Resolved	None	T393239 ProbeDown
Resolved	ABran-WMF	T387833 Gerrit switchover process
Declined	None	T257383 Update wikibugs's Gerrit ssh host keys
Stalled	None	T257382 Update libup for Gerrit's new ssh host keys
Resolved	Dzahn	T240266 gerrit.wikimedia.org:29418 uses 1024-bit RSA key only
Resolved	ABran-WMF	T406334 Gerrit switchover between secondary instances

Event Timeline

ABran-WMF created this task.Oct 3 2025, 1:35 PM

Restricted Application removed a project: Patch-For-Review. · View Herald TranscriptOct 3 2025, 1:35 PM

Restricted Application added a subscriber: Aklapper. · View Herald Transcript

This would only be a thing though if we permanently keep 3 or more gerrit servers around.

So far we have one prod and one replica and once old hardware is replaced we would be back to exactly 2 machines in production with one extra as standby for gerrit and phab. And that one we would only use if one of the 2 prod machines dies.

So then it would be more like "how to spin up a new replica" rather than failing over from one working one to another. right?

ABran-WMF added a project: Gerrit.Oct 6 2025, 3:34 PM

In scope: decide and document whether we keep the second replica around permanently.

LSobanski triaged this task as Medium priority.Oct 6 2025, 3:42 PM

LSobanski moved this task from Incoming to Backlog on the Collaboration-Services board.

This task is a probable blocker for T417247: Reimage gerrit2002

In T406334#11242622, @Dzahn wrote:

This would only be a thing though if we permanently keep 3 or more gerrit servers around.

The process will also become obsolete after we move Gerrit to Kubernetes; depending on the topology we end up choosing.
I think (perhaps foolishly) the process will not be that complicated to design, so it could be worth spending time on it, to fill in the time gap between now and when it'll become obsolete.

ABran-WMF added a parent task: T240266: gerrit.wikimedia.org:29418 uses 1024-bit RSA key only.Feb 20 2026, 4:31 PM

Change #1242268 had a related patch set uploaded (by Arnaudb; author: Arnaudb):

[operations/dns@master] gerrit: swap gerrit-replica and gerrit-spare

https://gerrit.wikimedia.org/r/1242268

gerritbot added a project: Patch-For-Review.Feb 23 2026, 9:09 AM

Change #1242269 had a related patch set uploaded (by Arnaudb; author: Arnaudb):

[operations/puppet@production] gerrit: swap gerrit-spare and gerrit-replica

https://gerrit.wikimedia.org/r/1242269

Change #1242268 merged by Arnaudb:

[operations/dns@master] gerrit: swap gerrit-replica and gerrit-spare

https://gerrit.wikimedia.org/r/1242268

Change #1242269 merged by Arnaudb: