Gerrit read-only plugin test
Closed, ResolvedPublic
Actions

Assigned To

Authored By

	ABran-WMF
	May 28 2025, 12:41 PM

Description

While working on improving the automation of Gerrit failover after the last incident we had; we opened the conversation to challenge the way we are synchronizing data across Gerrit instances. In this conversation, the idea of using Gerrit's read-only plugin emerged as a potential solution to simplify the way we are handling things. The best case scenario would be to be able to stop contributions from coming-in while embedded replication finishes its jobs, switch-over to the other instance and wait for the proper conditions to be satisfied to open the service again for writes.

ensure of it is still maintained as the last commit is old enough to raise the question.
explore the capabilities and compatibility of the read-only plugin to ensure it would fit our use case
test the solution
implement in production
update gerrit failover cookbook

Details

Related Changes in Gerrit:

	Subject	Repo	Branch	Lines +/-
	gerrit: read-only plugin orchestration in failover	operations/cookbooks	master	+129 -2

Customize query in gerrit

Related Objects
Search...

Status	Assigned	Task
Open	None	T407557 OpenSSH 10.1+ warns that Wikimedia SSH does not use post-quantum key exchange algorithm
Open	None	T407844 Gerrit ssh daemon does not offer post-quantum kex leading to a warning with OpenSSH 10
Open	None	T392448 Upgrade to Gerrit 3.12
Open	None	T379714 Upgrade to Gerrit 3.11
Open	None	T392465 Switch Gerrit from Java 17 to Java 21
Open	None	T384595 Upgrade Collab hosts to Bookworm
Open	None	T392464 Upgrade Gerrit hosts from Bullseye to Bookworm
Open	None	T387831 Standardize failover procedures for Collab services
Resolved	None	T393239 ProbeDown
In Progress	ABran-WMF	T387833 Gerrit failover process
Resolved	ABran-WMF	T395440 Gerrit read-only plugin test

Event Timeline

ABran-WMF created this task.May 28 2025, 12:41 PM

Restricted Application removed a project: Patch-For-Review. · View Herald TranscriptMay 28 2025, 12:41 PM

LSobanski triaged this task as High priority.Jun 2 2025, 5:52 AM

ABran-WMF moved this task from Incoming to Work in Progress on the collaboration-services board.Jun 2 2025, 5:53 AM

ABran-WMF updated the task description. (Show Details)Jun 2 2025, 6:33 AM

read-only is not flagged as deprecated or obsolete, its documentation is still in the main repository

The read-only plugin will be relevant to use in the context of switchovers to help wait for the replication threads to finish their jobs and to prevent writing on the wrong instance until we've reached a stable state after the switchover is done.
It should not be used in normal production situations. Using read-only too extensively in our situation might complicate switchovers and debugging given we don't rely on Gerrit internal SSH to replicate the data, to use instead the host's SSH server. That means a host could ignore the read-only status of a replication target, which could be unfortunate.

ABran-WMF updated the task description. (Show Details)Jun 13 2025, 4:23 PM

Change #1159395 had a related patch set uploaded (by Arnaudb; author: Arnaudb):

[operations/cookbooks@master] gerrit: read-only plugin orchestration in failover

https://gerrit.wikimedia.org/r/1159395

gerritbot added a project: Patch-For-Review.Jun 16 2025, 9:55 AM

ABran-WMF closed this task as Resolved.Jun 23 2025, 8:18 AM

ABran-WMF updated the task description. (Show Details)

Change #1159395 merged by jenkins-bot:

[operations/cookbooks@master] gerrit: read-only plugin orchestration in failover

https://gerrit.wikimedia.org/r/1159395