Page MenuHomePhabricator
Create Task
Maniphest T260666

Create a cookbook to automate gerrit's switchover
Closed, ResolvedPublic
Actions

Description

Switching datacenter for gerrit should be automated with a cookbook, while it's currently mostly a manual task.

Details

Related Changes in Gerrit:
Show related patches Customize query in gerrit

Related Objects
Search...

StatusSubtypeAssignedTask
 OpenNoneT407557 OpenSSH 10.1+ warns that Wikimedia SSH does not use post-quantum key exchange algorithm
 OpenNoneT407844 Gerrit ssh daemon does not offer post-quantum kex leading to a warning with OpenSSH 10
 OpenNoneT392448 Upgrade to Gerrit 3.12
 OpenNoneT379714 Upgrade to Gerrit 3.11
 OpenNoneT392465 Switch Gerrit from Java 17 to Java 21
 OpenNoneT384595 Upgrade Collab hosts to Bookworm
 OpenNoneT392464 Upgrade Gerrit hosts from Bullseye to Bookworm
 OpenNoneT387831 Standardize failover procedures for Collab services
 ResolvedNoneT393239 ProbeDown
 In ProgressABran-WMFT387833 Gerrit failover process
 OpenNoneT388507 Investigate missing commit from old changes
 ResolvedABran-WMFT260666 Create a cookbook to automate gerrit's switchover
 ResolvedABran-WMFT393050 ProbeDown
 In ProgressABran-WMFT338470 Rename gerrit2 unix user to gerrit and assign a fixed uid
 ResolvedDzahnT379213 PuppetFailure - gerrit2003
 ResolvedMatthewVernonT392186 Grant Gerrit admin to arnaudb
 ResolvedLSobanskiT393034 Investigate out of date refs following gerrit switchover
 ResolvedABran-WMFT395440 Gerrit read-only plugin test
 OpenABran-WMFT406334 Gerrit switchover between secondary instances
 ResolvedLSobanskiT406482 SystemdUnitFailed (jenkins.service on contint1002)
 ResolvedhasharT406762 gerrit2003 is trying to backup incrementally 3.5 million files every hour, clogging backus and filling in available disk space
 ResolvedhasharT406856 Reduce size of analytics/superset/deploy.git Gerrit repo
Restricted Task

Event Timeline

Joe created this task.Aug 18 2020, 8:57 AM
Restricted Application added a subscriber: Aklapper. · View Herald TranscriptAug 18 2020, 8:57 AM
Joe triaged this task as Medium priority.Aug 18 2020, 8:58 AM
Joe added projects: SRE, serviceops, SRE-tools.
Joe updated the task description. (Show Details)
Aklapper added a project: Infrastructure-Foundations.Jun 21 2021, 8:59 PM
jijiki moved this task from Incoming 🐫 to 🙈🙉🙊Backlog on the serviceops board.Sep 28 2022, 2:27 PM
LSobanski added a project: collaboration-services.May 10 2023, 2:51 PM
LSobanski lowered the priority of this task from Medium to Low.May 15 2023, 3:45 PM
LSobanski moved this task from Incoming to Backlog on the collaboration-services board.
thcipriani closed this task as a duplicate of T387833: Gerrit failover process.Mar 26 2025, 12:43 PM
ABran-WMF subscribed.Mar 26 2025, 12:49 PM
ABran-WMF mentioned this in T387833: Gerrit failover process.Apr 1 2025, 6:02 AM
LSobanski reopened this task as Open.Apr 1 2025, 10:11 AM
LSobanski subscribed.

This is separate from the activity in T387833: Gerrit failover process so let's keep it open.

ABran-WMF claimed this task.Apr 3 2025, 1:48 PM
ABran-WMF added a parent task: T387833: Gerrit failover process.Apr 7 2025, 8:05 AM
gerritbot added a comment.Apr 8 2025, 2:26 PM

Change #1135043 had a related patch set uploaded (by Arnaudb; author: Arnaudb):

[operations/cookbooks@master] gerrit: failover cookbook

https://gerrit.wikimedia.org/r/1135043

gerritbot added a project: Patch-For-Review.Apr 8 2025, 2:26 PM
ABran-WMF moved this task from Backlog to Work in Progress on the collaboration-services board.Apr 14 2025, 8:40 AM
gerritbot added a comment.Apr 14 2025, 8:57 AM

Change #1135043 merged by jenkins-bot:

[operations/cookbooks@master] gerrit: failover cookbook

https://gerrit.wikimedia.org/r/1135043

ABran-WMF added a comment.Apr 14 2025, 9:06 AM

This first iteration is still fairly manual but will give us a stepping stone to build upon.
I'll run an end to end test between gerrit-replica and gerrit2003 to emulate production behavior and shed off the remaining quirks.

Maintenance_bot removed a project: Patch-For-Review.Apr 14 2025, 9:31 AM
gerritbot added a comment.Apr 14 2025, 12:06 PM

Change #1136361 had a related patch set uploaded (by Arnaudb; author: Arnaudb):

[operations/cookbooks@master] gerrit: failover cookbook fix

https://gerrit.wikimedia.org/r/1136361

gerritbot added a project: Patch-For-Review.Apr 14 2025, 12:06 PM
gerritbot added a comment.Apr 14 2025, 12:31 PM

Change #1136361 merged by Arnaudb:

[operations/cookbooks@master] gerrit: failover cookbook fix

https://gerrit.wikimedia.org/r/1136361

gerritbot added a comment.Apr 14 2025, 12:39 PM

Change #1136372 had a related patch set uploaded (by Arnaudb; author: Arnaudb):

[operations/cookbooks@master] gerrit: failover cookbook dry_run mode

https://gerrit.wikimedia.org/r/1136372

gerritbot added a comment.Apr 14 2025, 12:55 PM

Change #1136372 abandoned by Arnaudb:

[operations/cookbooks@master] gerrit: failover cookbook dry_run mode

Reason:

fully supported via spicerack: https://gerrit.wikimedia.org/r/c/operations/cookbooks/+/1136372/comment/54d0654b_21bb043c/

https://gerrit.wikimedia.org/r/1136372

Maintenance_bot removed a project: Patch-For-Review.Apr 14 2025, 1:31 PM
gerritbot added a comment.Apr 14 2025, 1:45 PM

Change #1136385 had a related patch set uploaded (by Arnaudb; author: Arnaudb):

[operations/cookbooks@master] gerrit: failover cookbook

https://gerrit.wikimedia.org/r/1136385

gerritbot added a project: Patch-For-Review.Apr 14 2025, 1:45 PM
gerritbot added a comment.Apr 15 2025, 12:32 PM

Change #1136709 had a related patch set uploaded (by Arnaudb; author: Arnaudb):

[operations/cookbooks@master] gerrit: failover cookbook

https://gerrit.wikimedia.org/r/1136709

gerritbot added a comment.Apr 15 2025, 1:09 PM

Change #1136385 merged by jenkins-bot:

[operations/cookbooks@master] gerrit: failover cookbook

https://gerrit.wikimedia.org/r/1136385

gerritbot added a comment.Apr 15 2025, 1:09 PM

Change #1136709 merged by jenkins-bot:

[operations/cookbooks@master] gerrit: failover cookbook bugfix

https://gerrit.wikimedia.org/r/1136709

Maintenance_bot removed a project: Patch-For-Review.Apr 15 2025, 1:31 PM
gerritbot added a comment.Apr 17 2025, 4:29 AM

Change #1137099 had a related patch set uploaded (by Arnaudb; author: Arnaudb):

[operations/cookbooks@master] gerrit: failover cookbook test mode

https://gerrit.wikimedia.org/r/1137099

gerritbot added a project: Patch-For-Review.Apr 17 2025, 4:29 AM
gerritbot added a comment.Apr 30 2025, 5:25 AM

Change #1139977 had a related patch set uploaded (by Arnaudb; author: Arnaudb):

[operations/cookbooks@master] gerrit: failover bugfix

https://gerrit.wikimedia.org/r/1139977

gerritbot added a comment.Apr 30 2025, 8:29 AM

Change #1139977 merged by Arnaudb:

[operations/cookbooks@master] gerrit: failover bugfix

https://gerrit.wikimedia.org/r/1139977

gerritbot added a comment.Apr 30 2025, 12:57 PM

Change #1137099 abandoned by Arnaudb:

[operations/cookbooks@master] gerrit: failover cookbook test mode

https://gerrit.wikimedia.org/r/1137099

Maintenance_bot removed a project: Patch-For-Review.Apr 30 2025, 1:30 PM
ABran-WMF added a subtask: T393050: ProbeDown.Apr 30 2025, 7:22 PM
ABran-WMF closed subtask T393050: ProbeDown as Resolved.
ABran-WMF added a parent task: T393239: ProbeDown.May 5 2025, 7:32 AM
gerritbot added a comment.May 5 2025, 11:11 AM

Change #1141862 had a related patch set uploaded (by Arnaudb; author: Arnaudb):

[operations/cookbooks@master] gerrit: probe DNS on both hosts before doing stuff

https://gerrit.wikimedia.org/r/1141862

gerritbot added a project: Patch-For-Review.May 5 2025, 11:11 AM
gerritbot added a comment.May 7 2025, 6:17 AM

Change #1142793 had a related patch set uploaded (by Arnaudb; author: Arnaudb):

[operations/cookbooks@master] gerrit: safeguards against corruptions and mishaps

https://gerrit.wikimedia.org/r/1142793

gerritbot added a comment.May 7 2025, 1:55 PM

Change #1143102 had a related patch set uploaded (by Arnaudb; author: Arnaudb):

[operations/cookbooks@master] gerrit: grepping for misconfigurations

https://gerrit.wikimedia.org/r/1143102

gerritbot added a comment.May 12 2025, 12:58 PM

Change #1144565 had a related patch set uploaded (by Arnaudb; author: Arnaudb):

[operations/cookbooks@master] gerrit: git backup tree consistency checker

https://gerrit.wikimedia.org/r/1144565

gerritbot added a comment.May 13 2025, 2:16 PM

Change #1145208 had a related patch set uploaded (by Arnaudb; author: Arnaudb):

[operations/cookbooks@master] gerrit: lock and preflight checks

https://gerrit.wikimedia.org/r/1145208

ABran-WMF added a subtask: T338470: Rename gerrit2 unix user to gerrit and assign a fixed uid.May 23 2025, 5:39 AM
gerritbot added a comment.Jun 16 2025, 9:55 AM

Change #1159395 had a related patch set uploaded (by Arnaudb; author: Arnaudb):

[operations/cookbooks@master] gerrit: read-only plugin orchestration in failover

https://gerrit.wikimedia.org/r/1159395

gerritbot added a comment.Jun 25 2025, 12:49 PM

Change #1141862 merged by jenkins-bot:

[operations/cookbooks@master] gerrit: probe DNS on both hosts before doing stuff

https://gerrit.wikimedia.org/r/1141862

gerritbot added a comment.Jun 25 2025, 12:50 PM

Change #1142793 merged by jenkins-bot:

[operations/cookbooks@master] gerrit: rsync --checksum local backup safety net

https://gerrit.wikimedia.org/r/1142793

gerritbot added a comment.Jun 25 2025, 12:50 PM

Change #1143102 merged by jenkins-bot:

[operations/cookbooks@master] gerrit: grepping for misconfigurations

https://gerrit.wikimedia.org/r/1143102

gerritbot added a comment.Jun 25 2025, 12:50 PM

Change #1144565 merged by jenkins-bot:

[operations/cookbooks@master] gerrit: git backup tree consistency checker

https://gerrit.wikimedia.org/r/1144565

gerritbot added a comment.Jun 25 2025, 12:50 PM

Change #1145208 merged by jenkins-bot:

[operations/cookbooks@master] gerrit: lock, preflight checks, hieradata lookups, verbosity

https://gerrit.wikimedia.org/r/1145208

gerritbot added a comment.Jun 25 2025, 12:56 PM

Change #1159395 merged by jenkins-bot:

[operations/cookbooks@master] gerrit: read-only plugin orchestration in failover

https://gerrit.wikimedia.org/r/1159395

Maintenance_bot removed a project: Patch-For-Review.Jun 25 2025, 1:31 PM
gerritbot added a comment.Jul 8 2025, 2:01 PM

Change #1167226 had a related patch set uploaded (by Arnaudb; author: Arnaudb):

[operations/cookbooks@master] gerrit: standardize expected rc on systemctl check

https://gerrit.wikimedia.org/r/1167226

gerritbot added a project: Patch-For-Review.Jul 8 2025, 2:01 PM
gerritbot added a comment.Jul 9 2025, 9:41 AM

Change #1167226 merged by jenkins-bot:

[operations/cookbooks@master] gerrit: standardize expected rc on systemctl check

https://gerrit.wikimedia.org/r/1167226

Maintenance_bot removed a project: Patch-For-Review.Jul 9 2025, 10:32 AM
ABran-WMF closed this task as Resolved.Jul 28 2025, 9:29 AM

this can be considered as done with the merge of https://gerrit.wikimedia.org/r/c/operations/cookbooks/+/1170160 next steps will be handled in T387833: Gerrit failover process

ABran-WMF changed the status of subtask T338470: Rename gerrit2 unix user to gerrit and assign a fixed uid from Stalled to In Progress.Oct 2 2025, 12:05 PM
Content licensed under Creative Commons Attribution-ShareAlike (CC BY-SA) 4.0 unless otherwise noted; code licensed under GNU General Public License (GPL) 2.0 or later and other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL · Credits