Page MenuHomePhabricator
Create Task
Maniphest T387833

Gerrit switchover process
Closed, ResolvedPublic
Actions

Description

A part of the effort to standardize failover procedures for Collab services.

Details

Related Changes in Gerrit:
SubjectAuthorRepoBranchLines +/-
gerrit: remove read-only configArnaudboperations/puppetproduction+0 -7
gerrit: read-only confirmation wording tweakingArnaudboperations/cookbooksmaster+3 -5
gerrit: clarify confirmation wording on read-onlyArnaudboperations/cookbooksmaster+9 -9
gerrit: switchover from gerrit1003 to gerrit2003Arnaudboperations/dnsmaster+5 -6
gerrit: rename failover cookbook to switchoverArnaudboperations/cookbooksmaster+0 -0
gerrit: re-enable backups and monitoring on gerrit2003Arnaudboperations/puppetproduction+2 -2
gerrit: Switchover gerrit1003 → gerrit2003Arnaudboperations/puppetproduction+134 -65
gerrit: update switchover related cookbooksArnaudboperations/cookbooksmaster+23 -23
gerrit: unmask gerrit.service on gerrit2003Arnaudboperations/puppetproduction+1 -1
gerrit: temporarily disable replication on gerrit2003Arnaudboperations/puppetproduction+4 -4
gerrit: re-enable backups on gerrit2003Arnaudboperations/puppetproduction+1 -1
gerrit: unmask serviceArnaudboperations/puppetproduction+3 -3
gerrit: Switchover gerrit1003 → gerrit2003Arnaudboperations/puppetproduction+79 -40
gerrit: add a confirmation prompt on rsyncArnaudboperations/cookbooksmaster+2 -0
gerrit: testing syncArnaudboperations/cookbooksmaster+15 -14
gerrit: disable backups temporarilyArnaudboperations/puppetproduction+2 -0
gerrit: rsync logic extraction from failoverArnaudboperations/cookbooksmaster+289 -129
gerrit: add a layer of CNAME to ease switch oversHasharoperations/dnsmaster+25 -13
gerrit: Switchover gerrit1003 → gerrit2003Arnaudboperations/puppetproduction+79 -40
gerrit: remove localbackup logic from failoverArnaudboperations/cookbooksmaster+8 -48
gerrit: add a local backup cookbookArnaudboperations/cookbooksmaster+144 -0
gerrit: add dry run rsyncArnaudboperations/cookbooksmaster+12 -6
gerrit: remove localbackup logic from failoverArnaudboperations/cookbooksmaster+7 -43
gerrit: enable monitoring for other instancesArnaudboperations/puppetproduction+25 -0
gerrit: stop stopping gerrit.serviceArnaudboperations/cookbooksmaster+0 -8
gerrit: stop puppet across all instancesArnaudboperations/cookbooksmaster+6 -4
gerrit: rsync and chown fixesArnaudboperations/cookbooksmaster+12 -7
gerrit: disable gerrit service to enable backupsArnaudboperations/puppetproduction+2 -2
gerrit: ask the operator to merge puppet earlierArnaudboperations/cookbooksmaster+5 -3
gerrit: typo fix in post_sync_validationArnaudboperations/cookbooksmaster+1 -1
gerrit: re-enable backups on gerrit2003Arnaudboperations/puppetproduction+0 -2
Revert^4 "gerrit: switchover from gerrit1003 to gerrit2003"Arnaudboperations/dnsmaster+8 -8
Revert^4 "gerrit: Switchover gerrit1003 → gerrit2003"Arnaudboperations/puppetproduction+70 -33
gerrit: local backup on source server onlyArnaudboperations/cookbooksmaster+4 -4
Revert^2 "gerrit: switchover from gerrit1003 to gerrit2003"Arnaudboperations/dnsmaster+8 -8
Revert^2 "gerrit: Switchover gerrit1003 → gerrit2003"Arnaudboperations/puppetproduction+70 -33
gerrit: Switchover gerrit1003 → gerrit2003Arnaudboperations/puppetproduction+70 -33
Add a banner for a Gerrit switch over maintenanceHasharoperations/software/gerritdeploy/wmf/stable-3.10+114 -0
Disable component rather than motd pluginHasharoperations/software/gerritdeploy/wmf/stable-3.10+4 -4
Disable motd banner: maintenance window has closedHasharoperations/software/gerritdeploy/wmf/stable-3.10+5 -3
gerrit: fix typo in source pathArnaudboperations/cookbooksmaster+9 -10
gerrit: test 1193860Arnaudboperations/cookbooksmaster+98 -97
gerrit: switchover from gerrit1003 to gerrit2003Arnaudboperations/dnsmaster+8 -8
gerrit: avoid hardcoded hostnames, replace with hiera lookupsDzahnoperations/puppetproduction+5 -3
gerrit: typo on --systemd argArnaudboperations/cookbooksmaster+5 -3
gerrit: bugfixes on failoverArnaudboperations/cookbooksmaster+24 -9
gerrit: replace host names in replica config with variablesDzahnoperations/puppetproduction+2 -2
gerrit: Switchover gerrit1003 → gerrit2003Arnaudboperations/puppetproduction+46 -30
gerrit: add spare fqdn to apache vhostArnaudboperations/puppetproduction+19 -0
gerrit: add certificate request for gerrit-spare.w.oArnaudboperations/puppetproduction+1 -0
gerrit: site.pp cleanupArnaudboperations/puppetproduction+1 -6
gerrit: add httpbb monitoring to gerrit-spareArnaudboperations/puppetproduction+19 -0
gerrit: add service ip address for gerrit2003Arnaudboperations/puppetproduction+5 -3
Gerrit: Add service ip for gerrit2003Arnaudboperations/dnsmaster+8 -3
gerrit: Bugfixes - dry run testsArnaudboperations/cookbooksmaster+64 -218
gerrit: rename host and replica_hosts array to primary/replica_vhostDzahnoperations/puppetproduction+18 -21
gerrit: also rename "passive" to "spare" server in MOTDDzahnoperations/puppetproduction+1 -1
gerrit: replace hardcoded host name and codfw string for replicaDzahnoperations/puppetproduction+11 -2
Add readonly puginHasharoperations/software/gerritdeploy/wmf/stable-3.10+3 -0
gerrit: standardize expected rc on systemctl checkArnaudboperations/cookbooksmaster+1 -1
gerrit: sanity checks cookbook implementationArnaudboperations/cookbooksmaster+52 -150
gerrit: sanity checks as a cookbookArnaudboperations/cookbooksmaster+304 -0
gerrit: add readonly pluginArnaudboperations/software/gerritwmf/stable-3.10+4 -0
gerrit: add readonly pluginArnaudboperations/software/gerritwmf/stable-3.10+4 -0
gerrit: add readonly.jarArnaudboperations/software/gerritdeploy/wmf/stable-3.10+0 -0
gerrit: introduce second daemon_user nameDzahnoperations/puppetproduction+3 -2
gerrit: replace gerrit2003 RSA host key with ed25519 host keyDzahnoperations/puppetproduction+1 -1
gerrit: add a second replica, start replicating to gerrit2003Dzahnoperations/puppetproduction+16 -8
gerrit: add ssh_host_rsa public key for gerrit2003Dzahnoperations/puppetproduction+1 -0
gerrit: switchover to gerrit1003Arnaudboperations/puppetproduction+5 -3
gerrit: switchover to gerrit1003Arnaudboperations/dnsmaster+8 -8
gerrit: switchover to gerrit2002Arnaudboperations/puppetproduction+3 -3
gerrit: switchover to gerrit2002Arnaudboperations/dnsmaster+8 -8
Show related patches Customize query in gerrit

Related Objects
Search...

StatusSubtypeAssignedTask
Restricted Task
 ResolvedDzahnT411895 gerrit behind CDN
 ResolvedABran-WMFT420523 ProbeDown - Service gerrit2003:443 has failed probes (http_gerrit_tls_ip6)
Restricted Task
Restricted Task
 OpenNoneT392448 Upgrade to Gerrit 3.12
 OpenNoneT407557 OpenSSH 10.1+ warns that Wikimedia SSH does not use post-quantum key exchange algorithm
 OpenNoneT407844 Gerrit ssh daemon does not offer post-quantum kex leading to a warning with OpenSSH 10
Restricted Task
 OpenNoneT379714 Upgrade to Gerrit 3.11
 OpenNoneT392465 Switch Gerrit from Java 17 to Java 21
 OpenNoneT384595 Upgrade Collab hosts to Bookworm
 ResolvedABran-WMFT392464 Upgrade Gerrit hosts from Bullseye to Bookworm
 OpenNoneT387831 Standardize failover procedures for Collab services
 ResolvedNoneT393239 ProbeDown
 ResolvedABran-WMFT387833 Gerrit switchover process
 OpenNoneT388507 Investigate missing commit from old changes
 ResolvedABran-WMFT260666 Create a cookbook to automate gerrit's switchover
 ResolvedABran-WMFT393050 ProbeDown
 ResolvedABran-WMFT338470 Rename gerrit2 unix user to gerrit and assign a fixed uid
 ResolvedDzahnT379213 PuppetFailure - gerrit2003
 OpenNoneT417642 Rename gerrit2 Gerrit user to gerrit in the administrator group
 ResolvedMatthewVernonT392186 Grant Gerrit admin to arnaudb
 ResolvedLSobanskiT393034 Investigate out of date refs following gerrit switchover
 ResolvedABran-WMFT395440 Gerrit read-only plugin test
 ResolvedABran-WMFT406334 Gerrit switchover between secondary instances
 ResolvedLSobanskiT406482 SystemdUnitFailed (jenkins.service on contint1002)
 ResolvedhasharT406762 gerrit2003 is trying to backup incrementally 3.5 million files every hour, clogging backus and filling in available disk space
 ResolvedhasharT411583 Gerrit backups are growing
 ResolvedhasharT430774 Compact Gerrit H2 databases to reclaim disk
 OpenNoneT430775 Validate Gerrit backup recoverability after the fileset exclusions
 OpenNoneT433926 Backup Gerrit with LVM snapshots and Bacula pre/post job hooks
 ResolvedhasharT406856 Reduce size of analytics/superset/deploy.git Gerrit repo
 ResolvedABran-WMFT407200 Gerrit is down
 ResolvedABran-WMFT412779 Gerrit failover process
 ResolvedABran-WMFT417246 Reimage gerrit1003
 ResolveddancyT417767 scap error on Gerrit first setup
 ResolvedABran-WMFT417777 SystemdUnitFailed - apache2.service on gerrit1003:9100
 DuplicateNoneT417968 ProbeDown
 DuplicateABran-WMFT417966 GerritHAProxyBackendUnavailable
 ResolvedABran-WMFT417247 Reimage gerrit2002
 ResolvedABran-WMFT418242 SystemdUnitFailed - rsync-lfs_replica_sync.service on gerrit2002:9100
 ResolvedABran-WMFT418264 Fix gerrit-restart cookbook
 ResolvedABran-WMFT418266 GerritHAProxyServiceUnavailable
 ResolvedABran-WMFT418240 GerritReplicationUnavailable
 ResolvedABran-WMFT417279 Harmonize DNS on all gerrit instances
 ResolvedABran-WMFT417565 GerritHAProxyBackendUnavailable
 ResolvedABran-WMFT417566 GerritHAProxyServiceUnavailable

Event Timeline

There are a very large number of changes, so older changes are hidden. Show Older Changes
Dzahn added a comment.Oct 16 2025, 8:46 PM

Here are the notes / commands from a gerrit failover in the past:

https://phabricator.wikimedia.org/P47782

Dzahn added a comment.Oct 16 2025, 8:48 PM

Here is how we did the DNS change without having to merge while Gerrit is down:

  1. merge DNS change that removes gerrit-new and switches IP of gerrit.wikimedia.org - in web UI of gerrit(-old)
  2. run authdns-update on ns0.wikimedia.org, see the diff but do NOT commit yet
  3. disable puppet, stop gerrit, do the rsync, run chmod -R ...
  4. say "yes" to authdns-update and actually merge DNS change that removes gerrit-new and switches IP of gerrit.wikimedia.org
gerritbot added a comment.Oct 17 2025, 6:13 AM

Change #1196792 had a related patch set uploaded (by Arnaudb; author: Arnaudb):

[operations/puppet@production] gerrit: unmask service & disable backup temporarily

https://gerrit.wikimedia.org/r/1196792

ABran-WMF added a comment.Oct 17 2025, 6:40 AM
In T387833#11283327, @Dzahn wrote:

Here is how we did the DNS change without having to merge while Gerrit is down:

Thanks for the dig! I tweaked the process and the cookbook to be closer to this, the puppet merge timing was inconsistent with that approach.

ABran-WMF closed subtask T406762: gerrit2003 is trying to backup incrementally 3.5 million files every hour, clogging backus and filling in available disk space as Resolved.Oct 17 2025, 6:45 AM
gerritbot added a comment.Oct 17 2025, 6:49 AM

Change #1196684 merged by jenkins-bot:

[operations/cookbooks@master] gerrit: rsync and chown fixes

https://gerrit.wikimedia.org/r/1196684

gerritbot added a comment.Oct 17 2025, 6:50 AM

Change #1196694 merged by jenkins-bot:

[operations/cookbooks@master] gerrit: stop puppet across all instances

https://gerrit.wikimedia.org/r/1196694

gerritbot added a comment.Oct 17 2025, 7:49 AM

Change #1196695 merged by jenkins-bot:

[operations/cookbooks@master] gerrit: stop stopping gerrit.service

https://gerrit.wikimedia.org/r/1196695

hashar closed subtask T406856: Reduce size of analytics/superset/deploy.git Gerrit repo as Resolved.Oct 28 2025, 10:34 AM
gerritbot added a comment.Nov 19 2025, 10:50 AM

Change #1193599 abandoned by Arnaudb:

[operations/cookbooks@master] gerrit: remove localbackup logic from failover

Reason:

obsolete, will workaround the merge conflict with another change

https://gerrit.wikimedia.org/r/1193599

gerritbot added a comment.Nov 24 2025, 7:45 AM

Change #1210386 had a related patch set uploaded (by Arnaudb; author: Arnaudb):

[operations/cookbooks@master] gerrit: remove localbackup logic from failover

https://gerrit.wikimedia.org/r/1210386

gerritbot added a comment.Nov 24 2025, 7:46 AM

Change #1195437 merged by jenkins-bot:

[operations/cookbooks@master] gerrit: add dry run rsync

https://gerrit.wikimedia.org/r/1195437

gerritbot added a comment.Nov 24 2025, 7:46 AM

Change #1193590 merged by jenkins-bot:

[operations/cookbooks@master] gerrit: add a local backup cookbook

https://gerrit.wikimedia.org/r/1193590

gerritbot added a comment.Nov 24 2025, 11:41 AM

Change #1210560 had a related patch set uploaded (by Hashar; author: Hashar):

[operations/dns@master] gerrit: add a layer of CNAME to ease switch overs

https://gerrit.wikimedia.org/r/1210560

gerritbot added a comment.Nov 26 2025, 6:24 AM

Change #1210386 merged by jenkins-bot:

[operations/cookbooks@master] gerrit: remove localbackup logic from failover

https://gerrit.wikimedia.org/r/1210386

gerritbot added a comment.Nov 26 2025, 7:25 AM

Change #1211548 had a related patch set uploaded (by Arnaudb; author: Arnaudb):

[operations/puppet@production] gerrit: Switchover gerrit1003 → gerrit2003

https://gerrit.wikimedia.org/r/1211548

gerritbot added a comment.Nov 26 2025, 7:26 AM

Change #1211548 abandoned by Arnaudb:

[operations/puppet@production] gerrit: Switchover gerrit1003 → gerrit2003

Reason:

forgot a step

https://gerrit.wikimedia.org/r/1211548

gerritbot added a comment.Nov 26 2025, 7:29 AM

Change #1211549 had a related patch set uploaded (by Arnaudb; author: Arnaudb):

[operations/puppet@production] gerrit: Switchover gerrit1003 → gerrit2003

https://gerrit.wikimedia.org/r/1211549

gerritbot added a comment.Nov 26 2025, 7:33 AM

Change #1211551 had a related patch set uploaded (by Arnaudb; author: Arnaudb):

[operations/puppet@production] gerrit: re-enable backups on gerrit2003

https://gerrit.wikimedia.org/r/1211551

Dzahn reopened subtask T406762: gerrit2003 is trying to backup incrementally 3.5 million files every hour, clogging backus and filling in available disk space as Open.Dec 1 2025, 4:23 PM
hashar closed subtask T406762: gerrit2003 is trying to backup incrementally 3.5 million files every hour, clogging backus and filling in available disk space as Resolved.Dec 2 2025, 3:47 PM
gerritbot added a comment.Dec 3 2025, 10:05 AM

Change #1214466 had a related patch set uploaded (by Arnaudb; author: Arnaudb):

[operations/cookbooks@master] gerrit: rsync logic extraction from failover

https://gerrit.wikimedia.org/r/1214466

gerritbot added a comment.Dec 5 2025, 3:58 PM

Change #1210560 abandoned by Hashar:

[operations/dns@master] gerrit: add a layer of CNAME to ease switch overs

https://gerrit.wikimedia.org/r/1210560

gerritbot added a comment.Dec 8 2025, 9:49 AM

Change #1214466 merged by jenkins-bot:

[operations/cookbooks@master] gerrit: rsync logic extraction from failover

https://gerrit.wikimedia.org/r/1214466

gerritbot added a comment.Dec 8 2025, 1:21 PM

Change #1216583 had a related patch set uploaded (by Arnaudb; author: Arnaudb):

[operations/puppet@production] gerrit: disable backups temporarily

https://gerrit.wikimedia.org/r/1216583

gerritbot added a comment.Dec 8 2025, 1:24 PM

Change #1216583 merged by Arnaudb:

[operations/puppet@production] gerrit: disable backups temporarily

https://gerrit.wikimedia.org/r/1216583

gerritbot added a comment.Dec 8 2025, 1:30 PM

Change #1216586 had a related patch set uploaded (by Arnaudb; author: Arnaudb):

[operations/cookbooks@master] gerrit: testing sync

https://gerrit.wikimedia.org/r/1216586

gerritbot added a comment.Dec 8 2025, 2:20 PM

Change #1216586 abandoned by Arnaudb:

[operations/cookbooks@master] gerrit: testing sync

Reason:

testing over

https://gerrit.wikimedia.org/r/1216586

gerritbot added a comment.Dec 8 2025, 2:23 PM

Change #1216592 had a related patch set uploaded (by Arnaudb; author: Arnaudb):

[operations/cookbooks@master] gerrit: add a confirmation prompt on rsync

https://gerrit.wikimedia.org/r/1216592

gerritbot added a comment.Dec 10 2025, 10:17 AM

Change #1216592 merged by jenkins-bot:

[operations/cookbooks@master] gerrit: add a confirmation prompt on rsync

https://gerrit.wikimedia.org/r/1216592

gerritbot added a comment.Dec 10 2025, 10:28 AM

Change #1217133 had a related patch set uploaded (by Arnaudb; author: Arnaudb):

[operations/puppet@production] gerrit: Switchover gerrit1003 → gerrit2003

https://gerrit.wikimedia.org/r/1217133

gerritbot added a comment.Dec 10 2025, 10:29 AM

Change #1211549 abandoned by Arnaudb:

[operations/puppet@production] gerrit: Switchover gerrit1003 → gerrit2003

Reason:

1217133

https://gerrit.wikimedia.org/r/1211549

gerritbot added a comment.Dec 10 2025, 10:29 AM

Change #1196792 abandoned by Arnaudb:

[operations/puppet@production] gerrit: unmask service

Reason:

1217133

https://gerrit.wikimedia.org/r/1196792

gerritbot added a comment.Dec 10 2025, 10:29 AM

Change #1211551 abandoned by Arnaudb:

[operations/puppet@production] gerrit: re-enable backups on gerrit2003

Reason:

1217133

https://gerrit.wikimedia.org/r/1211551

gerritbot added a comment.Dec 10 2025, 10:30 AM

Change #1217134 had a related patch set uploaded (by Arnaudb; author: Arnaudb):

[operations/puppet@production] gerrit: re-enable backups and monitoring on gerrit2003

https://gerrit.wikimedia.org/r/1217134

ABran-WMF renamed this task from Gerrit failover process to Gerrit switchover process.Dec 16 2025, 6:17 AM
ABran-WMF updated the task description. (Show Details)
ABran-WMF created subtask T412779: Gerrit failover process.Dec 16 2025, 6:21 AM
ABran-WMF mentioned this in T412779: Gerrit failover process.
A_smart_kitten added a project: Gerrit.Dec 16 2025, 7:46 AM
ABran-WMF mentioned this in T414407: Gerrit topology evolution branch pick-up.Jan 13 2026, 6:50 AM
gerritbot added a comment.Feb 6 2026, 9:14 AM

Change #1237450 had a related patch set uploaded (by Arnaudb; author: Arnaudb):

[operations/puppet@production] gerrit: temporarily disable replication on gerrit2003

https://gerrit.wikimedia.org/r/1237450

ABran-WMF updated the task description. (Show Details)Feb 9 2026, 10:15 AM
gerritbot added a comment.Feb 9 2026, 1:40 PM

Change #1237450 merged by Arnaudb:

[operations/puppet@production] gerrit: temporarily disable replication on gerrit2003

https://gerrit.wikimedia.org/r/1237450

Jelto added a comment.Feb 10 2026, 12:12 PM

Regarding the DNS change, @Dzahn desribed the new situation here: T412779#11581372

The new central DNS CNAME which is used by the tcp-proxy and trafficserver is gerrit.discovery.wmnet. This entry currently points to:

gerrit                300 IN CNAME gerrit1003.wikimedia.org.

So for the switchover, the CNAME should be updated here.

The replica/spare hosts will not be behind the CDN most likely next week. So the DNS switch for those has to happen here.

gerritbot added a comment.Feb 11 2026, 10:04 AM

Change #1238686 had a related patch set uploaded (by Arnaudb; author: Arnaudb):

[operations/puppet@production] gerrit: unmask gerrit.service on gerrit2003

https://gerrit.wikimedia.org/r/1238686

gerritbot added a comment.Feb 11 2026, 12:39 PM

Change #1238686 merged by Arnaudb:

[operations/puppet@production] gerrit: unmask gerrit.service on gerrit2003

https://gerrit.wikimedia.org/r/1238686

gerritbot added a comment.Feb 11 2026, 1:41 PM

Change #1238708 had a related patch set uploaded (by Arnaudb; author: Arnaudb):

[operations/dns@master] gerrit: switchover from gerrit1003 to gerrit2003

https://gerrit.wikimedia.org/r/1238708

ABran-WMF updated the task description. (Show Details)Feb 11 2026, 1:42 PM
ABran-WMF updated the task description. (Show Details)Feb 11 2026, 2:42 PM
ABran-WMF updated the task description. (Show Details)Feb 11 2026, 4:42 PM
ABran-WMF updated the task description. (Show Details)
ABran-WMF updated the task description. (Show Details)Feb 12 2026, 8:52 AM
ABran-WMF created subtask T417246: Reimage gerrit1003.Feb 12 2026, 9:11 AM
ABran-WMF mentioned this in T417246: Reimage gerrit1003.
ABran-WMF created subtask T417247: Reimage gerrit2002.
ABran-WMF mentioned this in T417247: Reimage gerrit2002.
ABran-WMF created subtask T417279: Harmonize DNS on all gerrit instances.Feb 12 2026, 2:02 PM
gerritbot added a comment.Feb 12 2026, 3:00 PM

Change #1238376 had a related patch set uploaded (by Arnaudb; author: Arnaudb):

[operations/cookbooks@master] gerrit: update switchover related cookbooks

https://gerrit.wikimedia.org/r/1238376

gerritbot added a comment.Feb 12 2026, 3:18 PM

Change #1238376 merged by jenkins-bot:

[operations/cookbooks@master] gerrit: update switchover related cookbooks

https://gerrit.wikimedia.org/r/1238376

gerritbot added a comment.Feb 16 2026, 9:49 AM

Change #1238708 merged by Arnaudb:

[operations/dns@master] gerrit: switchover from gerrit1003 to gerrit2003

https://gerrit.wikimedia.org/r/1238708

gerritbot added a comment.Feb 16 2026, 9:49 AM

Change #1217133 merged by Arnaudb:

[operations/puppet@production] gerrit: Switchover gerrit1003 → gerrit2003

https://gerrit.wikimedia.org/r/1217133

gerritbot added a comment.Feb 16 2026, 10:24 AM

Change #1217134 merged by Arnaudb:

[operations/puppet@production] gerrit: re-enable backups and monitoring on gerrit2003

https://gerrit.wikimedia.org/r/1217134

ABran-WMF updated the task description. (Show Details)Feb 16 2026, 10:26 AM
ABran-WMF added a subtask: T417565: GerritHAProxyBackendUnavailable.Feb 16 2026, 11:06 AM
ABran-WMF added a subtask: T417566: GerritHAProxyServiceUnavailable.
ABran-WMF closed subtask T417566: GerritHAProxyServiceUnavailable as Resolved.
ABran-WMF mentioned this in T417566: GerritHAProxyServiceUnavailable.
ABran-WMF closed subtask T417565: GerritHAProxyBackendUnavailable as Resolved.
ABran-WMF mentioned this in T417565: GerritHAProxyBackendUnavailable.
ABran-WMF updated the task description. (Show Details)Feb 16 2026, 2:09 PM

The process has been updated, and the cookbook switchover instructions have also been added to the documentation.

gerritbot added a comment.Feb 16 2026, 2:26 PM

Change #1239693 had a related patch set uploaded (by Arnaudb; author: Arnaudb):

[operations/cookbooks@master] gerrit: rename failover cookbook to switchover

https://gerrit.wikimedia.org/r/1239693

ABran-WMF updated the task description. (Show Details)Feb 16 2026, 2:37 PM
ABran-WMF updated the task description. (Show Details)Feb 16 2026, 2:53 PM
gerritbot added a comment.Feb 16 2026, 2:56 PM

Change #1239693 merged by jenkins-bot:

[operations/cookbooks@master] gerrit: rename failover cookbook to switchover

https://gerrit.wikimedia.org/r/1239693

ABran-WMF closed this task as Resolved.Feb 16 2026, 2:57 PM
ABran-WMF closed subtask T412779: Gerrit failover process as Resolved.Feb 17 2026, 2:02 PM
gerritbot added a comment.Feb 18 2026, 8:55 AM

Change #1240213 had a related patch set uploaded (by Arnaudb; author: Arnaudb):

[operations/cookbooks@master] gerrit: clarify confirmation wording on read-only

https://gerrit.wikimedia.org/r/1240213

gerritbot added a comment.Feb 18 2026, 9:05 AM

Change #1240213 merged by jenkins-bot:

[operations/cookbooks@master] gerrit: clarify confirmation wording on read-only

https://gerrit.wikimedia.org/r/1240213

gerritbot added a comment.Feb 18 2026, 9:10 AM

Change #1240216 had a related patch set uploaded (by Arnaudb; author: Arnaudb):

[operations/cookbooks@master] gerrit: read-only confirmation wording tweaking

https://gerrit.wikimedia.org/r/1240216

gerritbot added a comment.Feb 18 2026, 9:15 AM

Change #1240217 had a related patch set uploaded (by Arnaudb; author: Arnaudb):

[operations/puppet@production] gerrit: remove read-only config

https://gerrit.wikimedia.org/r/1240217

gerritbot added a comment.Feb 18 2026, 10:09 AM

Change #1240216 merged by jenkins-bot:

[operations/cookbooks@master] gerrit: read-only confirmation wording tweaking

https://gerrit.wikimedia.org/r/1240216

ABran-WMF changed the status of subtask T417247: Reimage gerrit2002 from Open to Stalled.Feb 19 2026, 1:53 PM
ABran-WMF changed the status of subtask T417246: Reimage gerrit1003 from Open to Stalled.Feb 19 2026, 1:57 PM
ABran-WMF changed the status of subtask T417246: Reimage gerrit1003 from Stalled to In Progress.Feb 20 2026, 7:41 AM
ABran-WMF closed subtask T417246: Reimage gerrit1003 as Resolved.Feb 20 2026, 8:11 AM
ABran-WMF changed the status of subtask T417247: Reimage gerrit2002 from Stalled to Open.Feb 20 2026, 10:38 AM
ABran-WMF closed subtask T406334: Gerrit switchover between secondary instances as Resolved.Feb 24 2026, 1:09 PM
ABran-WMF changed the status of subtask T417247: Reimage gerrit2002 from Open to In Progress.Feb 24 2026, 1:15 PM
ABran-WMF closed subtask T338470: Rename gerrit2 unix user to gerrit and assign a fixed uid as Resolved.Feb 24 2026, 2:18 PM
ABran-WMF closed subtask T417247: Reimage gerrit2002 as Resolved.Feb 24 2026, 4:03 PM
hashar reopened subtask T338470: Rename gerrit2 unix user to gerrit and assign a fixed uid as Open.Feb 24 2026, 4:21 PM
jcrespo reopened subtask T417247: Reimage gerrit2002 as Open.Feb 24 2026, 5:21 PM
ABran-WMF closed subtask T417247: Reimage gerrit2002 as Resolved.Mar 2 2026, 1:54 PM
ABran-WMF closed subtask T338470: Rename gerrit2 unix user to gerrit and assign a fixed uid as Resolved.Mar 3 2026, 10:25 AM
gerritbot added a comment.Mar 24 2026, 2:17 PM

Change #1240217 abandoned by Arnaudb:

[operations/puppet@production] gerrit: remove read-only config

https://gerrit.wikimedia.org/r/1240217

ABran-WMF closed subtask T417279: Harmonize DNS on all gerrit instances as Resolved.Mar 27 2026, 6:13 AM
ABran-WMF mentioned this in T417137: Fix rsync commands to avoid issues during switch of Gerrit instances.Apr 28 2026, 1:42 PM
Content licensed under Creative Commons Attribution-ShareAlike (CC BY-SA) 4.0 unless otherwise noted; code licensed under GNU General Public License (GPL) 2.0 or later and other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL · Credits