Page MenuHomePhabricator
Create Task
Maniphest T387833

Gerrit switchover process
Open, In Progress, HighPublic
Actions

Description

A part of the effort to standardize failover procedures for Collab services.

Details

Related Changes in Gerrit:
SubjectRepoBranchLines +/-
gerrit: Switchover gerrit1003 → gerrit2003operations/puppetproduction+134 -65
gerrit: update switchover related cookbooksoperations/cookbooksmaster+23 -23
gerrit: re-enable backups and monitoring on gerrit2003operations/puppetproduction+2 -2
gerrit: switchover from gerrit1003 to gerrit2003operations/dnsmaster+5 -6
gerrit: unmask gerrit.service on gerrit2003operations/puppetproduction+1 -1
gerrit: temporarily disable replication on gerrit2003operations/puppetproduction+4 -4
gerrit: re-enable backups on gerrit2003operations/puppetproduction+1 -1
gerrit: unmask serviceoperations/puppetproduction+3 -3
gerrit: Switchover gerrit1003 → gerrit2003operations/puppetproduction+79 -40
gerrit: add a confirmation prompt on rsyncoperations/cookbooksmaster+2 -0
gerrit: testing syncoperations/cookbooksmaster+15 -14
gerrit: disable backups temporarilyoperations/puppetproduction+2 -0
gerrit: rsync logic extraction from failoveroperations/cookbooksmaster+289 -129
gerrit: add a layer of CNAME to ease switch oversoperations/dnsmaster+25 -13
gerrit: Switchover gerrit1003 → gerrit2003operations/puppetproduction+79 -40
gerrit: remove localbackup logic from failoveroperations/cookbooksmaster+8 -48
gerrit: add a local backup cookbookoperations/cookbooksmaster+144 -0
gerrit: add dry run rsyncoperations/cookbooksmaster+12 -6
gerrit: remove localbackup logic from failoveroperations/cookbooksmaster+7 -43
gerrit: enable monitoring for other instancesoperations/puppetproduction+25 -0
gerrit: stop stopping gerrit.serviceoperations/cookbooksmaster+0 -8
gerrit: stop puppet across all instancesoperations/cookbooksmaster+6 -4
gerrit: rsync and chown fixesoperations/cookbooksmaster+12 -7
gerrit: disable gerrit service to enable backupsoperations/puppetproduction+2 -2
gerrit: ask the operator to merge puppet earlieroperations/cookbooksmaster+5 -3
gerrit: typo fix in post_sync_validationoperations/cookbooksmaster+1 -1
gerrit: re-enable backups on gerrit2003operations/puppetproduction+0 -2
Revert^4 "gerrit: switchover from gerrit1003 to gerrit2003"operations/dnsmaster+8 -8
Revert^4 "gerrit: Switchover gerrit1003 → gerrit2003"operations/puppetproduction+70 -33
gerrit: local backup on source server onlyoperations/cookbooksmaster+4 -4
Revert^2 "gerrit: switchover from gerrit1003 to gerrit2003"operations/dnsmaster+8 -8
Revert^2 "gerrit: Switchover gerrit1003 → gerrit2003"operations/puppetproduction+70 -33
gerrit: Switchover gerrit1003 → gerrit2003operations/puppetproduction+70 -33
Add a banner for a Gerrit switch over maintenanceoperations/software/gerritdeploy/wmf/stable-3.10+114 -0
Disable component rather than motd pluginoperations/software/gerritdeploy/wmf/stable-3.10+4 -4
Disable motd banner: maintenance window has closedoperations/software/gerritdeploy/wmf/stable-3.10+5 -3
gerrit: fix typo in source pathoperations/cookbooksmaster+9 -10
gerrit: test 1193860operations/cookbooksmaster+98 -97
gerrit: switchover from gerrit1003 to gerrit2003operations/dnsmaster+8 -8
gerrit: avoid hardcoded hostnames, replace with hiera lookupsoperations/puppetproduction+5 -3
gerrit: typo on --systemd argoperations/cookbooksmaster+5 -3
gerrit: bugfixes on failoveroperations/cookbooksmaster+24 -9
gerrit: replace host names in replica config with variablesoperations/puppetproduction+2 -2
gerrit: Switchover gerrit1003 → gerrit2003operations/puppetproduction+46 -30
gerrit: add spare fqdn to apache vhostoperations/puppetproduction+19 -0
gerrit: add certificate request for gerrit-spare.w.ooperations/puppetproduction+1 -0
gerrit: site.pp cleanupoperations/puppetproduction+1 -6
gerrit: add httpbb monitoring to gerrit-spareoperations/puppetproduction+19 -0
gerrit: add service ip address for gerrit2003operations/puppetproduction+5 -3
Gerrit: Add service ip for gerrit2003operations/dnsmaster+8 -3
gerrit: Bugfixes - dry run testsoperations/cookbooksmaster+64 -218
gerrit: rename host and replica_hosts array to primary/replica_vhostoperations/puppetproduction+18 -21
gerrit: also rename "passive" to "spare" server in MOTDoperations/puppetproduction+1 -1
gerrit: replace hardcoded host name and codfw string for replicaoperations/puppetproduction+11 -2
Add readonly puginoperations/software/gerritdeploy/wmf/stable-3.10+3 -0
gerrit: standardize expected rc on systemctl checkoperations/cookbooksmaster+1 -1
gerrit: sanity checks cookbook implementationoperations/cookbooksmaster+52 -150
gerrit: sanity checks as a cookbookoperations/cookbooksmaster+304 -0
gerrit: add readonly pluginoperations/software/gerritwmf/stable-3.10+4 -0
gerrit: add readonly pluginoperations/software/gerritwmf/stable-3.10+4 -0
gerrit: add readonly.jaroperations/software/gerritdeploy/wmf/stable-3.10+0 -0
gerrit: introduce second daemon_user nameoperations/puppetproduction+3 -2
gerrit: replace gerrit2003 RSA host key with ed25519 host keyoperations/puppetproduction+1 -1
gerrit: add a second replica, start replicating to gerrit2003operations/puppetproduction+16 -8
gerrit: add ssh_host_rsa public key for gerrit2003operations/puppetproduction+1 -0
gerrit: switchover to gerrit1003operations/puppetproduction+5 -3
gerrit: switchover to gerrit1003operations/dnsmaster+8 -8
gerrit: switchover to gerrit2002operations/puppetproduction+3 -3
gerrit: switchover to gerrit2002operations/dnsmaster+8 -8
Show related patches Customize query in gerrit

Related Objects
Search...

StatusSubtypeAssignedTask
 OpenNoneT407557 OpenSSH 10.1+ warns that Wikimedia SSH does not use post-quantum key exchange algorithm
 OpenNoneT407844 Gerrit ssh daemon does not offer post-quantum kex leading to a warning with OpenSSH 10
 OpenNoneT392448 Upgrade to Gerrit 3.12
 OpenNoneT379714 Upgrade to Gerrit 3.11
 OpenNoneT392465 Switch Gerrit from Java 17 to Java 21
 OpenNoneT384595 Upgrade Collab hosts to Bookworm
 OpenNoneT392464 Upgrade Gerrit hosts from Bullseye to Bookworm
 OpenNoneT387831 Standardize failover procedures for Collab services
 ResolvedNoneT393239 ProbeDown
 In ProgressABran-WMFT387833 Gerrit switchover process
 OpenNoneT388507 Investigate missing commit from old changes
 ResolvedABran-WMFT260666 Create a cookbook to automate gerrit's switchover
 ResolvedABran-WMFT393050 ProbeDown
 In ProgressABran-WMFT338470 Rename gerrit2 unix user to gerrit and assign a fixed uid
 ResolvedDzahnT379213 PuppetFailure - gerrit2003
 ResolvedMatthewVernonT392186 Grant Gerrit admin to arnaudb
 ResolvedLSobanskiT393034 Investigate out of date refs following gerrit switchover
 ResolvedABran-WMFT395440 Gerrit read-only plugin test
 OpenABran-WMFT406334 Gerrit switchover between secondary instances
 ResolvedLSobanskiT406482 SystemdUnitFailed (jenkins.service on contint1002)
 ResolvedhasharT406762 gerrit2003 is trying to backup incrementally 3.5 million files every hour, clogging backus and filling in available disk space
 ResolvedABran-WMFT411583 Gerrit backups are growing
 ResolvedhasharT406856 Reduce size of analytics/superset/deploy.git Gerrit repo
Restricted Task
 OpenABran-WMFT412779 Gerrit failover process
 OpenABran-WMFT417246 Reimage gerrit1003
 OpenABran-WMFT417247 Reimage gerrit2002
 OpenNoneT417279 Harmonize DNS on all gerrit instances

Event Timeline

There are a very large number of changes, so older changes are hidden. Show Older Changes
gerritbot added a comment.Oct 7 2025, 9:18 AM

Change #1193860 merged by jenkins-bot:

[operations/cookbooks@master] gerrit: fix typo in source path

https://gerrit.wikimedia.org/r/1193860

ABran-WMF updated the task description. (Show Details)Oct 7 2025, 2:04 PM
gerritbot added a comment.Oct 7 2025, 2:55 PM

Change #1194220 had a related patch set uploaded (by Hashar; author: Hashar):

[operations/software/gerrit@deploy/wmf/stable-3.10] Disable motd banner: maintenance window has closed

https://gerrit.wikimedia.org/r/1194220

gerritbot added a comment.Oct 7 2025, 3:01 PM

Change #1194220 merged by jenkins-bot:

[operations/software/gerrit@deploy/wmf/stable-3.10] Disable motd banner: maintenance window has closed

https://gerrit.wikimedia.org/r/1194220

Stashbot added a comment.Oct 7 2025, 3:03 PM

Mentioned in SAL (#wikimedia-operations) [2025-10-07T15:03:16Z] <hashar@deploy2002> Started deploy [gerrit/gerrit@21d2848]: Disable motd banner: maintenance window has closed - T387833

Stashbot added a comment.Oct 7 2025, 3:03 PM

Mentioned in SAL (#wikimedia-operations) [2025-10-07T15:03:37Z] <hashar@deploy2002> Finished deploy [gerrit/gerrit@21d2848]: Disable motd banner: maintenance window has closed - T387833 (duration: 00m 30s)

gerritbot added a comment.Oct 7 2025, 3:21 PM

Change #1194225 had a related patch set uploaded (by Hashar; author: Hashar):

[operations/software/gerrit@deploy/wmf/stable-3.10] Disable component rather than motd plugin

https://gerrit.wikimedia.org/r/1194225

gerritbot added a comment.Oct 7 2025, 3:23 PM

Change #1194225 merged by jenkins-bot:

[operations/software/gerrit@deploy/wmf/stable-3.10] Disable component rather than motd plugin

https://gerrit.wikimedia.org/r/1194225

ABran-WMF updated the task description. (Show Details)Oct 9 2025, 6:06 AM
gerritbot added a comment.Oct 9 2025, 12:01 PM

Change #1193845 merged by Arnaudb:

[operations/puppet@production] Revert^2 "gerrit: Switchover gerrit1003 → gerrit2003"

https://gerrit.wikimedia.org/r/1193845

gerritbot added a comment.Oct 9 2025, 12:02 PM

Change #1193846 merged by Arnaudb:

[operations/dns@master] Revert^2 "gerrit: switchover from gerrit1003 to gerrit2003"

https://gerrit.wikimedia.org/r/1193846

ABran-WMF added a subtask: T406762: gerrit2003 is trying to backup incrementally 3.5 million files every hour, clogging backus and filling in available disk space.Oct 9 2025, 1:17 PM
gerritbot added a comment.Oct 9 2025, 1:20 PM

Change #1194932 had a related patch set uploaded (by Arnaudb; author: Arnaudb):

[operations/dns@master] Revert^4 "gerrit: switchover from gerrit1003 to gerrit2003"

https://gerrit.wikimedia.org/r/1194932

gerritbot added a comment.Oct 9 2025, 1:21 PM

Change #1194931 had a related patch set uploaded (by Arnaudb; author: Arnaudb):

[operations/puppet@production] Revert^4 "gerrit: Switchover gerrit1003 → gerrit2003"

https://gerrit.wikimedia.org/r/1194931

gerritbot added a comment.Oct 9 2025, 1:58 PM

Change #1194949 had a related patch set uploaded (by Arnaudb; author: Arnaudb):

[operations/cookbooks@master] gerrit: local backup on source server only

https://gerrit.wikimedia.org/r/1194949

ABran-WMF added a subtask: T406856: Reduce size of analytics/superset/deploy.git Gerrit repo.Oct 9 2025, 2:23 PM
ABran-WMF added a comment.EditedOct 13 2025, 4:47 AM

things are looking better now:

arnaudb@gerrit2003:git $ fd | wc -l
236218
arnaudb@gerrit2003:git $ pwd
/srv/gerrit/git
arnaudb@gerrit2003:git $ ls -l /srv/backup/
total 0

vs the previous situation:

In T406762#11259697, @hashar wrote:
gerrit 2003$ find /srv/gerrit/git |wc -l
5037253

That is 5 millions files.

[...]

 JobId  Level      Files    Bytes   Status   Finished        Name 
====================================================================
656136  Incr    3,745,079    54.98 G  OK       08-Oct-25 13:09 gerrit2003.wikimedia.org-Hourly-Mon-productionEqiad-gerrit-repo-data
ABran-WMF mentioned this in T406762: gerrit2003 is trying to backup incrementally 3.5 million files every hour, clogging backus and filling in available disk space.Oct 13 2025, 4:49 AM
gerritbot added a comment.Oct 13 2025, 4:52 AM

Change #1195432 had a related patch set uploaded (by Arnaudb; author: Arnaudb):

[operations/puppet@production] gerrit: re-enable backups on gerrit2003

https://gerrit.wikimedia.org/r/1195432

gerritbot added a comment.Oct 13 2025, 5:38 AM

Change #1195437 had a related patch set uploaded (by Arnaudb; author: Arnaudb):

[operations/cookbooks@master] gerrit: add dry run rsync

https://gerrit.wikimedia.org/r/1195437

ABran-WMF updated the task description. (Show Details)Oct 13 2025, 7:29 AM
gerritbot added a comment.Oct 13 2025, 8:34 AM

Change #1194949 merged by jenkins-bot:

[operations/cookbooks@master] gerrit: local backup on source server only

https://gerrit.wikimedia.org/r/1194949

gerritbot added a comment.Oct 14 2025, 9:00 AM

Change #1194932 merged by Arnaudb:

[operations/dns@master] Revert^4 "gerrit: switchover from gerrit1003 to gerrit2003"

https://gerrit.wikimedia.org/r/1194932

gerritbot added a comment.Oct 14 2025, 9:00 AM

Change #1194931 merged by Arnaudb:

[operations/puppet@production] Revert^4 "gerrit: Switchover gerrit1003 → gerrit2003"

https://gerrit.wikimedia.org/r/1194931

LSobanski added a subtask: Restricted Task.Oct 14 2025, 11:58 AM
gerritbot added a comment.Oct 14 2025, 12:31 PM

Change #1196051 had a related patch set uploaded (by Arnaudb; author: Arnaudb):

[operations/cookbooks@master] gerrit: typo fix in post_sync_validation

https://gerrit.wikimedia.org/r/1196051

ABran-WMF closed subtask Restricted Task as Resolved.Oct 14 2025, 12:43 PM
gerritbot added a comment.Oct 15 2025, 6:33 AM

Change #1196227 had a related patch set uploaded (by Arnaudb; author: Arnaudb):

[operations/cookbooks@master] gerrit: ask the operator to merge puppet earlier

https://gerrit.wikimedia.org/r/1196227

gerritbot added a comment.Oct 15 2025, 6:27 PM

Change #1195432 merged by Dzahn:

[operations/puppet@production] gerrit: re-enable backups on gerrit2003

https://gerrit.wikimedia.org/r/1195432

gerritbot added a comment.Oct 15 2025, 6:28 PM

Change #1196051 merged by jenkins-bot:

[operations/cookbooks@master] gerrit: typo fix in post_sync_validation

https://gerrit.wikimedia.org/r/1196051

gerritbot added a comment.Oct 16 2025, 8:45 AM

Change #1196629 had a related patch set uploaded (by Arnaudb; author: Arnaudb):

[operations/puppet@production] gerrit: disable gerrit service to enable backups

https://gerrit.wikimedia.org/r/1196629

gerritbot added a comment.Oct 16 2025, 8:47 AM

Change #1196227 merged by jenkins-bot:

[operations/cookbooks@master] gerrit: ask the operator to merge puppet earlier

https://gerrit.wikimedia.org/r/1196227

gerritbot added a comment.Oct 16 2025, 1:21 PM

Change #1196684 had a related patch set uploaded (by Arnaudb; author: Arnaudb):

[operations/cookbooks@master] gerrit: rsync and chown fixes

https://gerrit.wikimedia.org/r/1196684

gerritbot added a comment.Oct 16 2025, 1:56 PM

Change #1196694 had a related patch set uploaded (by Arnaudb; author: Arnaudb):

[operations/cookbooks@master] gerrit: stop puppet across all instances

https://gerrit.wikimedia.org/r/1196694

gerritbot added a comment.Oct 16 2025, 2:00 PM

Change #1196695 had a related patch set uploaded (by Arnaudb; author: Arnaudb):

[operations/cookbooks@master] gerrit: stop stopping gerrit.service

https://gerrit.wikimedia.org/r/1196695

gerritbot added a comment.Oct 16 2025, 8:30 PM

Change #1196629 merged by Dzahn:

[operations/puppet@production] gerrit: disable gerrit service to enable backups

https://gerrit.wikimedia.org/r/1196629

Dzahn added a comment.Oct 16 2025, 8:46 PM

Here are the notes / commands from a gerrit failover in the past:

https://phabricator.wikimedia.org/P47782

Dzahn added a comment.Oct 16 2025, 8:48 PM

Here is how we did the DNS change without having to merge while Gerrit is down:

  1. merge DNS change that removes gerrit-new and switches IP of gerrit.wikimedia.org - in web UI of gerrit(-old)
  2. run authdns-update on ns0.wikimedia.org, see the diff but do NOT commit yet
  3. disable puppet, stop gerrit, do the rsync, run chmod -R ...
  4. say "yes" to authdns-update and actually merge DNS change that removes gerrit-new and switches IP of gerrit.wikimedia.org
gerritbot added a comment.Oct 17 2025, 6:13 AM

Change #1196792 had a related patch set uploaded (by Arnaudb; author: Arnaudb):

[operations/puppet@production] gerrit: unmask service & disable backup temporarily

https://gerrit.wikimedia.org/r/1196792

ABran-WMF added a comment.Oct 17 2025, 6:40 AM
In T387833#11283327, @Dzahn wrote:

Here is how we did the DNS change without having to merge while Gerrit is down:

Thanks for the dig! I tweaked the process and the cookbook to be closer to this, the puppet merge timing was inconsistent with that approach.

ABran-WMF closed subtask T406762: gerrit2003 is trying to backup incrementally 3.5 million files every hour, clogging backus and filling in available disk space as Resolved.Oct 17 2025, 6:45 AM
gerritbot added a comment.Oct 17 2025, 6:49 AM

Change #1196684 merged by jenkins-bot:

[operations/cookbooks@master] gerrit: rsync and chown fixes

https://gerrit.wikimedia.org/r/1196684

gerritbot added a comment.Oct 17 2025, 6:50 AM

Change #1196694 merged by jenkins-bot:

[operations/cookbooks@master] gerrit: stop puppet across all instances

https://gerrit.wikimedia.org/r/1196694

gerritbot added a comment.Oct 17 2025, 7:49 AM

Change #1196695 merged by jenkins-bot:

[operations/cookbooks@master] gerrit: stop stopping gerrit.service

https://gerrit.wikimedia.org/r/1196695

hashar closed subtask T406856: Reduce size of analytics/superset/deploy.git Gerrit repo as Resolved.Oct 28 2025, 10:34 AM
gerritbot added a comment.Nov 19 2025, 10:50 AM

Change #1193599 abandoned by Arnaudb:

[operations/cookbooks@master] gerrit: remove localbackup logic from failover

Reason:

obsolete, will workaround the merge conflict with another change

https://gerrit.wikimedia.org/r/1193599

gerritbot added a comment.Nov 24 2025, 7:45 AM

Change #1210386 had a related patch set uploaded (by Arnaudb; author: Arnaudb):

[operations/cookbooks@master] gerrit: remove localbackup logic from failover

https://gerrit.wikimedia.org/r/1210386

gerritbot added a comment.Nov 24 2025, 7:46 AM

Change #1195437 merged by jenkins-bot:

[operations/cookbooks@master] gerrit: add dry run rsync

https://gerrit.wikimedia.org/r/1195437

gerritbot added a comment.Nov 24 2025, 7:46 AM

Change #1193590 merged by jenkins-bot:

[operations/cookbooks@master] gerrit: add a local backup cookbook

https://gerrit.wikimedia.org/r/1193590

gerritbot added a comment.Nov 24 2025, 11:41 AM

Change #1210560 had a related patch set uploaded (by Hashar; author: Hashar):

[operations/dns@master] gerrit: add a layer of CNAME to ease switch overs

https://gerrit.wikimedia.org/r/1210560

gerritbot added a comment.Nov 26 2025, 6:24 AM

Change #1210386 merged by jenkins-bot:

[operations/cookbooks@master] gerrit: remove localbackup logic from failover

https://gerrit.wikimedia.org/r/1210386

gerritbot added a comment.Nov 26 2025, 7:25 AM

Change #1211548 had a related patch set uploaded (by Arnaudb; author: Arnaudb):

[operations/puppet@production] gerrit: Switchover gerrit1003 → gerrit2003

https://gerrit.wikimedia.org/r/1211548

gerritbot added a comment.Nov 26 2025, 7:26 AM

Change #1211548 abandoned by Arnaudb:

[operations/puppet@production] gerrit: Switchover gerrit1003 → gerrit2003

Reason:

forgot a step

https://gerrit.wikimedia.org/r/1211548

gerritbot added a comment.Nov 26 2025, 7:29 AM

Change #1211549 had a related patch set uploaded (by Arnaudb; author: Arnaudb):

[operations/puppet@production] gerrit: Switchover gerrit1003 → gerrit2003

https://gerrit.wikimedia.org/r/1211549

gerritbot added a comment.Nov 26 2025, 7:33 AM

Change #1211551 had a related patch set uploaded (by Arnaudb; author: Arnaudb):

[operations/puppet@production] gerrit: re-enable backups on gerrit2003

https://gerrit.wikimedia.org/r/1211551

Dzahn reopened subtask T406762: gerrit2003 is trying to backup incrementally 3.5 million files every hour, clogging backus and filling in available disk space as Open.Dec 1 2025, 4:23 PM
hashar closed subtask T406762: gerrit2003 is trying to backup incrementally 3.5 million files every hour, clogging backus and filling in available disk space as Resolved.Dec 2 2025, 3:47 PM
gerritbot added a comment.Dec 3 2025, 10:05 AM

Change #1214466 had a related patch set uploaded (by Arnaudb; author: Arnaudb):

[operations/cookbooks@master] gerrit: rsync logic extraction from failover

https://gerrit.wikimedia.org/r/1214466

gerritbot added a comment.Dec 5 2025, 3:58 PM

Change #1210560 abandoned by Hashar:

[operations/dns@master] gerrit: add a layer of CNAME to ease switch overs

https://gerrit.wikimedia.org/r/1210560

gerritbot added a comment.Dec 8 2025, 9:49 AM

Change #1214466 merged by jenkins-bot:

[operations/cookbooks@master] gerrit: rsync logic extraction from failover

https://gerrit.wikimedia.org/r/1214466

gerritbot added a comment.Dec 8 2025, 1:21 PM

Change #1216583 had a related patch set uploaded (by Arnaudb; author: Arnaudb):

[operations/puppet@production] gerrit: disable backups temporarily

https://gerrit.wikimedia.org/r/1216583

gerritbot added a comment.Dec 8 2025, 1:24 PM

Change #1216583 merged by Arnaudb:

[operations/puppet@production] gerrit: disable backups temporarily

https://gerrit.wikimedia.org/r/1216583

gerritbot added a comment.Dec 8 2025, 1:30 PM

Change #1216586 had a related patch set uploaded (by Arnaudb; author: Arnaudb):

[operations/cookbooks@master] gerrit: testing sync

https://gerrit.wikimedia.org/r/1216586

gerritbot added a comment.Dec 8 2025, 2:20 PM

Change #1216586 abandoned by Arnaudb:

[operations/cookbooks@master] gerrit: testing sync

Reason:

testing over

https://gerrit.wikimedia.org/r/1216586

gerritbot added a comment.Dec 8 2025, 2:23 PM

Change #1216592 had a related patch set uploaded (by Arnaudb; author: Arnaudb):

[operations/cookbooks@master] gerrit: add a confirmation prompt on rsync

https://gerrit.wikimedia.org/r/1216592

gerritbot added a comment.Dec 10 2025, 10:17 AM

Change #1216592 merged by jenkins-bot:

[operations/cookbooks@master] gerrit: add a confirmation prompt on rsync

https://gerrit.wikimedia.org/r/1216592

gerritbot added a comment.Dec 10 2025, 10:28 AM

Change #1217133 had a related patch set uploaded (by Arnaudb; author: Arnaudb):

[operations/puppet@production] gerrit: Switchover gerrit1003 → gerrit2003

https://gerrit.wikimedia.org/r/1217133

gerritbot added a comment.Dec 10 2025, 10:29 AM

Change #1211549 abandoned by Arnaudb:

[operations/puppet@production] gerrit: Switchover gerrit1003 → gerrit2003

Reason:

1217133

https://gerrit.wikimedia.org/r/1211549

gerritbot added a comment.Dec 10 2025, 10:29 AM

Change #1196792 abandoned by Arnaudb:

[operations/puppet@production] gerrit: unmask service

Reason:

1217133

https://gerrit.wikimedia.org/r/1196792

gerritbot added a comment.Dec 10 2025, 10:29 AM

Change #1211551 abandoned by Arnaudb:

[operations/puppet@production] gerrit: re-enable backups on gerrit2003

Reason:

1217133

https://gerrit.wikimedia.org/r/1211551

gerritbot added a comment.Dec 10 2025, 10:30 AM

Change #1217134 had a related patch set uploaded (by Arnaudb; author: Arnaudb):

[operations/puppet@production] gerrit: re-enable backups and monitoring on gerrit2003

https://gerrit.wikimedia.org/r/1217134

ABran-WMF renamed this task from Gerrit failover process to Gerrit switchover process.Dec 16 2025, 6:17 AM
ABran-WMF updated the task description. (Show Details)
ABran-WMF created subtask T412779: Gerrit failover process.Dec 16 2025, 6:21 AM
ABran-WMF mentioned this in T412779: Gerrit failover process.
A_smart_kitten added a project: Gerrit.Dec 16 2025, 7:46 AM
ABran-WMF mentioned this in T414407: Gerrit topology evolution branch pick-up.Jan 13 2026, 6:50 AM
gerritbot added a comment.Fri, Feb 6, 9:14 AM

Change #1237450 had a related patch set uploaded (by Arnaudb; author: Arnaudb):

[operations/puppet@production] gerrit: temporarily disable replication on gerrit2003

https://gerrit.wikimedia.org/r/1237450

ABran-WMF updated the task description. (Show Details)Mon, Feb 9, 10:15 AM
gerritbot added a comment.Mon, Feb 9, 1:40 PM

Change #1237450 merged by Arnaudb:

[operations/puppet@production] gerrit: temporarily disable replication on gerrit2003

https://gerrit.wikimedia.org/r/1237450

Jelto added a comment.Tue, Feb 10, 12:12 PM

Regarding the DNS change, @Dzahn desribed the new situation here: T412779#11581372

The new central DNS CNAME which is used by the tcp-proxy and trafficserver is gerrit.discovery.wmnet. This entry currently points to:

gerrit                300 IN CNAME gerrit1003.wikimedia.org.

So for the switchover, the CNAME should be updated here.

The replica/spare hosts will not be behind the CDN most likely next week. So the DNS switch for those has to happen here.

gerritbot added a comment.Wed, Feb 11, 10:04 AM

Change #1238686 had a related patch set uploaded (by Arnaudb; author: Arnaudb):

[operations/puppet@production] gerrit: unmask gerrit.service on gerrit2003

https://gerrit.wikimedia.org/r/1238686

gerritbot added a comment.Wed, Feb 11, 12:39 PM

Change #1238686 merged by Arnaudb:

[operations/puppet@production] gerrit: unmask gerrit.service on gerrit2003

https://gerrit.wikimedia.org/r/1238686

gerritbot added a comment.Wed, Feb 11, 1:41 PM

Change #1238708 had a related patch set uploaded (by Arnaudb; author: Arnaudb):

[operations/dns@master] gerrit: switchover from gerrit1003 to gerrit2003

https://gerrit.wikimedia.org/r/1238708

ABran-WMF updated the task description. (Show Details)Wed, Feb 11, 1:42 PM
ABran-WMF updated the task description. (Show Details)Wed, Feb 11, 2:42 PM
ABran-WMF updated the task description. (Show Details)Wed, Feb 11, 4:42 PM
ABran-WMF updated the task description. (Show Details)
ABran-WMF updated the task description. (Show Details)Thu, Feb 12, 8:52 AM
ABran-WMF created subtask T417246: Reimage gerrit1003.Thu, Feb 12, 9:11 AM
ABran-WMF mentioned this in T417246: Reimage gerrit1003.
ABran-WMF created subtask T417247: Reimage gerrit2002.
ABran-WMF mentioned this in T417247: Reimage gerrit2002.
ABran-WMF created subtask T417279: Harmonize DNS on all gerrit instances.Thu, Feb 12, 2:02 PM
gerritbot added a comment.Thu, Feb 12, 3:00 PM

Change #1238376 had a related patch set uploaded (by Arnaudb; author: Arnaudb):

[operations/cookbooks@master] gerrit: update switchover related cookbooks

https://gerrit.wikimedia.org/r/1238376

gerritbot added a comment.Thu, Feb 12, 3:18 PM

Change #1238376 merged by jenkins-bot:

[operations/cookbooks@master] gerrit: update switchover related cookbooks

https://gerrit.wikimedia.org/r/1238376

Content licensed under Creative Commons Attribution-ShareAlike (CC BY-SA) 4.0 unless otherwise noted; code licensed under GNU General Public License (GPL) 2.0 or later and other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL · Credits