Page MenuHomePhabricator
Create Task
Maniphest T345531

Switchover gitlab (gitlab1004 -> gitlab2002) - October 2023
Closed, ResolvedPublic
Actions

Description

GitLab will be switched during September 2023 Datacenter Switchover from eqiad to codfw (one week after the actual switchover, to not block dependencies). This task tracks the failover of the GitLab production instance in eqiad (gitlab1004) to codfw (gitlab2002).

Docs: https://wikitech.wikimedia.org/wiki/GitLab/Failover
Time: Thursday, 5th October 2023, 09:00 UTC

Checklist:

Preparations before downtime:

  • prepare change to set profile::gitlab::service_name: 'gitlab.wikimedia.org' on gitlab2002, set gitlab2002 as profile::gitlab::active_host:, and set profile::gitlab::service_name: 'gitlab-replica-old.wikimedia.org on gitlab1004 (operations/puppet/+/963160/)
  • Prepare change to point DNS entry for gitlab.wikimedia.org to gitlab2002, and gitlab-replica-old.wikimedia.org to gitlab1004 (operations/dns/+/963161)
  • apply gitlab-settings to gitlab2002 and gitlab1004 (releng/gitlab-settings/-/merge_requests/46/diffs)
  • announce downtime some days ahead on ops/releng list/broadcast message
  • run a failover backup on the source host one day in advance sudo /srv/gitlab-backup/gitlab-backup.sh failover

Scheduled downtime:

  • Announce downtime in #wikimedia-gitlab
  • Start gitlab failover cookbook on the cumin host with cookbook sre.gitlab.failover --switch-from gitlab1004 --switch-to gitlab2002 -t T345531
  • When prompted, merge the puppet change prepared above
  • When prompted, merge the DNS change prepared above and run `authdns-update on the DNS master, following the DNS update instructions
  • Update https://wikitech.wikimedia.org/wiki/GitLab to reflect the new reality
  • Announce end of downtime

Falling back to manual steps:

If, for some reason, the cookbook cannot be used, the manual steps for failing over can be used here:

  • Announce downtime in #wikimedia-gitlab
  • pause all GitLab Runners (gitlab-settings ./runners active | tee active.txt && ./runners pause < active.txt)
  • downtime gitlab1004 sudo cookbook sre.hosts.downtime -r "Running failover to gitlab2002 - T329931" -M 120 'gitlab1004.wikimedia.org'
  • stop puppet on gitlab1004 with sudo disable-puppet "Running failover to gitlab2002 - T329931"
  • stop GitLab on gitlab1004 with gitlab-ctl stop nginx
  • stop ssh-gitlab daemon on gitlab1004 with systemctl stop ssh-gitlab
  • create full backup on gitlab1004 with /usr/bin/gitlab-backup create CRON=1 GZIP_RSYNCABLE="true" GITLAB_BACKUP_MAX_CONCURRENCY="4" GITLAB_BACKUP_MAX_STORAGE_CONCURRENCY="1"
  • sync backup, on gitlab1004 run /usr/bin/rsync -avp /srv/gitlab-backup/ rsync://gitlab2002.wikimedia.org/data-backup
  • merge change to set profile::gitlab::service_name: 'gitlab.wikimedia.org' on gitlab2002 and run puppet
  • trigger restore on gitlab2002 run sudo systemctl start backup-restore.service (for logs, run journalctl -f -u backup-restore.service)
  • merge change to point DNS entry for gitlab.wikimedia.org to gitlab2002 gitlab-replica-old.wikimedia.org to gitlab1004
  • verify installation
  • enable puppet on gitlab1004 with sudo run-puppet-agent -e "Running failover to gitlab2002 - T329931"
  • start ssh-gitlab daemon on gitlab1004 with systemctl start ssh-gitlab
  • unpause all GitLab Runners (gitlab-settings ./runners unpause < active.txt)
  • announce end of downtime

Details

Related Changes in Gerrit:
SubjectRepoBranchLines +/-
gitlab: install warning banner only on replicas when doing a restoreoperations/puppetproduction+8 -4
gitlab/failover: remove deploy-page at the end of cookbookoperations/cookbooksmaster+2 -0
gitlab: make gitlab2002 the active hostoperations/puppetproduction+2 -2
[gitlab/switchover] Update DNS for gitlab/gitlab-replicaoperations/dnsmaster+8 -8
[gitlab/switchover] Change profile::gitlab::service_name for switchoveroperations/puppetproduction+2 -2
Customize query in gerrit

Related Objects
Search...

Event Timeline

• eoghan created this task.Sep 4 2023, 7:53 AM
Restricted Application added a subscriber: Aklapper. · View Herald TranscriptSep 4 2023, 7:53 AM
LSobanski subscribed.Sep 4 2023, 1:06 PM
LSobanski triaged this task as Medium priority.Sep 4 2023, 3:15 PM
LSobanski moved this task from Incoming to Backlog on the collaboration-services board.
• eoghan mentioned this in T345590: Switchover gitlab-replica (gitlab2002 -> gitlab1003) - September 2023.Sep 4 2023, 8:41 PM
• eoghan added a project: GitLab (Infrastructure).
• eoghan updated the task description. (Show Details)
• eoghan added subscribers: Jelto, Arnoldokoth.
LSobanski renamed this task from Switchover gitlab (gitlab1004 -> gitlab2002) to Switchover gitlab (gitlab1004 -> gitlab2002) - September 2023.Sep 5 2023, 12:04 PM
LSobanski added a parent task: T345263: September 2023 Datacenter Switchover.Sep 5 2023, 12:10 PM
Jelto assigned this task to • eoghan.Sep 25 2023, 9:47 AM
LSobanski moved this task from Backlog to Work in Progress on the collaboration-services board.Sep 25 2023, 3:43 PM
LSobanski updated the task description. (Show Details)Sep 27 2023, 1:10 PM
thcipriani subscribed.Sep 27 2023, 5:00 PM

Aside from announcing in IRC, we should probably also announce on Wikitech.

Jelto added a comment.Sep 27 2023, 5:02 PM
In T345531#9204242, @thcipriani wrote:

Aside from announcing in IRC, we should probably also announce on Wikitech.

Please note that we moved the switchover to Thursday, 5th October 2023, 09:00 UTC. We'll do a bit more testing with the replica switchover first T345590.

• eoghan reassigned this task from • eoghan to Jelto.Oct 2 2023, 3:30 PM
• eoghan renamed this task from Switchover gitlab (gitlab1004 -> gitlab2002) - September 2023 to Switchover gitlab (gitlab1004 -> gitlab2002) - October 2023.Oct 3 2023, 8:07 AM
Trizek-WMF subscribed.Oct 3 2023, 1:09 PM

This switchover wasn't part of T345265: CommRel support for September 2023 Datacenter Switchover, so it wasn't announced the same way the Mediawiki switchover was covered.

In T345531#9204242, @thcipriani wrote:

Aside from announcing in IRC, we should probably also announce on Wikitech.

I got the news from wikitech-l.
Next time, it should also be in Tech News. :)

gerritbot added a comment.Oct 3 2023, 9:25 PM

Change 963160 had a related patch set uploaded (by EoghanGaffney; author: EoghanGaffney):

[operations/puppet@production] [gitlab/switchover] Change profile::gitlab::service_name for switchover

https://gerrit.wikimedia.org/r/963160

gerritbot added a project: Patch-For-Review.Oct 3 2023, 9:25 PM
gerritbot added a comment.Oct 3 2023, 9:28 PM

Change 963161 had a related patch set uploaded (by EoghanGaffney; author: EoghanGaffney):

[operations/dns@master] [gitlab/switchover] Update DNS for gitlab/gitlab-replica

https://gerrit.wikimedia.org/r/963161

CodeReviewBot added a comment.Oct 3 2023, 9:46 PM

eoghan opened https://gitlab.wikimedia.org/repos/releng/gitlab-settings/-/merge_requests/46

[gitlab/switchover] Update home_page_url for gitlab/gitlab-replica

• eoghan updated the task description. (Show Details)Oct 3 2023, 9:49 PM
Jelto updated the task description. (Show Details)Oct 4 2023, 12:44 PM
Jelto updated the task description. (Show Details)Oct 4 2023, 1:40 PM
Jelto updated the task description. (Show Details)
CodeReviewBot added a comment.Oct 5 2023, 8:44 AM

jelto merged https://gitlab.wikimedia.org/repos/releng/gitlab-settings/-/merge_requests/46

[gitlab/switchover] Update home_page_url for gitlab/gitlab-replica

Jelto updated the task description. (Show Details)Oct 5 2023, 8:55 AM
ops-monitoring-bot added a comment.Oct 5 2023, 9:01 AM

Cookbook cookbooks.sre.gitlab.failover (Failover of gitlab from gitlab1004.wikimedia.org to gitlab2002.wikimedia.org) started

gerritbot added a comment.Oct 5 2023, 10:40 AM

Change 963160 merged by Jelto:

[operations/puppet@production] [gitlab/switchover] Change profile::gitlab::service_name for switchover

https://gerrit.wikimedia.org/r/963160

gerritbot added a comment.Oct 5 2023, 11:21 AM

Change 963161 merged by Jelto:

[operations/dns@master] [gitlab/switchover] Update DNS for gitlab/gitlab-replica

https://gerrit.wikimedia.org/r/963161

Maintenance_bot removed a project: Patch-For-Review.Oct 5 2023, 11:30 AM
gerritbot added a comment.Oct 5 2023, 11:35 AM

Change 963706 had a related patch set uploaded (by Jelto; author: Jelto):

[operations/puppet@production] gitlab: make gitlab2002 the active host

https://gerrit.wikimedia.org/r/963706

gerritbot added a project: Patch-For-Review.Oct 5 2023, 11:35 AM

Change 963706 merged by Jelto:

[operations/puppet@production] gitlab: make gitlab2002 the active host

https://gerrit.wikimedia.org/r/963706

ops-monitoring-bot added a comment.Oct 5 2023, 11:46 AM

Cookbook cookbooks.sre.gitlab.failover (Failover of gitlab from gitlab1004.wikimedia.org to gitlab2002.wikimedia.org) finished

LSobanski mentioned this in T348234: ProbeDown - GitLab.Oct 5 2023, 12:10 PM
Maintenance_bot removed a project: Patch-For-Review.Oct 5 2023, 12:10 PM
LSobanski updated the task description. (Show Details)Oct 5 2023, 12:26 PM
Jelto updated the task description. (Show Details)Oct 5 2023, 1:32 PM
gerritbot added a comment.Oct 5 2023, 2:00 PM

Change 963739 had a related patch set uploaded (by Jelto; author: Jelto):

[operations/cookbooks@master] gitlab/failover: remove deploy-page at the end of cookbook

https://gerrit.wikimedia.org/r/963739

gerritbot added a project: Patch-For-Review.Oct 5 2023, 2:00 PM
Trizek-WMF mentioned this in T345265: CommRel support for September 2023 Datacenter Switchover.Oct 6 2023, 10:20 AM
gerritbot added a comment.Oct 6 2023, 12:12 PM

Change 963739 merged by jenkins-bot:

[operations/cookbooks@master] gitlab/failover: remove deploy-page at the end of cookbook

https://gerrit.wikimedia.org/r/963739

Maintenance_bot removed a project: Patch-For-Review.Oct 6 2023, 12:30 PM
gerritbot added a comment.Oct 6 2023, 12:33 PM

Change 964003 had a related patch set uploaded (by Jelto; author: Jelto):

[operations/puppet@production] gitlab: install warning banner only on replicas when doing a restore

https://gerrit.wikimedia.org/r/964003

gerritbot added a project: Patch-For-Review.Oct 6 2023, 12:33 PM
gerritbot added a comment.Oct 9 2023, 2:45 PM

Change 964003 merged by Jelto:

[operations/puppet@production] gitlab: install warning banner only on replicas when doing a restore

https://gerrit.wikimedia.org/r/964003

Jelto closed this task as Resolved.Oct 9 2023, 2:46 PM

production GitLab was switched from gitlab1004 to gitlab2002 in codfw. I'll close the task.

Maintenance_bot removed a project: Patch-For-Review.Oct 9 2023, 3:11 PM
Content licensed under Creative Commons Attribution-ShareAlike (CC BY-SA) 4.0 unless otherwise noted; code licensed under GNU General Public License (GPL) 2.0 or later and other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL · Credits