Page MenuHomePhabricator
Create Task
Maniphest T400121

Gitlab-replica switchover (gitlab1004 → gitlab1003)
Closed, ResolvedPublic
Actions

Description

Docs: https://wikitech.wikimedia.org/wiki/GitLab/Failover

Checklist:

Preparations before downtime:

  • prepare the required Puppet changes (patch)
  • Prepare the required DNS changes (patch)
  • apply gitlab-settings to gitlab1003 and gitlab1004 (merge request)
  • announce downtime some days ahead on ops/releng list/broadcast message not needed for replicas
  • run a failover backup on the source host one day in advance sudo /srv/gitlab-backup/gitlab-backup.sh failover - we should double check what is the purpose of this

Scheduled downtime:

  • Announce downtime in #wikimedia-gitlab
  • Start gitlab failover cookbook on the cumin host with cookbook sre.gitlab.failover --switch-from gitlab1004 --switch-to gitlab1003 -t T400121
  • When prompted, merge the puppet change prepared above
  • When prompted, merge the DNS change prepared above
  • run authdns-update on the DNS master, following the DNS update instructions

Details

Related Changes in Gerrit:
SubjectRepoBranchLines +/-
gitlab failover: improve message for API tokenoperations/cookbooksmaster+1 -1
Gitlab: switchover between gitlab-replica-a and gitlab-replica-boperations/dnsmaster+8 -8
Gitlab: switchover between gitlab-replica-a and gitlab-replica-boperations/puppetproduction+2 -2
Customize query in gerrit
Related Changes in GitLab:
TitleReferenceAuthorSource BranchDest Branch
Preparing gitlab replica switchoverrepos/releng/gitlab-settings!74arnaudbswitchover/T400121main
Customize query in GitLab

Related Objects
Search...

Event Timeline

ABran-WMF created this task.Jul 22 2025, 7:33 AM
Restricted Application added a subscriber: Aklapper. · View Herald TranscriptJul 22 2025, 7:33 AM
ABran-WMF changed the task status from Open to In Progress.Jul 22 2025, 7:34 AM
ABran-WMF triaged this task as High priority.
ABran-WMF moved this task from Incoming to Work in Progress on the collaboration-services board.
ABran-WMF updated the task description. (Show Details)
CodeReviewBot added a project: Patch-For-Review.Jul 22 2025, 7:44 AM

arnaudb opened https://gitlab.wikimedia.org/repos/releng/gitlab-settings/-/merge_requests/74

Preparing gitlab replica switchover

gerritbot added a comment.Jul 22 2025, 7:45 AM

Change #1171537 had a related patch set uploaded (by Arnaudb; author: Arnaudb):

[operations/dns@master] Gitlab: switchover between gitlab-replica-a and gitlab-replica-b

https://gerrit.wikimedia.org/r/1171537

ABran-WMF updated the task description. (Show Details)Jul 22 2025, 7:48 AM
gerritbot added a comment.Jul 22 2025, 7:50 AM

Change #1171539 had a related patch set uploaded (by Arnaudb; author: Arnaudb):

[operations/puppet@production] Gitlab: switchover between gitlab-replica-a and gitlab-replica-b

https://gerrit.wikimedia.org/r/1171539

ABran-WMF updated the task description. (Show Details)Jul 22 2025, 7:51 AM
ABran-WMF updated the task description. (Show Details)
Jelto added a parent task: T399306: upgrade gitlab hosts to bookworm.Jul 22 2025, 9:42 AM
ABran-WMF mentioned this in T400208: SystemdUnitFailed - backup-restore.service - gitlab1004.Jul 23 2025, 5:59 AM
Jelto updated the task description. (Show Details)Jul 23 2025, 8:48 AM
CodeReviewBot added a comment.Jul 23 2025, 9:00 AM

arnaudb merged https://gitlab.wikimedia.org/repos/releng/gitlab-settings/-/merge_requests/74

Preparing gitlab replica switchover

ops-monitoring-bot added a comment.Jul 23 2025, 9:15 AM

Cookbook cookbooks.sre.gitlab.failover (Failover of gitlab from gitlab1004.wikimedia.org to gitlab1003.wikimedia.org) started

gerritbot added a comment.Jul 23 2025, 9:33 AM

Change #1171539 merged by Arnaudb:

[operations/puppet@production] Gitlab: switchover between gitlab-replica-a and gitlab-replica-b

https://gerrit.wikimedia.org/r/1171539

Jelto updated the task description. (Show Details)Jul 23 2025, 9:35 AM
gerritbot added a comment.Jul 23 2025, 10:21 AM

Change #1171537 merged by Arnaudb:

[operations/dns@master] Gitlab: switchover between gitlab-replica-a and gitlab-replica-b

https://gerrit.wikimedia.org/r/1171537

Jelto updated the task description. (Show Details)Jul 23 2025, 10:23 AM
ops-monitoring-bot added a comment.Jul 23 2025, 10:27 AM

Cookbook cookbooks.sre.gitlab.failover (Failover of gitlab from gitlab1004.wikimedia.org to gitlab1003.wikimedia.org) finished

Maintenance_bot removed a project: Patch-For-Review.Jul 23 2025, 10:31 AM
ABran-WMF closed this task as Resolved.Jul 23 2025, 10:31 AM
gerritbot added a comment.Jul 23 2025, 10:39 AM

Change #1172015 had a related patch set uploaded (by Jelto; author: Jelto):

[operations/cookbooks@master] gitlab failover: improve message for API token

https://gerrit.wikimedia.org/r/1172015

gerritbot added a project: Patch-For-Review.Jul 23 2025, 10:40 AM
gerritbot added a comment.Jul 23 2025, 11:59 AM

Change #1172015 merged by jenkins-bot:

[operations/cookbooks@master] gitlab failover: improve message for API token

https://gerrit.wikimedia.org/r/1172015

Maintenance_bot removed a project: Patch-For-Review.Jul 23 2025, 12:31 PM
Content licensed under Creative Commons Attribution-ShareAlike (CC BY-SA) 4.0 unless otherwise noted; code licensed under GNU General Public License (GPL) 2.0 or later and other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL · Credits