Page MenuHomePhabricator
Create Task
Maniphest T346474

Sept 2023 Switchover Checklist: MediaWiki
Closed, ResolvedPublic
Actions

Description

Run everything in a tmux named switchover (Collaborative_tmux_sessions)

  • (Monday) Run sre.switchdc.mediawiki.03-set-db-readonly and sre.switchdc.mediawiki.06-set-db-readwrite in live-test mode
  • Update statuspage
  • scap lock --all "Datacenter Switchover: MediaWiki - T346474" on deploy2002
  • Run the cookbook: sudo cookbook sre.switchdc.mediawiki --ro-reason 'DC switchover (T346474)' eqiad codfw up to 09-restore-ttl.py
  • Merge https://gerrit.wikimedia.org/r/c/operations/dns/+/958462
  • Run authdns-update from any authdns node
  • SAL log !log Phase 9.5 Update DNS records for new database masters - T346474
  • finish running cookbook
  • Check reading & editing works
  • Remove scap lock
  • T346472
  • Check the banner is gone
  • Check https://en.wikipedia.org/wiki/Special:RecentChanges and EventStreams curl -s -H 'Accept: application/json' https://stream.wikimedia.org/v2/stream/recentchange | jq .
  • Run exim4 -bp | exiqsumm on mx1001/mx2001; test an email
  • Update statuspage

Details

Related Changes in Gerrit:
SubjectAuthorRepoBranchLines +/-
geo-maps: reorder codfw/eqiad in the defaultKamila Součkováoperations/dnsmaster+1 -1
wmnet: Update maintenance.eqiad.wmnet to point to mwmaint2002Kamila Součkováoperations/dnsmaster+3 -3
wmnet: Update pc cnames to codfwMarosteguioperations/dnsmaster+3 -3
wmnet: Update pc cnames to codfwMarosteguioperations/dnsmaster+3 -3
db: Switch DNS master alias to codfwKamila Součkováoperations/dnsmaster+15 -15
Customize query in gerrit

Related Objects
Search...

StatusSubtypeAssignedTask
 ResolvedRaineT345263 September 2023 Datacenter Switchover
 ResolvedRaineT346474 Sept 2023 Switchover Checklist: MediaWiki

Event Timeline

Raine created this task.Sep 16 2023, 12:47 PM
Restricted Application added a subscriber: Aklapper. · View Herald TranscriptSep 16 2023, 12:47 PM
Raine triaged this task as High priority.Sep 16 2023, 1:20 PM
Raine updated the task description. (Show Details)
Volans updated the task description. (Show Details)Sep 16 2023, 7:58 PM
Clement_Goubert moved this task from Incoming 🐫 to Doing 😎 on the serviceops-deprecated board.Sep 18 2023, 11:28 AM
gerritbot added a comment.Sep 18 2023, 11:58 AM

Change 958462 had a related patch set uploaded (by Kamila Součková; author: Kamila Součková):

[operations/dns@master] db: Switch DNS master alias to codfw

https://gerrit.wikimedia.org/r/958462

gerritbot added a project: Patch-For-Review.Sep 18 2023, 11:58 AM
gerritbot added a comment.Sep 18 2023, 12:03 PM

Change 958464 had a related patch set uploaded (by Marostegui; author: Marostegui):

[operations/dns@master] wmnet: Update pc cnames to codfw

https://gerrit.wikimedia.org/r/958464

gerritbot added a comment.Sep 18 2023, 12:37 PM

Change 958472 had a related patch set uploaded (by Kamila Součková; author: Kamila Součková):

[operations/dns@master] wmnet: Update maintenance.eqiad.wmnet to point to mwmaint2002

https://gerrit.wikimedia.org/r/958472

Raine updated the task description. (Show Details)Sep 18 2023, 12:48 PM
Raine updated the task description. (Show Details)Sep 19 2023, 12:21 PM
gerritbot added a comment.Sep 20 2023, 10:45 AM

Change 959182 had a related patch set uploaded (by Kamila Součková; author: Kamila Součková):

[operations/dns@master] geo-maps: reorder codfw/eqiad in the default

https://gerrit.wikimedia.org/r/959182

Raine updated the task description. (Show Details)Sep 20 2023, 10:46 AM
Raine updated the task description. (Show Details)Sep 20 2023, 1:31 PM
Stashbot added a comment.Sep 20 2023, 1:56 PM

Mentioned in SAL (#wikimedia-operations) [2023-09-20T13:56:36Z] <kamila@deploy2002> Locking from deployment [ALL REPOSITORIES]: Datacenter Switchover: MediaWiki - T346474

gerritbot added a comment.Sep 20 2023, 2:07 PM

Change 958462 merged by Kamila Součková:

[operations/dns@master] db: Switch DNS master alias to codfw

https://gerrit.wikimedia.org/r/958462

Stashbot added a comment.Sep 20 2023, 2:07 PM

Mentioned in SAL (#wikimedia-operations) [2023-09-20T14:07:42Z] <kamila_> Phase 9.5 Update DNS records for new database masters - T346474

Stashbot added a comment.Sep 20 2023, 2:09 PM

Mentioned in SAL (#wikimedia-operations) [2023-09-20T14:09:31Z] <kamila@deploy2002> Unlocked for deployment [ALL REPOSITORIES]: Datacenter Switchover: MediaWiki - T346474 (duration: 12m 54s)

gerritbot added a comment.Sep 20 2023, 2:09 PM

Change 959243 had a related patch set uploaded (by Marostegui; author: Marostegui):

[operations/dns@master] wmnet: Update pc cnames to codfw

https://gerrit.wikimedia.org/r/959243

gerritbot added a comment.Sep 20 2023, 2:10 PM

Change 958464 abandoned by Marostegui:

[operations/dns@master] wmnet: Update pc cnames to codfw

Reason:

https://gerrit.wikimedia.org/r/958464

gerritbot added a comment.Sep 20 2023, 2:13 PM

Change 959243 merged by Marostegui:

[operations/dns@master] wmnet: Update pc cnames to codfw

https://gerrit.wikimedia.org/r/959243

Raine updated the task description. (Show Details)Sep 20 2023, 2:15 PM
gerritbot added a comment.Sep 20 2023, 2:34 PM

Change 958472 merged by Kamila Součková:

[operations/dns@master] wmnet: Update maintenance.eqiad.wmnet to point to mwmaint2002

https://gerrit.wikimedia.org/r/958472

Raine updated the task description. (Show Details)Sep 20 2023, 2:40 PM
gerritbot added a comment.Sep 25 2023, 12:50 PM

Change 959182 merged by Kamila Součková:

[operations/dns@master] geo-maps: reorder codfw/eqiad in the default

https://gerrit.wikimedia.org/r/959182

Maintenance_bot removed a project: Patch-For-Review.Sep 25 2023, 1:11 PM
Raine closed this task as Resolved.Sep 25 2023, 1:53 PM
Raine updated the task description. (Show Details)
Content licensed under Creative Commons Attribution-ShareAlike (CC BY-SA) 4.0 unless otherwise noted; code licensed under GNU General Public License (GPL) 2.0 or later and other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL · Credits