Page MenuHomePhabricator
Create Task
Maniphest T346330

Sept 2023 Switchover Checklist: Services & Traffic
Closed, ResolvedPublic
Actions

Description

Run everything in a tmux named switchover

Services

  • `scap lock --all "Datacenter Switchover: Services & Traffic - T346330" on deploy1002
  • sudo cookbook sre.discovery.datacenter depool eqiad --all --reason "Datacenter Switchover: Services" --task-id T346330 on cumin1001

Traffic

deployment server

  • log SAL: !log Switch deployment server - T346330
  • sudo cumin 'R:class = role::deployment_server' 'disable-puppet "Switchover of the deployment server"'
  • merge https://gerrit.wikimedia.org/r/c/operations/dns/+/957734
  • merge https://gerrit.wikimedia.org/r/c/operations/puppet/+/957736
  • Run puppet on deploy2002.codfw.wmnet: sudo cumin deploy2002.codfw.wmnet 'run-puppet-agent --enable "Switchover of the deployment server"'
  • Run puppet on all other deployment servers sudo cumin 'R:class = role::deployment_server' 'run-puppet-agent --enable "Switchover of the deployment server"'
  • Run puppet on alert* sudo cumin 'A:icinga' 'run-puppet-agent -q'
  • Cronjob check sudo cumin deploy2002.codfw.wmnet 'systemctl list-units | grep -A1 sync_deployment_dir' out of date docs, TODO fix
  • remove scap lock
  • Test scap deployment cd /srv/mediawiki-staging; scap sync-world "check the deployment server after switchover"
  • Test scap3 deployments work (restbase?) => nope, due to T346354
  • email ops@ and wikitech-l about the switch

Details

ProjectBranchLines +/-Subject
operations/puppetproduction+1 -1mw-on-k8s: Lower traffic to 3%
operations/dnsmaster+1 -0traffic: Depool eqiad from user traffic for switchover
operations/puppetproduction+2 -2Switch deployment server to deploy2002.codfw.wmnet
operations/dnsmaster+2 -2wmnet: switch deployment CNAMEs to codfw
Customize query in gerrit

Related Objects
Search...

Event Timeline

kamila created this task.Thu, Sep 14, 1:03 PM
Restricted Application added a subscriber: Aklapper. · View Herald TranscriptThu, Sep 14, 1:03 PM
Clement_Goubert added a subscriber: Clement_Goubert.Thu, Sep 14, 1:04 PM
gerritbot added a comment.Thu, Sep 14, 1:11 PM

Change 957734 had a related patch set uploaded (by Kamila Součková; author: Kamila Součková):

[operations/dns@master] wmnet: switch deployment CNAMEs to codfw

https://gerrit.wikimedia.org/r/957734

gerritbot added a project: Patch-For-Review.Thu, Sep 14, 1:11 PM
gerritbot added a comment.Thu, Sep 14, 1:19 PM

Change 957736 had a related patch set uploaded (by Kamila Součková; author: Kamila Součková):

[operations/puppet@production] Switch deployment server to deploy2002.codfw.wmnet

https://gerrit.wikimedia.org/r/957736

kamila updated the task description. (Show Details)Thu, Sep 14, 2:33 PM
kamila updated the task description. (Show Details)Thu, Sep 14, 2:47 PM
kamila updated the task description. (Show Details)Thu, Sep 14, 4:03 PM
kamila updated the task description. (Show Details)Thu, Sep 14, 4:39 PM
kamila triaged this task as High priority.Sat, Sep 16, 12:45 PM
Clement_Goubert moved this task from Incoming 🐫 to Doing 😎 on the serviceops board.Mon, Sep 18, 11:28 AM
kamila updated the task description. (Show Details)Tue, Sep 19, 11:42 AM
gerritbot added a comment.Tue, Sep 19, 12:16 PM

Change 958920 had a related patch set uploaded (by Kamila Součková; author: Kamila Součková):

[operations/dns@master] traffic: Depool eqiad from user traffic for switchover

https://gerrit.wikimedia.org/r/958920

kamila updated the task description. (Show Details)Tue, Sep 19, 12:20 PM
Stashbot added a comment.Tue, Sep 19, 2:00 PM

Mentioned in SAL (#wikimedia-operations) [2023-09-19T14:00:55Z] <kamila@deploy1002> Locking from deployment [ALL REPOSITORIES]: Datacenter Switchover: Services & Traffic - T346330

ops-monitoring-bot added a comment.Tue, Sep 19, 2:01 PM

kamila@cumin1001 - Cookbook cookbooks.sre.discovery.datacenter depool all services in eqiad: Datacenter Switchover: Services - T346330 started.

Stashbot added a comment.Tue, Sep 19, 2:01 PM

Mentioned in SAL (#wikimedia-operations) [2023-09-19T14:01:19Z] <kamila@cumin1001> START - Cookbook sre.discovery.datacenter depool all services in eqiad: Datacenter Switchover: Services - T346330

Stashbot added a comment.Tue, Sep 19, 2:20 PM

Mentioned in SAL (#wikimedia-operations) [2023-09-19T14:20:23Z] <kamila@deploy1002> Unlocked for deployment [ALL REPOSITORIES]: Datacenter Switchover: Services & Traffic - T346330 (duration: 19m 27s)

ops-monitoring-bot added a comment.Tue, Sep 19, 2:28 PM

kamila@cumin1001 - Cookbook cookbooks.sre.discovery.datacenter depool all services in eqiad: Datacenter Switchover: Services - T346330 completed.

Stashbot added a comment.Tue, Sep 19, 2:28 PM

Mentioned in SAL (#wikimedia-operations) [2023-09-19T14:28:47Z] <kamila@cumin1001> END (PASS) - Cookbook sre.discovery.datacenter (exit_code=0) depool all services in eqiad: Datacenter Switchover: Services - T346330

Stashbot added a comment.Tue, Sep 19, 2:30 PM

Mentioned in SAL (#wikimedia-operations) [2023-09-19T14:30:32Z] <kamila@deploy1002> Locking from deployment [ALL REPOSITORIES]: Datacenter Switchover: Services & Traffic - T346330

Stashbot added a comment.Tue, Sep 19, 2:32 PM

Mentioned in SAL (#wikimedia-operations) [2023-09-19T14:32:03Z] <kamila_> Switch deployment server - T346330

gerritbot added a comment.Tue, Sep 19, 2:33 PM

Change 957734 merged by Kamila Součková:

[operations/dns@master] wmnet: switch deployment CNAMEs to codfw

https://gerrit.wikimedia.org/r/957734

gerritbot added a comment.Tue, Sep 19, 2:34 PM

Change 957736 merged by Kamila Součková:

[operations/puppet@production] Switch deployment server to deploy2002.codfw.wmnet

https://gerrit.wikimedia.org/r/957736

kamila updated the task description. (Show Details)Tue, Sep 19, 2:38 PM
Stashbot added a comment.Tue, Sep 19, 3:05 PM

Mentioned in SAL (#wikimedia-operations) [2023-09-19T15:05:18Z] <kamila@deploy1002> Unlocked for deployment [ALL REPOSITORIES]: Datacenter Switchover: Services & Traffic - T346330 (duration: 34m 46s)

kamila updated the task description. (Show Details)Tue, Sep 19, 3:07 PM
gerritbot added a comment.Tue, Sep 19, 3:21 PM

Change 958955 had a related patch set uploaded (by Clément Goubert; author: Clément Goubert):

[operations/puppet@production] mw-on-k8s: Lower traffic to 3%

https://gerrit.wikimedia.org/r/958955

gerritbot added a comment.Tue, Sep 19, 3:24 PM

Change 958955 merged by Clément Goubert:

[operations/puppet@production] mw-on-k8s: Lower traffic to 3%

https://gerrit.wikimedia.org/r/958955

Stashbot added a comment.Tue, Sep 19, 3:25 PM

Mentioned in SAL (#wikimedia-operations) [2023-09-19T15:25:50Z] <claime> reduce mw-on-k8s traffic to 3% waiting on new nodes - T346330

Stashbot added a comment.Tue, Sep 19, 3:26 PM

Mentioned in SAL (#wikimedia-operations) [2023-09-19T15:26:28Z] <claime> running puppet on 'A:cp-text and P{P:trafficserver::backend}' - T346330

gerritbot added a comment.Tue, Sep 19, 4:00 PM

Change 958920 merged by Kamila Součková:

[operations/dns@master] traffic: Depool eqiad from user traffic for switchover

https://gerrit.wikimedia.org/r/958920

Stashbot added a comment.Tue, Sep 19, 4:04 PM

Mentioned in SAL (#wikimedia-operations) [2023-09-19T16:04:33Z] <kamila_> DC Switchover: traffic - T346330

Maintenance_bot removed a project: Patch-For-Review.Tue, Sep 19, 4:11 PM
kamila updated the task description. (Show Details)Tue, Sep 19, 4:28 PM
kamila closed this task as Resolved.Tue, Sep 19, 4:31 PM

While there are some outstanding issues due to lack of capacity in codfw, overall we're done here :-)

Content licensed under Creative Commons Attribution-ShareAlike 4.0 (CC-BY-SA) unless otherwise noted; code licensed under GNU General Public License (GPL) or other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL