Change Details

**Run everything in a tmux named `switchover`** - [ ] `scap lock --all "Datacenter Switchover: Services & Traffic - T346330" on deploy1002 - [ ] `sudo cookbook sre.discovery.datacenter depool eqiad --all --reason "Datacenter Switchover: Services" --task-id T346330` on cumin1001 - [ ] deployment server: - [ ] log SAL: `!log Switch deployment server - T346330` - [ ] `sudo cumin 'R:class = role::deployment_server' 'disable-puppet "Switchover of the deployment server"'` - [ ] merge https://gerrit.wikimedia.org/r/c/operations/dns/+/957734 - [ ] merge https://gerrit.wikimedia.org/r/c/operations/puppet/+/957736 - [ ] Run puppet on deploy2002.codfw.wmnet: `sudo cumin deploy2002.codfw.wmnet 'run-puppet-agent --enable "Switchover of the deployment server"'` - [ ] Run puppet on all other deployment servers `sudo cumin 'R:class = role::deployment_server' 'run-puppet-agent --enable "Switchover of the deployment server"'` - [ ] Cronjob check `sudo cumin deploy2002.codfw.wmnet 'systemctl list-units | grep -A1 sync_deployment_dir'` TODO finish checklist when firefox stops being broken

**Run everything in a tmux named `switchover`** - [ ] `scap lock --all "Datacenter Switchover: Services & Traffic - T346330" on deploy1002 - [ ] `sudo cookbook sre.discovery.datacenter depool eqiad --all --reason "Datacenter Switchover: Services" --task-id T346330` on cumin1001 - [ ] deployment server: - [ ] log SAL: `!log Switch deployment server - T346330` - [ ] `sudo cumin 'R:class = role::deployment_server' 'disable-puppet "Switchover of the deployment server"'` - [ ] merge https://gerrit.wikimedia.org/r/c/operations/dns/+/957734 - [ ] merge https://gerrit.wikimedia.org/r/c/operations/puppet/+/957736 - [ ] Run puppet on deploy2002.codfw.wmnet: `sudo cumin deploy2002.codfw.wmnet 'run-puppet-agent --enable "Switchover of the deployment server"'` - [ ] Run puppet on all other deployment servers `sudo cumin 'R:class = role::deployment_server' 'run-puppet-agent --enable "Switchover of the deployment server"'` - [ ] Run puppet on alert* -- **TODO command** - [ ] Cronjob check `sudo cumin deploy2002.codfw.wmnet 'systemctl list-units | grep -A1 sync_deployment_dir'` - [ ] remove scap lock - [ ] Test scap deployment `cd /srv/mediawiki-staging; scap sync-world "check the deployment server after switchover"` - [ ] Test scap3 deployments work (restbase?) - **TODO do we really want to given the Thursday breakage?** - [ ] email ops@ about the switch (**TODO write the email**)

**Run everything in a tmux named `switchover`** - [ ] `scap lock --all "Datacenter Switchover: Services & Traffic - T346330" on deploy1002 - [ ] `sudo cookbook sre.discovery.datacenter depool eqiad --all --reason "Datacenter Switchover: Services" --task-id T346330` on cumin1001 - [ ] deployment server: - [ ] log SAL: `!log Switch deployment server - T346330` - [ ] `sudo cumin 'R:class = role::deployment_server' 'disable-puppet "Switchover of the deployment server"'` - [ ] merge https://gerrit.wikimedia.org/r/c/operations/dns/+/957734 - [ ] merge https://gerrit.wikimedia.org/r/c/operations/puppet/+/957736 - [ ] Run puppet on deploy2002.codfw.wmnet: `sudo cumin deploy2002.codfw.wmnet 'run-puppet-agent --enable "Switchover of the deployment server"'` - [ ] Run puppet on all other deployment servers `sudo cumin 'R:class = role::deployment_server' 'run-puppet-agent --enable "Switchover of the deployment server"'` - [ ] Run puppet on alert* -- **TODO command** - [ ] Cronjob check `sudo cumin deploy2002.codfw.wmnet 'systemctl list-units | grep -A1 sync_deployment_dir'` TODO finish checklist when firefox stops being broken - [ ] remove scap lock - [ ] Test scap deployment `cd /srv/mediawiki-staging; scap sync-world "check the deployment server after switchover"` - [ ] Test scap3 deployments work (restbase?) - **TODO do we really want to given the Thursday breakage?** - [ ] email ops@ about the switch (**TODO write the email**)