Page MenuHomePhabricator

Refactor the switchdc script
Closed, ResolvedPublic

Description

As part of the parent task goal of performing a datacenter switchover, the switchdc scripts must be refactored into a more re-usable automation library and updated to the newer switchover requirements, taking into account also the results of the breakout session at the SRE offsite.

See also Switch_Datacenter on Wikitech for the list of steps to perform (to be updated based on https://etherpad.wikimedia.org/p/switchover-2018-kickoff )

Details

ProjectBranchLines +/-Subject
operations/cookbooksmaster+1 -74
operations/software/spicerackmaster+190 -135
operations/cookbooksmaster+3 -8
operations/software/spicerackmaster+168 -56
operations/software/spicerackmaster+48 -6
operations/software/spicerackmaster+29 -17
operations/software/spicerackmaster+6 -1
operations/software/spicerackmaster+35 -7
operations/software/spicerackmaster+1 -1
operations/software/spicerackmaster+9 -0
operations/cookbooksmaster+2 -1
operations/cookbooksmaster+2 -1
operations/cookbooksmaster+3 -0
operations/software/spicerackdebian+6 -0
operations/software/spicerackmaster+19 -14
operations/software/spicerackmaster+7 -3
operations/cookbooksmaster+0 -3
operations/cookbooksmaster+4 -0
operations/software/spicerackdebian+6 -0
operations/software/spicerackmaster+20 -2
operations/software/spicerackdebian+6 -0
operations/software/spicerackmaster+1 -1
operations/cookbooksmaster+7 -7
operations/software/spicerackmaster+131 -21
operations/puppetproduction+4 -4
integration/configmaster+1 -1
operations/cookbooksmaster+4 -0
operations/software/spicerackdebian+6 -0
operations/software/spicerackmaster+55 -26
operations/cookbooksmaster+2 -1
operations/cookbooksmaster+33 -12
operations/cookbooksmaster+1 -1
operations/cookbooksmaster+3 -2
operations/cookbooksmaster+1 -1
operations/cookbooksmaster+0 -0
operations/software/spicerackdebian+7 -0
operations/cookbooksmaster+16 -0
operations/cookbooksmaster+16 -0
operations/cookbooksmaster+21 -0
operations/cookbooksmaster+31 -0
operations/cookbooksmaster+22 -0
operations/puppetproduction+23 -2
operations/cookbooksmaster+3 -0
operations/cookbooksmaster+20 -0
operations/software/spicerackmaster+13 -1
operations/cookbooksmaster+95 -1
operations/software/spicerackmaster+317 -1
operations/cookbooksmaster+32 -0
operations/cookbooksmaster+0 -0
operations/cookbooksmaster+4 -5
operations/cookbooksmaster+684 -0
operations/puppetproduction+2 -0
operations/cookbooksmaster+86 -0
operations/cookbooksmaster+20 -0
operations/cookbooksmaster+17 -0
operations/cookbooksmaster+22 -0
operations/cookbooksmaster+17 -0
operations/cookbooksmaster+16 -0
operations/cookbooksmaster+80 -0
operations/software/spicerackmaster+18 -1
operations/software/spicerackmaster+14 -2
operations/software/spicerackmaster+41 -5
operations/software/spicerackmaster+20 -47
operations/software/spicerackmaster+41 -4
operations/software/spicerackmaster+51 -18
operations/software/spicerackmaster+9 -9
operations/cookbooksmaster+41 -0
operations/cookbooksmaster+21 -0
operations/cookbooksmaster+142 -3
integration/configmaster+4 -0
operations/software/spicerackmaster+2 -1
operations/software/spicerackmaster+1 -1
operations/software/spicerackmaster+108 -3
operations/puppetproduction+1 -1
operations/puppetproduction+1 -1
operations/puppetproduction+51 -125
operations/software/spicerackdebian+86 -0
operations/software/spicerackmaster+86 -0
operations/software/spicerackmaster+2 -1
operations/software/spicerackmaster+5 -1
operations/software/spicerackmaster+2 -2
operations/software/spicerackmaster+48 -1
operations/software/spicerackmaster+1 -1
operations/software/spicerackmaster+32 -68
operations/software/spicerackmaster+148 -81
operations/software/spicerackmaster+433 -9
operations/cookbooksmaster+1 -1
operations/software/spicerackmaster+26 -7
operations/software/spicerackmaster+3 -3
operations/cookbooksmaster+5 -0
operations/software/spicerackmaster+16 -5
operations/software/spicerackmaster+97 -19
operations/software/spicerackmaster+36 -8
operations/software/spicerackmaster+23 -5
operations/software/spicerackmaster+25 -3
operations/software/spicerackmaster+44 -27
operations/software/spicerackmaster+10 -8
operations/software/spicerackmaster+15 -0
operations/software/spicerackmaster+26 -23
operations/software/spicerackmaster+242 -0
operations/software/spicerackmaster+36 -32
operations/software/spicerackmaster+238 -0
operations/software/spicerackmaster+256 -2
operations/software/spicerackmaster+304 -176
operations/software/spicerackmaster+11 -12
operations/software/spicerackmaster+6 -7
operations/software/spicerackmaster+270 -0
operations/software/spicerackmaster+229 -0
operations/software/spicerackmaster+10 -6
operations/software/spicerackmaster+965 -6
operations/software/spicerackmaster+12 -16
operations/software/spicerackmaster+0 -46
operations/software/spicerackmaster+39 -30
operations/software/spicerackmaster+6 -9
operations/software/spicerackmaster+10 -12
operations/software/spicerackmaster+5 -5
operations/software/spicerackmaster+585 -3
operations/software/spicerackmaster+863 -0
integration/configmaster+4 -0
Show related patches Customize query in gerrit

Related Objects

Event Timeline

There are a very large number of changes, so older changes are hidden. Show Older Changes

Change 457711 merged by jenkins-bot:
[operations/software/spicerack@master] Add redis_cluster module

https://gerrit.wikimedia.org/r/457711

Change 456588 merged by jenkins-bot:
[operations/cookbooks@master] sre.switchdc.mediawiki: add Phase 4 cookbooks

https://gerrit.wikimedia.org/r/456588

Change 457367 merged by jenkins-bot:
[operations/software/spicerack@master] mediawiki: improve stop_cronjobs() method

https://gerrit.wikimedia.org/r/457367

Change 457859 merged by jenkins-bot:
[operations/cookbooks@master] sre.switchdc.mediawiki: add Phase 5 cookbook

https://gerrit.wikimedia.org/r/457859

Change 457936 had a related patch set uploaded (by Volans; owner: Volans):
[operations/cookbooks@master] sre.switchdc.mediawiki: wait TTL expiration

https://gerrit.wikimedia.org/r/457936

Change 457943 had a related patch set uploaded (by Volans; owner: Volans):
[operations/cookbooks@master] sre.switchdc.mediawiki: add validation of CLI args

https://gerrit.wikimedia.org/r/457943

Change 457944 had a related patch set uploaded (by Volans; owner: Volans):
[operations/cookbooks@master] sre.switchdc.mediawiki: add --live-test option

https://gerrit.wikimedia.org/r/457944

Change 457873 had a related patch set uploaded (by Giuseppe Lavagetto; owner: Giuseppe Lavagetto):
[operations/cookbooks@master] sre.switchdc.services: Add init for the recipe

https://gerrit.wikimedia.org/r/457873

Change 457874 had a related patch set uploaded (by Giuseppe Lavagetto; owner: Giuseppe Lavagetto):
[operations/cookbooks@master] sre.switchdc.services: Add phase 0

https://gerrit.wikimedia.org/r/457874

Change 457875 had a related patch set uploaded (by Giuseppe Lavagetto; owner: Giuseppe Lavagetto):
[operations/cookbooks@master] sre.switchdc.services: add phase 1

https://gerrit.wikimedia.org/r/457875

Change 457876 had a related patch set uploaded (by Giuseppe Lavagetto; owner: Giuseppe Lavagetto):
[operations/cookbooks@master] sre.switchdc.services: add phase 2

https://gerrit.wikimedia.org/r/457876

Change 457943 merged by jenkins-bot:
[operations/cookbooks@master] sre.switchdc.mediawiki: add validation of CLI args

https://gerrit.wikimedia.org/r/457943

Change 458115 had a related patch set uploaded (by Volans; owner: Volans):
[operations/software/spicerack@master] cookbook: split main() into setup() and run()

https://gerrit.wikimedia.org/r/458115

Change 457836 merged by Volans:
[operations/puppet@production] spicerack: add redis sessions configuration

https://gerrit.wikimedia.org/r/457836

Change 456510 merged by jenkins-bot:
[operations/cookbooks@master] sre.switchdc.mediawiki: add Phase 3 cookbook

https://gerrit.wikimedia.org/r/456510

Change 457873 merged by jenkins-bot:
[operations/cookbooks@master] sre.switchdc.services: Add init for the recipe

https://gerrit.wikimedia.org/r/457873

Change 457874 merged by jenkins-bot:
[operations/cookbooks@master] sre.switchdc.services: Add phase 0

https://gerrit.wikimedia.org/r/457874

Change 457875 merged by jenkins-bot:
[operations/cookbooks@master] sre.switchdc.services: add phase 1

https://gerrit.wikimedia.org/r/457875

Change 457876 merged by jenkins-bot:
[operations/cookbooks@master] sre.switchdc.services: add phase 2

https://gerrit.wikimedia.org/r/457876

Change 458123 had a related patch set uploaded (by Volans; owner: Volans):
[operations/software/spicerack@debian] Upstream release v0.0.4

https://gerrit.wikimedia.org/r/458123

Change 458123 merged by jenkins-bot:
[operations/software/spicerack@debian] Upstream release v0.0.4

https://gerrit.wikimedia.org/r/458123

Mentioned in SAL (#wikimedia-operations) [2018-09-05T09:16:13Z] <volans> uploaded spicerack_0.0.4-1{,+deb9u1} to apt.wikimedia.org {jessie,stretch}-wikimedia - T199079

Change 458131 had a related patch set uploaded (by Volans; owner: Volans):
[operations/cookbooks@master] sre.switchdc.services: rename cookbooks

https://gerrit.wikimedia.org/r/458131

Change 458131 merged by jenkins-bot:
[operations/cookbooks@master] sre.switchdc.services: rename cookbooks

https://gerrit.wikimedia.org/r/458131

Change 457944 merged by jenkins-bot:
[operations/cookbooks@master] sre.switchdc.mediawiki: add --live-test option

https://gerrit.wikimedia.org/r/457944

Mentioned in SAL (#wikimedia-operations) [2018-09-05T10:00:03Z] <volans> upgraded spicerack to version 0.0.4 on sarin/neodymium - T199079

Change 458150 had a related patch set uploaded (by Volans; owner: Volans):
[operations/cookbooks@master] sre.switchdc.mediawiki: fix typo in parse args

https://gerrit.wikimedia.org/r/458150

Change 458150 merged by jenkins-bot:
[operations/cookbooks@master] sre.switchdc.mediawiki: fix typo in parse args

https://gerrit.wikimedia.org/r/458150

Change 458155 had a related patch set uploaded (by Volans; owner: Volans):
[operations/cookbooks@master] sre.switchdc.mediawiki: fix parse args validation

https://gerrit.wikimedia.org/r/458155

Change 458155 merged by jenkins-bot:
[operations/cookbooks@master] sre.switchdc.mediawiki: fix parse args validation

https://gerrit.wikimedia.org/r/458155

Change 458161 had a related patch set uploaded (by Volans; owner: Volans):
[operations/cookbooks@master] sre.switchdc.mediawiki: phase 0 fix typo

https://gerrit.wikimedia.org/r/458161

Change 458161 merged by jenkins-bot:
[operations/cookbooks@master] sre.switchdc.mediawiki: phase 0 fix typo

https://gerrit.wikimedia.org/r/458161

Change 458218 had a related patch set uploaded (by Volans; owner: Volans):
[operations/software/spicerack@master] mediawiki: improve validation checks

https://gerrit.wikimedia.org/r/458218

Change 458219 had a related patch set uploaded (by Volans; owner: Volans):
[operations/cookbooks@master] sre.switchdc.mediawiki: phase 0 improve logging

https://gerrit.wikimedia.org/r/458219

Change 458221 had a related patch set uploaded (by Volans; owner: Volans):
[operations/cookbooks@master] sre.switchdc.mediawiki: phase 2 add sleep

https://gerrit.wikimedia.org/r/458221

Change 458219 merged by jenkins-bot:
[operations/cookbooks@master] sre.switchdc.mediawiki: phase 0 improve logging

https://gerrit.wikimedia.org/r/458219

Change 458218 merged by jenkins-bot:
[operations/software/spicerack@master] mediawiki: improve validation checks

https://gerrit.wikimedia.org/r/458218

Change 458302 had a related patch set uploaded (by Volans; owner: Volans):
[operations/software/spicerack@debian] Upstream release v0.0.5

https://gerrit.wikimedia.org/r/458302

Change 458303 had a related patch set uploaded (by Volans; owner: Volans):
[integration/config@master] spicerack: use the backport version of debian-glue

https://gerrit.wikimedia.org/r/458303

Change 458302 merged by jenkins-bot:
[operations/software/spicerack@debian] Upstream release v0.0.5

https://gerrit.wikimedia.org/r/458302

Mentioned in SAL (#wikimedia-operations) [2018-09-05T21:54:04Z] <volans> uploaded spicerack_0.0.5-1{,+deb9u1} to apt.wikimedia.org {jessie,stretch}-wikimedia - T199079

Change 458221 merged by jenkins-bot:
[operations/cookbooks@master] sre.switchdc.mediawiki: phase 2 add sleep

https://gerrit.wikimedia.org/r/458221

Change 458303 merged by jenkins-bot:
[integration/config@master] spicerack: use the backport version of debian-glue

https://gerrit.wikimedia.org/r/458303

Change 458470 had a related patch set uploaded (by Volans; owner: Volans):
[operations/software/spicerack@master] WIP mysql: refactor sync check to avoid GTID

https://gerrit.wikimedia.org/r/458470

Change 457850 merged by Ema:
[operations/puppet@production] cache_canary: switch mediawiki to codfw

https://gerrit.wikimedia.org/r/457850

Change 458520 had a related patch set uploaded (by Volans; owner: Volans):
[operations/software/spicerack@master] log: remove relic from switchdc

https://gerrit.wikimedia.org/r/458520

Change 458470 merged by jenkins-bot:
[operations/software/spicerack@master] mysql: refactor sync check to avoid GTID

https://gerrit.wikimedia.org/r/458470

Change 458511 had a related patch set uploaded (by Volans; owner: Mark Bergsma):
[operations/cookbooks@master] sre.switchdc.mediawiki: fix typo s/mainteance/maintenance/

https://gerrit.wikimedia.org/r/458511

Change 458511 merged by jenkins-bot:
[operations/cookbooks@master] sre.switchdc.mediawiki: fix typo s/mainteance/maintenance/

https://gerrit.wikimedia.org/r/458511

Change 458520 merged by jenkins-bot:
[operations/software/spicerack@master] log: remove relic from switchdc

https://gerrit.wikimedia.org/r/458520

Change 458531 had a related patch set uploaded (by Volans; owner: Volans):
[operations/software/spicerack@debian] Upstream release v0.0.6

https://gerrit.wikimedia.org/r/458531

Change 458531 merged by jenkins-bot:
[operations/software/spicerack@debian] Upstream release v0.0.6

https://gerrit.wikimedia.org/r/458531

Mentioned in SAL (#wikimedia-operations) [2018-09-06T16:38:36Z] <volans> uploaded spicerack_0.0.6-1{,+deb9u1} to apt.wikimedia.org {jessie,stretch}-wikimedia - T199079

Mentioned in SAL (#wikimedia-operations) [2018-09-06T16:40:00Z] <volans> upgraded spicerack to version 0.0.6 on sarin/neodymium - T199079

Change 458539 had a related patch set uploaded (by Volans; owner: Volans):
[operations/software/spicerack@master] dnsdisc: fix dry-run in check_if_depoolable

https://gerrit.wikimedia.org/r/458539

Change 458539 merged by jenkins-bot:
[operations/software/spicerack@master] dnsdisc: fix dry-run in check_if_depoolable

https://gerrit.wikimedia.org/r/458539

Change 458541 had a related patch set uploaded (by Volans; owner: Volans):
[operations/software/spicerack@debian] Upstream release v0.0.7

https://gerrit.wikimedia.org/r/458541

Change 458541 merged by jenkins-bot:
[operations/software/spicerack@debian] Upstream release v0.0.7

https://gerrit.wikimedia.org/r/458541

Mentioned in SAL (#wikimedia-operations) [2018-09-06T17:34:10Z] <volans> uploaded spicerack_0.0.7-1{,+deb9u1} to apt.wikimedia.org {jessie,stretch}-wikimedia - T199079

Change 459511 had a related patch set uploaded (by Volans; owner: Volans):
[operations/cookbooks@master] sre.switchdc.mediawiki: ask for confirmation

https://gerrit.wikimedia.org/r/459511

Change 459511 merged by jenkins-bot:
[operations/cookbooks@master] sre.switchdc.mediawiki: ask for confirmation

https://gerrit.wikimedia.org/r/459511

Change 459517 had a related patch set uploaded (by Volans; owner: Volans):
[operations/cookbooks@master] sre.switchdc.mediawiki: remove memcache wipe

https://gerrit.wikimedia.org/r/459517

Change 459517 abandoned by Volans:
sre.switchdc.mediawiki: remove memcache wipe

Reason:
Already included in I528bb689a3a94588c15e68a2faef952d91d89c7d

https://gerrit.wikimedia.org/r/459517

Change 459595 had a related patch set uploaded (by Volans; owner: Volans):
[operations/software/spicerack@master] mediawiki: ignore exit codes on stop_cronjobs

https://gerrit.wikimedia.org/r/459595

Change 459606 had a related patch set uploaded (by Volans; owner: Volans):
[operations/software/spicerack@master] logging: minor improvements and a fix

https://gerrit.wikimedia.org/r/459606

Change 459607 had a related patch set uploaded (by Volans; owner: Volans):
[operations/cookbooks@master] sre.switchdc.mediawiki: parsoid skip broken host

https://gerrit.wikimedia.org/r/459607

Change 459607 merged by jenkins-bot:
[operations/cookbooks@master] sre.switchdc.mediawiki: parsoid skip broken host

https://gerrit.wikimedia.org/r/459607

Change 459606 merged by jenkins-bot:
[operations/software/spicerack@master] logging: minor improvements and a fix

https://gerrit.wikimedia.org/r/459606

Change 459595 merged by jenkins-bot:
[operations/software/spicerack@master] mediawiki: ignore exit codes on stop_cronjobs

https://gerrit.wikimedia.org/r/459595

Change 459771 had a related patch set uploaded (by Volans; owner: Volans):
[operations/software/spicerack@debian] Upstream release v0.0.8

https://gerrit.wikimedia.org/r/459771

Change 459771 merged by jenkins-bot:
[operations/software/spicerack@debian] Upstream release v0.0.8

https://gerrit.wikimedia.org/r/459771

Mentioned in SAL (#wikimedia-operations) [2018-09-11T11:49:35Z] <volans> uploaded spicerack_0.0.8-1{,+deb9u1} to apt.wikimedia.org {jessie,stretch}-wikimedia - T199079

Mentioned in SAL (#wikimedia-operations) [2018-09-11T13:02:09Z] <volans> upgraded spicerack to version 0.0.8 on sarin/neodymium - T199079

Change 459791 had a related patch set uploaded (by Volans; owner: Volans):
[operations/software/spicerack@master] dnsdisc: fine tune update_ttl and check_ttl

https://gerrit.wikimedia.org/r/459791

Change 459804 had a related patch set uploaded (by Volans; owner: Volans):
[operations/software/spicerack@master] exceptions: add SpicerackCheckError

https://gerrit.wikimedia.org/r/459804

Change 459805 had a related patch set uploaded (by Volans; owner: Volans):
[operations/software/spicerack@master] dnsdisc: catch dnspython exceptions

https://gerrit.wikimedia.org/r/459805

Change 457936 abandoned by Volans:
sre.switchdc.mediawiki: wait TTL expiration

Reason:
I've add a note in the wikipage, no need for it anymore

https://gerrit.wikimedia.org/r/457936

Change 460011 had a related patch set uploaded (by Volans; owner: Volans):
[operations/cookbooks@master] sre.switchdc.mediawiki: update read only reason

https://gerrit.wikimedia.org/r/460011

Change 460011 merged by jenkins-bot:
[operations/cookbooks@master] sre.switchdc.mediawiki: update read only reason

https://gerrit.wikimedia.org/r/460011

Change 460114 had a related patch set uploaded (by Volans; owner: Volans):
[operations/software/spicerack@master] mediawiki: improve siteinfo checks

https://gerrit.wikimedia.org/r/460114

Change 459804 merged by jenkins-bot:
[operations/software/spicerack@master] exceptions: add SpicerackCheckError

https://gerrit.wikimedia.org/r/459804

Change 460348 had a related patch set uploaded (by Volans; owner: Volans):
[operations/software/spicerack@master] setup.py: fix missing comma

https://gerrit.wikimedia.org/r/460348

Change 460348 merged by jenkins-bot:
[operations/software/spicerack@master] setup.py: fix missing comma

https://gerrit.wikimedia.org/r/460348

Change 460425 had a related patch set uploaded (by Volans; owner: Volans):
[operations/software/spicerack@master] setup.py: add missing fields

https://gerrit.wikimedia.org/r/460425

Change 460426 had a related patch set uploaded (by Volans; owner: Volans):
[operations/software/spicerack@master] tests: improve prospector tests

https://gerrit.wikimedia.org/r/460426

Change 460425 merged by jenkins-bot:
[operations/software/spicerack@master] setup.py: add missing fields

https://gerrit.wikimedia.org/r/460425

Change 460426 merged by jenkins-bot:
[operations/software/spicerack@master] tests: improve prospector tests

https://gerrit.wikimedia.org/r/460426

Change 459791 merged by jenkins-bot:
[operations/software/spicerack@master] dnsdisc: improve TTL checks

https://gerrit.wikimedia.org/r/459791

Change 459805 merged by jenkins-bot:
[operations/software/spicerack@master] dnsdisc: catch dnspython exceptions

https://gerrit.wikimedia.org/r/459805

Change 460114 merged by jenkins-bot:
[operations/software/spicerack@master] mediawiki: improve siteinfo checks

https://gerrit.wikimedia.org/r/460114

Change 463747 had a related patch set uploaded (by Volans; owner: Volans):
[operations/cookbooks@master] sre.switchdc.mediawiki: remove HHVM restart

https://gerrit.wikimedia.org/r/463747

Change 463747 merged by jenkins-bot:
[operations/cookbooks@master] sre.switchdc.mediawiki: remove HHVM restart

https://gerrit.wikimedia.org/r/463747

Mentioned in SAL (#wikimedia-operations) [2018-10-05T10:37:54Z] <volans> uploaded spicerack_0.0.9-1{,+deb9u1} to apt.wikimedia.org {jessie,stretch}-wikimedia - T199079

Mentioned in SAL (#wikimedia-operations) [2018-10-05T13:02:30Z] <volans> upgraded spicerack to version 0.0.9 on sarin/neodymium/cumin* - T199079

Volans closed this task as Resolved.Oct 5 2018, 1:03 PM

I'm closing this as resolved as the main goal was reached and we've already done the switch.

Volans moved this task from In Progress to In Code Review on the SRE-tools board.Oct 5 2018, 1:04 PM

Change 477565 had a related patch set uploaded (by Volans; owner: Volans):
[operations/cookbooks@master] README: update API documentation

https://gerrit.wikimedia.org/r/477565

Change 458115 merged by jenkins-bot:
[operations/software/spicerack@master] cookbook: split main into argument_parser and run

https://gerrit.wikimedia.org/r/458115

Change 477565 merged by jenkins-bot:
[operations/cookbooks@master] README: move API documentation

https://gerrit.wikimedia.org/r/477565