db1066 is on a6, which will be involved in the PDU maintenance T227142: a6-eqiad pdu refresh (Tuesday 10/22 @11am UTC)
We need to failover db1066 to db1122 which is on D6.
db1066 is also an old master that will be decommissioned T217396: Decommission db1061-db1073
Date&Time: 17th September at 05:00 UTC
read-only window will be required.
| Status | Subtype | Assigned | Task | ||
|---|---|---|---|---|---|
| Resolved | • Cmjohnson | T226778 Install new PDUs in rows A/B (Top level tracking task) | |||
| Resolved | None | T227142 a6-eqiad pdu refresh (Tuesday 10/22 @11am UTC) | |||
| Resolved | • Marostegui | T217396 Decommission db1061-db1073 | |||
| Resolved | • Marostegui | T230785 Switchover s2 primary database master db1066 -> db1122 - 17th Sept @05:00 UTC | |||
| Resolved | Trizek-WMF | T230788 Community Relations support needed for several read-only windows (s2, s3, s4 and s8) |
Change 531335 had a related patch set uploaded (by Marostegui; owner: Marostegui):
[operations/puppet@production] db1122: Change binlog format to STATEMENT
Change 531335 merged by Marostegui:
[operations/puppet@production] db1122: Change binlog format to STATEMENT
Change 531336 had a related patch set uploaded (by Marostegui; owner: Marostegui):
[operations/mediawiki-config@master] db-eqiad.php: Clarify that db1122 is the candidate master
Change 531336 merged by jenkins-bot:
[operations/mediawiki-config@master] db-eqiad.php: Clarify that db1122 is the candidate master
Mentioned in SAL (#wikimedia-operations) [2019-08-21T05:04:31Z] <marostegui@deploy1001> Synchronized wmf-config/db-eqiad.php: Clarify db1122 status: candidate master for s2 - T230785 (duration: 00m 55s)
Mentioned in SAL (#wikimedia-operations) [2019-08-21T05:05:46Z] <marostegui> Restart MySQL on db1122 for binlog format change - T230785
Mentioned in SAL (#wikimedia-operations) [2019-08-30T06:07:03Z] <marostegui@cumin1001> dbctl commit (dc=all): 'Depool db1076 for upgrade - T230785', diff saved to https://phabricator.wikimedia.org/P9006 and previous config saved to /var/cache/conftool/dbconfig/20190830-060702-marostegui.json
Mentioned in SAL (#wikimedia-operations) [2019-09-11T07:06:36Z] <marostegui@cumin1001> dbctl commit (dc=all): 'Depool db1122 to reboot for kernel upgrade T230785', diff saved to https://phabricator.wikimedia.org/P9083 and previous config saved to /var/cache/conftool/dbconfig/20190911-070635-marostegui.json
Mentioned in SAL (#wikimedia-operations) [2019-09-11T07:07:18Z] <marostegui> Stop MySQL on db1122 to reboot for a kernel upgrade T230785
Change 535839 had a related patch set uploaded (by Marostegui; owner: Marostegui):
[operations/puppet@production] mariadb: Promote db1122 as s2 primary master
Change 535842 had a related patch set uploaded (by Marostegui; owner: Marostegui):
[operations/dns@master] wmnet: Change s2 CNAME to db1122
Reserved window on the Deployments page: https://wikitech.wikimedia.org/w/index.php?title=Deployments&type=revision&diff=1837698&oldid=1837689
Mentioned in SAL (#wikimedia-operations) [2019-09-17T04:11:41Z] <marostegui> Start s2 pre-switchover steps T230785
Mentioned in SAL (#wikimedia-operations) [2019-09-17T04:14:42Z] <marostegui@cumin1001> dbctl commit (dc=all): 'Set db1122 with weight 0 and depool it from API T230785', diff saved to https://phabricator.wikimedia.org/P9111 and previous config saved to /var/cache/conftool/dbconfig/20190917-041441-marostegui.json
Change 535839 merged by Marostegui:
[operations/puppet@production] mariadb: Promote db1122 as s2 primary master
Mentioned in SAL (#wikimedia-operations) [2019-09-17T05:00:14Z] <marostegui> Starting s2 failover from db1066 to db1122 - T230785
Mentioned in SAL (#wikimedia-operations) [2019-09-17T05:00:44Z] <marostegui@cumin1001> dbctl commit (dc=all): 'Set s2 as read-only for maintenance T230785', diff saved to https://phabricator.wikimedia.org/P9112 and previous config saved to /var/cache/conftool/dbconfig/20190917-050043-marostegui.json
Mentioned in SAL (#wikimedia-operations) [2019-09-17T05:01:34Z] <marostegui@cumin1001> dbctl commit (dc=all): 'Promote db1122 to s2 master and remove read-only from s2 T230785', diff saved to https://phabricator.wikimedia.org/P9113 and previous config saved to /var/cache/conftool/dbconfig/20190917-050133-marostegui.json
Change 535842 merged by Marostegui:
[operations/dns@master] wmnet: Change s2 CNAME to db1122
This was done
read-only start: 05:00:44
read-only stop: 05:01:34
Total read-only time: 50 seconds.