Page MenuHomePhabricator

Switchover s4 (commonswiki) primary database master db1081 -> db1138 - 26th Sept @05:00 UTC
Closed, ResolvedPublic

Description

db1081 is on A2, which will be involved in the PDU maintenance T227138: a2-eqiad pdu refresh (Tuesday 10/8 @11am UTC)
We need to failover db1081 to db1138 which is on D8.

Date&Time: 26th September at 05:00 UTC

read-only window will be required.

Event Timeline

Restricted Application added a subscriber: Aklapper. · View Herald TranscriptAug 20 2019, 10:15 AM
Marostegui triaged this task as Normal priority.Aug 20 2019, 10:16 AM
Marostegui moved this task from Triage to Next on the DBA board.
Marostegui moved this task from Next to In progress on the DBA board.

Change 538747 had a related patch set uploaded (by Marostegui; owner: Marostegui):
[operations/puppet@production] mariadb: Promote db1138 to master

https://gerrit.wikimedia.org/r/538747

Change 538748 had a related patch set uploaded (by Marostegui; owner: Marostegui):
[operations/dns@master] wmnet: Update s4-master to point to db1138

https://gerrit.wikimedia.org/r/538748

Mentioned in SAL (#wikimedia-operations) [2019-09-26T04:10:42Z] <marostegui> Start pre-switchover s4 steps T230784

Mentioned in SAL (#wikimedia-operations) [2019-09-26T04:15:09Z] <marostegui@cumin1001> dbctl commit (dc=all): 'Set db1138 with weight 0 T230784', diff saved to https://phabricator.wikimedia.org/P9188 and previous config saved to /var/cache/conftool/dbconfig/20190926-041508-marostegui.json

Change 538748 merged by Marostegui:
[operations/dns@master] wmnet: Update s4-master to point to db1138

https://gerrit.wikimedia.org/r/538748

Change 538747 merged by Marostegui:
[operations/puppet@production] mariadb: Promote db1138 to master

https://gerrit.wikimedia.org/r/538747

Mentioned in SAL (#wikimedia-operations) [2019-09-26T05:00:32Z] <marostegui> Starting s4 failover from db1081 to db1138 - T230784

Mentioned in SAL (#wikimedia-operations) [2019-09-26T05:00:51Z] <marostegui@cumin1001> dbctl commit (dc=all): 'Set s4 as read-only for maintenance T230784', diff saved to https://phabricator.wikimedia.org/P9189 and previous config saved to /var/cache/conftool/dbconfig/20190926-050050-marostegui.json

Mentioned in SAL (#wikimedia-operations) [2019-09-26T05:01:42Z] <marostegui@cumin1001> dbctl commit (dc=all): 'Promote db1138 to s4 master and remove read-only from s4 T230784', diff saved to https://phabricator.wikimedia.org/P9190 and previous config saved to /var/cache/conftool/dbconfig/20190926-050140-marostegui.json

Mentioned in SAL (#wikimedia-operations) [2019-09-26T05:07:23Z] <marostegui@cumin1001> dbctl commit (dc=all): 'Give some weight to db1081 - T230784', diff saved to https://phabricator.wikimedia.org/P9191 and previous config saved to /var/cache/conftool/dbconfig/20190926-050722-marostegui.json

Mentioned in SAL (#wikimedia-operations) [2019-09-26T05:09:39Z] <marostegui@cumin1001> dbctl commit (dc=all): 'Give some API weight to db1081 - T230784', diff saved to https://phabricator.wikimedia.org/P9192 and previous config saved to /var/cache/conftool/dbconfig/20190926-050937-marostegui.json

The switchover was done successfully

Read-only start: 05:00:51
Read-only stop: 05:01:42
Total read-only time: 51 seconds

Mentioned in SAL (#wikimedia-operations) [2019-09-26T05:19:19Z] <marostegui@cumin1001> dbctl commit (dc=all): 'Increase weight for db1081 - T230784', diff saved to https://phabricator.wikimedia.org/P9193 and previous config saved to /var/cache/conftool/dbconfig/20190926-051916-marostegui.json

Mentioned in SAL (#wikimedia-operations) [2019-09-26T05:30:30Z] <marostegui@cumin1001> dbctl commit (dc=all): 'Fully pool db1081 - T230784', diff saved to https://phabricator.wikimedia.org/P9194 and previous config saved to /var/cache/conftool/dbconfig/20190926-053029-marostegui.json

Marostegui closed this task as Resolved.Thu, Sep 26, 5:30 AM