Page MenuHomePhabricator

Switchover s8 (wikidata) primary database master db1104 -> db1109 - 10th Sept @05:00 UTC
Closed, ResolvedPublic

Description

db1104 is on B3, which will be involved in the PDU maintenance T227539: b3-eqiad pdu refresh (Tuesday 9/17 @11am UTC)
We need to failover db1104 to db1109 which is on D8.

Date&Time: 10th September at 05:00 UTC

read-only window will be required.

Event Timeline

Restricted Application added a subscriber: Aklapper. · View Herald TranscriptAug 20 2019, 6:07 AM
Marostegui triaged this task as Normal priority.Aug 20 2019, 6:08 AM
Marostegui moved this task from Triage to Next on the DBA board.Aug 20 2019, 6:10 AM
Marostegui updated the task description. (Show Details)Aug 20 2019, 10:04 AM
Marostegui renamed this task from Switchover s8 (wikidata) primary database master db1104 -> db1109 to Switchover s8 (wikidata) primary database master db1104 -> db1109 - 10th Sept @05:00 UTC.Aug 20 2019, 10:11 AM

Change 531189 had a related patch set uploaded (by Marostegui; owner: Marostegui):
[operations/puppet@production] mariadb: Promote db1109 to s8 master

https://gerrit.wikimedia.org/r/531189

Marostegui moved this task from Next to In progress on the DBA board.Aug 21 2019, 5:20 AM

Change 531455 had a related patch set uploaded (by Marostegui; owner: Marostegui):
[operations/dns@master] wmnet: Update s8-master record

https://gerrit.wikimedia.org/r/531455

I have reserved the window on the Deployments page.

Mentioned in SAL (#wikimedia-operations) [2019-09-10T04:18:45Z] <marostegui> Start s8 (wikidata) pre switchover steps T230762

Mentioned in SAL (#wikimedia-operations) [2019-09-10T04:22:44Z] <marostegui@cumin1001> dbctl commit (dc=all): 'Set db1109 with weight 0 and depool it from API T230762', diff saved to https://phabricator.wikimedia.org/P9068 and previous config saved to /var/cache/conftool/dbconfig/20190910-042243-marostegui.json

Mentioned in SAL (#wikimedia-operations) [2019-09-10T04:23:40Z] <marostegui> Start topology changes on s8, connect everything under db1109 - T230762

Change 531189 merged by Marostegui:
[operations/puppet@production] mariadb: Promote db1109 to s8 master

https://gerrit.wikimedia.org/r/531189

Mentioned in SAL (#wikimedia-operations) [2019-09-10T05:00:48Z] <marostegui@cumin1001> dbctl commit (dc=all): 'Set s8 as read-only for maintenance T230762', diff saved to https://phabricator.wikimedia.org/P9069 and previous config saved to /var/cache/conftool/dbconfig/20190910-050046-marostegui.json

Change 531455 merged by Marostegui:
[operations/dns@master] wmnet: Update s8-master record

https://gerrit.wikimedia.org/r/531455

This was done.
read-only start: Tue Sep 10 05:00:47 UTC 2019
read-only stop: Tue Sep 10 05:02:14 UTC 2019

Total read-only time: 1 minute 27 seconds.

Mentioned in SAL (#wikimedia-operations) [2019-09-10T05:15:30Z] <marostegui@cumin1001> dbctl commit (dc=all): 'Pool db1104 into API T230762', diff saved to https://phabricator.wikimedia.org/P9071 and previous config saved to /var/cache/conftool/dbconfig/20190910-051529-marostegui.json

Marostegui closed this task as Resolved.Tue, Sep 10, 5:19 AM