⚓ T230784 Switchover s4 (commonswiki) primary database master db1081 -> db1138

	Project	Branch	Lines +/-	Subject
	operations/puppet	production	+4 -4	mariadb: Promote db1138 to master
	operations/dns	master	+1 -1	wmnet: Update s4-master to point to db1138

Status	Assigned	Task
Resolved	Cmjohnson	T226778 Install new PDUs in rows A/B (Top level tracking task)
Resolved	Jclark-ctr	T227138 a2-eqiad pdu refresh (Tuesday 10/8 @11am UTC)
Resolved	• Marostegui	T230784 Switchover s4 (commonswiki) primary database master db1081 -> db1138 - 26th Sept @05:00 UTC
Resolved	Trizek-WMF	T230788 Community Relations support needed for several read-only windows (s2, s3, s4 and s8)

• Marostegui triaged this task as Medium priority.Aug 20 2019, 10:16 AM

• Marostegui moved this task from Triage to Pending comment on the DBA board.

Change 538747 had a related patch set uploaded (by Marostegui; owner: Marostegui):
[operations/puppet@production] mariadb: Promote db1138 to master

Change 538748 had a related patch set uploaded (by Marostegui; owner: Marostegui):
[operations/dns@master] wmnet: Update s4-master to point to db1138

Mentioned in SAL (#wikimedia-operations) [2019-09-26T04:10:42Z] <marostegui> Start pre-switchover s4 steps T230784

Mentioned in SAL (#wikimedia-operations) [2019-09-26T04:15:09Z] <marostegui@cumin1001> dbctl commit (dc=all): 'Set db1138 with weight 0 T230784', diff saved to https://phabricator.wikimedia.org/P9188 and previous config saved to /var/cache/conftool/dbconfig/20190926-041508-marostegui.json

Change 538748 merged by Marostegui:
[operations/dns@master] wmnet: Update s4-master to point to db1138

Change 538747 merged by Marostegui:
[operations/puppet@production] mariadb: Promote db1138 to master

Mentioned in SAL (#wikimedia-operations) [2019-09-26T05:00:32Z] <marostegui> Starting s4 failover from db1081 to db1138 - T230784

Mentioned in SAL (#wikimedia-operations) [2019-09-26T05:00:51Z] <marostegui@cumin1001> dbctl commit (dc=all): 'Set s4 as read-only for maintenance T230784', diff saved to https://phabricator.wikimedia.org/P9189 and previous config saved to /var/cache/conftool/dbconfig/20190926-050050-marostegui.json

Mentioned in SAL (#wikimedia-operations) [2019-09-26T05:01:42Z] <marostegui@cumin1001> dbctl commit (dc=all): 'Promote db1138 to s4 master and remove read-only from s4 T230784', diff saved to https://phabricator.wikimedia.org/P9190 and previous config saved to /var/cache/conftool/dbconfig/20190926-050140-marostegui.json

Mentioned in SAL (#wikimedia-operations) [2019-09-26T05:07:23Z] <marostegui@cumin1001> dbctl commit (dc=all): 'Give some weight to db1081 - T230784', diff saved to https://phabricator.wikimedia.org/P9191 and previous config saved to /var/cache/conftool/dbconfig/20190926-050722-marostegui.json

Mentioned in SAL (#wikimedia-operations) [2019-09-26T05:09:39Z] <marostegui@cumin1001> dbctl commit (dc=all): 'Give some API weight to db1081 - T230784', diff saved to https://phabricator.wikimedia.org/P9192 and previous config saved to /var/cache/conftool/dbconfig/20190926-050937-marostegui.json

The switchover was done successfully

Read-only start: 05:00:51
Read-only stop: 05:01:42
Total read-only time: 51 seconds

Mentioned in SAL (#wikimedia-operations) [2019-09-26T05:19:19Z] <marostegui@cumin1001> dbctl commit (dc=all): 'Increase weight for db1081 - T230784', diff saved to https://phabricator.wikimedia.org/P9193 and previous config saved to /var/cache/conftool/dbconfig/20190926-051916-marostegui.json

Mentioned in SAL (#wikimedia-operations) [2019-09-26T05:30:30Z] <marostegui@cumin1001> dbctl commit (dc=all): 'Fully pool db1081 - T230784', diff saved to https://phabricator.wikimedia.org/P9194 and previous config saved to /var/cache/conftool/dbconfig/20190926-053029-marostegui.json

Switchover s4 (commonswiki) primary database master db1081 -> db1138 - 26th Sept @05:00 UTC
Closed, ResolvedPublic
Actions