Page MenuHomePhabricator
Create Task
Maniphest T230784

Switchover s4 (commonswiki) primary database master db1081 -> db1138 - 26th Sept @05:00 UTC
Closed, ResolvedPublic
Actions

Description

db1081 is on A2, which will be involved in the PDU maintenance T227138: a2-eqiad pdu refresh (Tuesday 10/8 @11am UTC)
We need to failover db1081 to db1138 which is on D8.

Date&Time: 26th September at 05:00 UTC

read-only window will be required.

Details

SubjectRepoBranchLines +/-
mariadb: Promote db1138 to masteroperations/puppetproduction+4 -4
wmnet: Update s4-master to point to db1138operations/dnsmaster+1 -1
Customize query in gerrit

Related Objects
Search...

Event Timeline

Marostegui created this task.Aug 20 2019, 10:15 AM
Restricted Application added a subscriber: Aklapper. · View Herald TranscriptAug 20 2019, 10:15 AM
Marostegui triaged this task as Medium priority.Aug 20 2019, 10:16 AM
Marostegui moved this task from Triage to Pending comment on the DBA board.
Marostegui added a subtask: T230788: Community Relations support needed for several read-only windows (s2, s3, s4 and s8).Aug 20 2019, 10:36 AM
Marostegui mentioned this in T230788: Community Relations support needed for several read-only windows (s2, s3, s4 and s8).Aug 20 2019, 10:41 AM
RhinosF1 subscribed.Sep 8 2019, 11:14 PM
Marostegui added a comment.Sep 16 2019, 12:56 PM

Reserved window on the deployments calendar: https://wikitech.wikimedia.org/w/index.php?title=Deployments&type=revision&diff=1837750&oldid=1837737

Marostegui claimed this task.Sep 24 2019, 6:24 AM
Marostegui moved this task from Pending comment to In progress on the DBA board.
gerritbot added a comment.Sep 24 2019, 7:18 AM

Change 538747 had a related patch set uploaded (by Marostegui; owner: Marostegui):
[operations/puppet@production] mariadb: Promote db1138 to master

https://gerrit.wikimedia.org/r/538747

gerritbot added a project: Patch-For-Review.Sep 24 2019, 7:18 AM
gerritbot added a comment.Sep 24 2019, 7:20 AM

Change 538748 had a related patch set uploaded (by Marostegui; owner: Marostegui):
[operations/dns@master] wmnet: Update s4-master to point to db1138

https://gerrit.wikimedia.org/r/538748

Trizek-WMF closed subtask T230788: Community Relations support needed for several read-only windows (s2, s3, s4 and s8) as Resolved.Sep 24 2019, 12:30 PM
Stashbot added a comment.Sep 26 2019, 4:10 AM

Mentioned in SAL (#wikimedia-operations) [2019-09-26T04:10:42Z] <marostegui> Start pre-switchover s4 steps T230784

Stashbot added a comment.Sep 26 2019, 4:15 AM

Mentioned in SAL (#wikimedia-operations) [2019-09-26T04:15:09Z] <marostegui@cumin1001> dbctl commit (dc=all): 'Set db1138 with weight 0 T230784', diff saved to https://phabricator.wikimedia.org/P9188 and previous config saved to /var/cache/conftool/dbconfig/20190926-041508-marostegui.json

gerritbot added a comment.Sep 26 2019, 4:35 AM

Change 538748 merged by Marostegui:
[operations/dns@master] wmnet: Update s4-master to point to db1138

https://gerrit.wikimedia.org/r/538748

gerritbot added a comment.Sep 26 2019, 4:36 AM

Change 538747 merged by Marostegui:
[operations/puppet@production] mariadb: Promote db1138 to master

https://gerrit.wikimedia.org/r/538747

Stashbot added a comment.Sep 26 2019, 5:00 AM

Mentioned in SAL (#wikimedia-operations) [2019-09-26T05:00:32Z] <marostegui> Starting s4 failover from db1081 to db1138 - T230784

Stashbot added a comment.Sep 26 2019, 5:00 AM

Mentioned in SAL (#wikimedia-operations) [2019-09-26T05:00:51Z] <marostegui@cumin1001> dbctl commit (dc=all): 'Set s4 as read-only for maintenance T230784', diff saved to https://phabricator.wikimedia.org/P9189 and previous config saved to /var/cache/conftool/dbconfig/20190926-050050-marostegui.json

Stashbot added a comment.Sep 26 2019, 5:01 AM

Mentioned in SAL (#wikimedia-operations) [2019-09-26T05:01:42Z] <marostegui@cumin1001> dbctl commit (dc=all): 'Promote db1138 to s4 master and remove read-only from s4 T230784', diff saved to https://phabricator.wikimedia.org/P9190 and previous config saved to /var/cache/conftool/dbconfig/20190926-050140-marostegui.json

Stashbot added a comment.Sep 26 2019, 5:07 AM

Mentioned in SAL (#wikimedia-operations) [2019-09-26T05:07:23Z] <marostegui@cumin1001> dbctl commit (dc=all): 'Give some weight to db1081 - T230784', diff saved to https://phabricator.wikimedia.org/P9191 and previous config saved to /var/cache/conftool/dbconfig/20190926-050722-marostegui.json

Stashbot added a comment.Sep 26 2019, 5:09 AM

Mentioned in SAL (#wikimedia-operations) [2019-09-26T05:09:39Z] <marostegui@cumin1001> dbctl commit (dc=all): 'Give some API weight to db1081 - T230784', diff saved to https://phabricator.wikimedia.org/P9192 and previous config saved to /var/cache/conftool/dbconfig/20190926-050937-marostegui.json

Maintenance_bot removed a project: Patch-For-Review.Sep 26 2019, 5:10 AM
Marostegui added a comment.Sep 26 2019, 5:10 AM

The switchover was done successfully

Read-only start: 05:00:51
Read-only stop: 05:01:42
Total read-only time: 51 seconds

Stashbot added a comment.Sep 26 2019, 5:19 AM

Mentioned in SAL (#wikimedia-operations) [2019-09-26T05:19:19Z] <marostegui@cumin1001> dbctl commit (dc=all): 'Increase weight for db1081 - T230784', diff saved to https://phabricator.wikimedia.org/P9193 and previous config saved to /var/cache/conftool/dbconfig/20190926-051916-marostegui.json

Stashbot added a comment.Sep 26 2019, 5:30 AM

Mentioned in SAL (#wikimedia-operations) [2019-09-26T05:30:30Z] <marostegui@cumin1001> dbctl commit (dc=all): 'Fully pool db1081 - T230784', diff saved to https://phabricator.wikimedia.org/P9194 and previous config saved to /var/cache/conftool/dbconfig/20190926-053029-marostegui.json

Marostegui closed this task as Resolved.Sep 26 2019, 5:30 AM
jcrespo awarded a token.Sep 26 2019, 6:30 AM
Content licensed under Creative Commons Attribution-ShareAlike (CC BY-SA) 4.0 unless otherwise noted; code licensed under GNU General Public License (GPL) 2.0 or later and other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL · Credits