Page MenuHomePhabricator

Switchover s6 primary database master db1061 -> db1131 - 19th Nov 06:00 - 06:30 UTC
Closed, ResolvedPublic

Description

The current s6 master, db1061 is old and needs to be decommissioned.

We are going to switchover db1061 to db1131 on Tue 19th Nov from 06:00 to 06:30
Read only is required, and the following wikis will not allow writes during this maintenance window:

frwiki
jawiki
ruwiki

Details

Related Gerrit Patches:
operations/dns : masterwmnet: Update s6-master CNAME
operations/puppet : productionmariadb: Promote db1131 to s6 master

Event Timeline

Restricted Application added a subscriber: Aklapper. · View Herald TranscriptOct 15 2019, 6:32 AM
Marostegui triaged this task as Medium priority.Oct 15 2019, 6:33 AM
Marostegui moved this task from Triage to Next on the DBA board.

I just realised that when we created this task and scheduled the maintenance I didn't take into account that we'd have had the change from CEST to CET, so I am going to update the task to reflect that this will happen at 06:00 AM UTC instead of 05:00 AM UTC.
Sorry for the confusion.

Marostegui renamed this task from Switchover s6 primary database master db1061 -> db1131 - 19th Nov 05:00 - 05:30 UTC to Switchover s6 primary database master db1061 -> db1131 - 19th Nov 06:00 - 06:30 UTC.Tue, Nov 12, 7:27 AM
Marostegui updated the task description. (Show Details)
Marostegui moved this task from Next to In progress on the DBA board.Fri, Nov 15, 7:11 AM

Change 551040 had a related patch set uploaded (by Marostegui; owner: Marostegui):
[operations/puppet@production] mariadb: Promote db1131 to s6 master

https://gerrit.wikimedia.org/r/551040

Change 551041 had a related patch set uploaded (by Marostegui; owner: Marostegui):
[operations/dns@master] wmnet: Update s6-master CNAME

https://gerrit.wikimedia.org/r/551041

Mentioned in SAL (#wikimedia-operations) [2019-11-19T05:02:13Z] <marostegui> Start pre-switchover steps T235469

Mentioned in SAL (#wikimedia-operations) [2019-11-19T05:07:49Z] <marostegui@cumin1001> dbctl commit (dc=all): 'Set db1131 with weight 0 T235469 ', diff saved to https://phabricator.wikimedia.org/P9661 and previous config saved to /var/cache/conftool/dbconfig/20191119-050748-marostegui.json

Change 551040 merged by Marostegui:
[operations/puppet@production] mariadb: Promote db1131 to s6 master

https://gerrit.wikimedia.org/r/551040

Mentioned in SAL (#wikimedia-operations) [2019-11-19T06:01:03Z] <marostegui> Starting s6 failover from db1061 to db1131 - T235469

Mentioned in SAL (#wikimedia-operations) [2019-11-19T06:01:23Z] <marostegui@cumin2001> dbctl commit (dc=all): 'Set s6 as read-only for maintenance T235469', diff saved to https://phabricator.wikimedia.org/P9666 and previous config saved to /var/cache/conftool/dbconfig/20191119-060122-marostegui.json

Mentioned in SAL (#wikimedia-operations) [2019-11-19T06:02:04Z] <marostegui@cumin2001> dbctl commit (dc=all): 'Promote db1131 to s6 master and remove read-only from s6 T235469', diff saved to https://phabricator.wikimedia.org/P9667 and previous config saved to /var/cache/conftool/dbconfig/20191119-060203-marostegui.json

Change 551041 merged by Marostegui:
[operations/dns@master] wmnet: Update s6-master CNAME

https://gerrit.wikimedia.org/r/551041

Marostegui closed this task as Resolved.Tue, Nov 19, 6:12 AM

This was done successfully.
Read only starts: 06:01:23Read only stops: 06:02:04
Total read only time: 41 seconds