Page MenuHomePhabricator

Switchover s1 primary database master db1067 -> db1083 - 14th Nov 06:00 - 06:30 UTC
Closed, ResolvedPublic

Description

The current s1 master, db1067 is old and needs to be decommissioned.

We are going to switchover db1067 to db1083 on Thu 14th Nov from 06:00 to 06:30
Read only is required, and the following wikis will not allow writes during this maintenance window:

enwiki

Details

Related Gerrit Patches:
operations/puppet : productiondb1134: Set it as candidate master for s1
operations/dns : masterwmnet: Point s1-master to db1083
operations/puppet : productionmariadb: Promote db1083 to s1 master
operations/puppet : productionmariadb: db1134 will be the future candidate master for s1

Related Objects

Event Timeline

Marostegui triaged this task as Medium priority.Oct 7 2019, 9:14 AM
Marostegui created this task.
Marostegui moved this task from Triage to Next on the DBA board.
Xaosflux added a subscriber: User-notice.
Xaosflux removed a subscriber: User-notice.
Johan added a subscriber: Johan.Oct 10 2019, 4:50 PM

This only affects English Wikipedia, right?

In that case, we don't need to take the less efficient way of writing in Tech News, better to contact the wiki directly.

jcrespo added a subscriber: jcrespo.EditedNov 8 2019, 9:16 AM

Hi, a reminder this was scheduled well in advance, probably announcement was already done, but I thought it was worth giving a reminder here one week away.

Marostegui moved this task from Next to In progress on the DBA board.Tue, Nov 12, 7:04 AM

Mentioned in SAL (#wikimedia-operations) [2019-11-12T07:04:38Z] <marostegui@cumin1001> dbctl commit (dc=all): 'Depool db1083 for kernel upgrade - T234800', diff saved to https://phabricator.wikimedia.org/P9589 and previous config saved to /var/cache/conftool/dbconfig/20191112-070436-marostegui.json

I just realised that when we created this task and scheduled the maintenance I didn't take into account that we'd have had the change from CEST to CET, so I am going to update the task to reflect that this will happen at 06:00 AM UTC instead of 05:00 AM UTC.
Sorry for the confusion.

Marostegui renamed this task from Switchover s1 primary database master db1067 -> db1083 - 14th Nov 05:00 - 05:30 UTC to Switchover s1 primary database master db1067 -> db1083 - 14th Nov 06:00 - 06:30 UTC.Tue, Nov 12, 7:26 AM
Marostegui updated the task description. (Show Details)

Mentioned in SAL (#wikimedia-operations) [2019-11-12T09:11:59Z] <marostegui@cumin1001> dbctl commit (dc=all): 'Depool db1083 for mariadb upgrade to 10.1.39 - T234800', diff saved to https://phabricator.wikimedia.org/P9594 and previous config saved to /var/cache/conftool/dbconfig/20191112-091158-marostegui.json

Change 550514 had a related patch set uploaded (by Marostegui; owner: Marostegui):
[operations/puppet@production] mariadb: db1134 will be the future candidate master for s1

https://gerrit.wikimedia.org/r/550514

Change 550514 merged by Marostegui:
[operations/puppet@production] mariadb: db1134 will be the future candidate master for s1

https://gerrit.wikimedia.org/r/550514

Change 550587 had a related patch set uploaded (by Marostegui; owner: Marostegui):
[operations/puppet@production] mariadb: Promote db1083 to s1 master

https://gerrit.wikimedia.org/r/550587

Change 550588 had a related patch set uploaded (by Marostegui; owner: Marostegui):
[operations/dns@master] wmnet: Point s1-master to db1083

https://gerrit.wikimedia.org/r/550588

Mentioned in SAL (#wikimedia-operations) [2019-11-14T05:07:24Z] <marostegui> Start pre-failover steps T234800

Mentioned in SAL (#wikimedia-operations) [2019-11-14T05:09:41Z] <marostegui@cumin1001> dbctl commit (dc=all): 'Set db1083 with weight 0 T234800', diff saved to https://phabricator.wikimedia.org/P9625 and previous config saved to /var/cache/conftool/dbconfig/20191114-050940-marostegui.json

Mentioned in SAL (#wikimedia-operations) [2019-11-14T05:13:36Z] <marostegui> Move replicas from db1067 to db1083 T234800

Change 550587 merged by Marostegui:
[operations/puppet@production] mariadb: Promote db1083 to s1 master

https://gerrit.wikimedia.org/r/550587

Mentioned in SAL (#wikimedia-operations) [2019-11-14T06:00:16Z] <marostegui> Starting s1 failover from db1067 to db1083 - T234800

Mentioned in SAL (#wikimedia-operations) [2019-11-14T06:00:28Z] <marostegui@cumin2001> dbctl commit (dc=all): 'Set s1 as read-only for maintenance T234800', diff saved to https://phabricator.wikimedia.org/P9628 and previous config saved to /var/cache/conftool/dbconfig/20191114-060026-marostegui.json

Mentioned in SAL (#wikimedia-operations) [2019-11-14T06:01:39Z] <marostegui@cumin2001> dbctl commit (dc=all): 'Promote db1083 to s1 master and remove read-only from s1 T234800', diff saved to https://phabricator.wikimedia.org/P9629 and previous config saved to /var/cache/conftool/dbconfig/20191114-060138-marostegui.json

Change 550588 merged by Marostegui:
[operations/dns@master] wmnet: Point s1-master to db1083

https://gerrit.wikimedia.org/r/550588

Change 550793 had a related patch set uploaded (by Marostegui; owner: Marostegui):
[operations/puppet@production] db1134: Set it as candidate master for s1

https://gerrit.wikimedia.org/r/550793

Change 550793 merged by Marostegui:
[operations/puppet@production] db1134: Set it as candidate master for s1

https://gerrit.wikimedia.org/r/550793

This was done.

Read only start: 06:00:28
Read only stop: 06:01:39

Total read only time: 01:11 minutes

Marostegui closed this task as Resolved.Thu, Nov 14, 6:40 AM