Page MenuHomePhabricator

Switchover s7 primary database master db1062 -> db1086 - 26th Nov 06:00 - 06:30 UTC
Closed, ResolvedPublic

Description

The current s7 master, db1062 is old and needs to be decommissioned.

We are going to switchover db1062 to db1086 on Tue 26th Nov from 06:00 to 06:30
Read only is required, and the following wikis will not allow writes during this maintenance window:

arwiki
cawiki
eswiki
fawiki
frwiktionary
hewiki
huwiki
kowiki
metawiki
rowiki
ukwiki
viwiki

This also includes centralauth database, which means some operations might fail during the read-only period, such as: GlobalRenames, Changing/Confirming emails, logging into new wikis, password changes...

Details

Related Gerrit Patches:
operations/puppet : productiondb1136: Make db1136 candidate master for s7
operations/dns : masterwmnet: Update s7-master CNAME
operations/puppet : productionmariadb: Promote db1086 to s7 primary master

Event Timeline

Restricted Application added subscribers: Strainu, Base, revi, Aklapper. · View Herald TranscriptTue, Nov 12, 9:25 AM
Marostegui triaged this task as Medium priority.Tue, Nov 12, 9:25 AM
Marostegui moved this task from Triage to Next on the DBA board.

There is a typo on the date: it should be "Tue 26th Nov", not "Tue 24th Nov", despite the correct date is shown in the title.

JJMC89 updated the task description. (Show Details)Fri, Nov 15, 11:51 PM

Thank you both for pointing it out and for fixing it! :)

Marostegui moved this task from Next to In progress on the DBA board.Wed, Nov 20, 6:27 AM

All the hosts in s7 have been upgraded to 10.1.43 (including labs and sanitariums)

Strainu removed a subscriber: Strainu.Thu, Nov 21, 10:52 AM

Change 552381 had a related patch set uploaded (by Marostegui; owner: Marostegui):
[operations/puppet@production] mariadb: Promote db1086 to s7 primary master

https://gerrit.wikimedia.org/r/552381

Change 552382 had a related patch set uploaded (by Marostegui; owner: Marostegui):
[operations/dns@master] wmnet: Update s7-master CNAME

https://gerrit.wikimedia.org/r/552382

Mentioned in SAL (#wikimedia-operations) [2019-11-22T06:31:46Z] <marostegui@cumin1001> dbctl commit (dc=all): 'Rebalance weights on s7 in preparation for s7 failover on Tuesday T238044', diff saved to https://phabricator.wikimedia.org/P9722 and previous config saved to /var/cache/conftool/dbconfig/20191122-063145-marostegui.json

Mentioned in SAL (#wikimedia-operations) [2019-11-26T05:08:21Z] <marostegui> Start pre-steps for s7 failover - T238044

Mentioned in SAL (#wikimedia-operations) [2019-11-26T05:10:35Z] <marostegui@cumin1001> dbctl commit (dc=all): 'Set weight 0 to db1086 as it will be the new s7 master - T238044', diff saved to https://phabricator.wikimedia.org/P9741 and previous config saved to /var/cache/conftool/dbconfig/20191126-051034-marostegui.json

Change 552381 merged by Marostegui:
[operations/puppet@production] mariadb: Promote db1086 to s7 primary master

https://gerrit.wikimedia.org/r/552381

Mentioned in SAL (#wikimedia-operations) [2019-11-26T06:00:14Z] <marostegui> Starting s7 failover from db1062 to db1086 - T238044

Mentioned in SAL (#wikimedia-operations) [2019-11-26T06:00:24Z] <marostegui@cumin2001> dbctl commit (dc=all): 'Set s7 as read-only for maintenance T238044', diff saved to https://phabricator.wikimedia.org/P9742 and previous config saved to /var/cache/conftool/dbconfig/20191126-060023-marostegui.json

Mentioned in SAL (#wikimedia-operations) [2019-11-26T06:01:09Z] <marostegui@cumin2001> dbctl commit (dc=all): 'Promote db1086 on s7 master and remove read-only from s7 T238044', diff saved to https://phabricator.wikimedia.org/P9743 and previous config saved to /var/cache/conftool/dbconfig/20191126-060108-marostegui.json

Change 552382 merged by Marostegui:
[operations/dns@master] wmnet: Update s7-master CNAME

https://gerrit.wikimedia.org/r/552382

Marostegui closed this task as Resolved.Tue, Nov 26, 6:29 AM

This was done successfully.

Read only starts: 06:00:24
Read only stops: 06:01:09

Total read only time: 45 seconds

Change 552957 had a related patch set uploaded (by Marostegui; owner: Marostegui):
[operations/puppet@production] db1136: Make db1136 candidate master for s7

https://gerrit.wikimedia.org/r/552957

Change 552957 merged by Marostegui:
[operations/puppet@production] db1136: Make db1136 candidate master for s7

https://gerrit.wikimedia.org/r/552957