Page MenuHomePhabricator
Create Task
Maniphest T238044

Switchover s7 primary database master db1062 -> db1086 - 26th Nov 06:00 - 06:30 UTC
Closed, ResolvedPublic
Actions

Description

The current s7 master, db1062 is old and needs to be decommissioned.

We are going to switchover db1062 to db1086 on Tue 26th Nov from 06:00 to 06:30
Read only is required, and the following wikis will not allow writes during this maintenance window:

arwiki
cawiki
eswiki
fawiki
frwiktionary
hewiki
huwiki
kowiki
metawiki
rowiki
ukwiki
viwiki

This also includes centralauth database, which means some operations might fail during the read-only period, such as: GlobalRenames, Changing/Confirming emails, logging into new wikis, password changes...

Details

SubjectRepoBranchLines +/-
db1136: Make db1136 candidate master for s7operations/puppetproduction+1 -0
wmnet: Update s7-master CNAMEoperations/dnsmaster+1 -1
mariadb: Promote db1086 to s7 primary masteroperations/puppetproduction+3 -4
Customize query in gerrit

Related Objects
Search...

Event Timeline

• Marostegui created this task.Nov 12 2019, 9:25 AM
Restricted Application added subscribers: Strainu, Base, revi, Aklapper. · View Herald TranscriptNov 12 2019, 9:25 AM
• Marostegui triaged this task as Medium priority.Nov 12 2019, 9:25 AM
• Marostegui added parent tasks: T186188: Failover DB masters in row D, T217396: Decommission db1061-db1073.
• Marostegui moved this task from Triage to Pending comment on the DBA board.
• Marostegui mentioned this in T238046: Community Relations support needed for a read-only window for s7 (several wikis).Nov 12 2019, 9:30 AM
• Marostegui added a subtask: T238046: Community Relations support needed for a read-only window for s7 (several wikis).
• Marostegui added a project: User-notice.
Johan moved this task from To Triage to In current Tech/News draft on the User-notice board.Nov 14 2019, 12:33 PM
Agusbou2015 subscribed.Nov 15 2019, 11:27 PM

There is a typo on the date: it should be "Tue 26th Nov", not "Tue 24th Nov", despite the correct date is shown in the title.

JJMC89 updated the task description. (Show Details)Nov 15 2019, 11:51 PM
• Marostegui added a comment.Nov 18 2019, 5:55 AM

Thank you both for pointing it out and for fixing it! :)

RhinosF1 subscribed.Nov 18 2019, 8:30 PM
• Marostegui moved this task from Pending comment to In progress on the DBA board.Nov 20 2019, 6:27 AM
Johan moved this task from In current Tech/News draft to Already announced/Archive on the User-notice board.Nov 20 2019, 3:03 PM
• Marostegui added a comment.Nov 21 2019, 9:38 AM

All the hosts in s7 have been upgraded to 10.1.43 (including labs and sanitariums)

Strainu unsubscribed.Nov 21 2019, 10:52 AM
• Marostegui claimed this task.Nov 22 2019, 6:20 AM
gerritbot added a comment.Nov 22 2019, 6:24 AM

Change 552381 had a related patch set uploaded (by Marostegui; owner: Marostegui):
[operations/puppet@production] mariadb: Promote db1086 to s7 primary master

https://gerrit.wikimedia.org/r/552381

gerritbot added a project: Patch-For-Review.Nov 22 2019, 6:24 AM

Change 552382 had a related patch set uploaded (by Marostegui; owner: Marostegui):
[operations/dns@master] wmnet: Update s7-master CNAME

https://gerrit.wikimedia.org/r/552382

Stashbot added a comment.Nov 22 2019, 6:31 AM

Mentioned in SAL (#wikimedia-operations) [2019-11-22T06:31:46Z] <marostegui@cumin1001> dbctl commit (dc=all): 'Rebalance weights on s7 in preparation for s7 failover on Tuesday T238044', diff saved to https://phabricator.wikimedia.org/P9722 and previous config saved to /var/cache/conftool/dbconfig/20191122-063145-marostegui.json

Stashbot added a comment.Nov 26 2019, 5:08 AM

Mentioned in SAL (#wikimedia-operations) [2019-11-26T05:08:21Z] <marostegui> Start pre-steps for s7 failover - T238044

Stashbot added a comment.Nov 26 2019, 5:10 AM

Mentioned in SAL (#wikimedia-operations) [2019-11-26T05:10:35Z] <marostegui@cumin1001> dbctl commit (dc=all): 'Set weight 0 to db1086 as it will be the new s7 master - T238044', diff saved to https://phabricator.wikimedia.org/P9741 and previous config saved to /var/cache/conftool/dbconfig/20191126-051034-marostegui.json

gerritbot added a comment.Nov 26 2019, 5:25 AM

Change 552381 merged by Marostegui:
[operations/puppet@production] mariadb: Promote db1086 to s7 primary master

https://gerrit.wikimedia.org/r/552381

Stashbot added a comment.Nov 26 2019, 6:00 AM

Mentioned in SAL (#wikimedia-operations) [2019-11-26T06:00:14Z] <marostegui> Starting s7 failover from db1062 to db1086 - T238044

Stashbot added a comment.Nov 26 2019, 6:00 AM

Mentioned in SAL (#wikimedia-operations) [2019-11-26T06:00:24Z] <marostegui@cumin2001> dbctl commit (dc=all): 'Set s7 as read-only for maintenance T238044', diff saved to https://phabricator.wikimedia.org/P9742 and previous config saved to /var/cache/conftool/dbconfig/20191126-060023-marostegui.json

Stashbot added a comment.Nov 26 2019, 6:01 AM

Mentioned in SAL (#wikimedia-operations) [2019-11-26T06:01:09Z] <marostegui@cumin2001> dbctl commit (dc=all): 'Promote db1086 on s7 master and remove read-only from s7 T238044', diff saved to https://phabricator.wikimedia.org/P9743 and previous config saved to /var/cache/conftool/dbconfig/20191126-060108-marostegui.json

gerritbot added a comment.Nov 26 2019, 6:03 AM

Change 552382 merged by Marostegui:
[operations/dns@master] wmnet: Update s7-master CNAME

https://gerrit.wikimedia.org/r/552382

• Marostegui closed subtask T238046: Community Relations support needed for a read-only window for s7 (several wikis) as Resolved.Nov 26 2019, 6:19 AM
• Marostegui mentioned this in T239187: ErrorException from line 1432 of /srv/mediawiki/php-1.35.0-wmf.5/includes/media/FormatMetadata.php: PHP Notice: A non well formed numeric value encountered.Nov 26 2019, 6:25 AM
• Marostegui closed this task as Resolved.Nov 26 2019, 6:29 AM

This was done successfully.

Read only starts: 06:00:24
Read only stops: 06:01:09

Total read only time: 45 seconds

gerritbot added a comment.Nov 26 2019, 6:35 AM

Change 552957 had a related patch set uploaded (by Marostegui; owner: Marostegui):
[operations/puppet@production] db1136: Make db1136 candidate master for s7

https://gerrit.wikimedia.org/r/552957

gerritbot added a comment.Nov 26 2019, 6:37 AM

Change 552957 merged by Marostegui:
[operations/puppet@production] db1136: Make db1136 candidate master for s7

https://gerrit.wikimedia.org/r/552957

Maintenance_bot removed a project: Patch-For-Review.Nov 26 2019, 11:54 AM
Ladsgroup edited projects, added User-notice-archive; removed User-notice.Aug 13 2022, 1:54 PM
Restricted Application added a subscriber: Huji. · View Herald TranscriptAug 13 2022, 1:54 PM
Maintenance_bot moved this task from In progress to Done on the DBA board.Aug 13 2022, 2:16 PM
Content licensed under Creative Commons Attribution-ShareAlike (CC BY-SA) 4.0 unless otherwise noted; code licensed under GNU General Public License (GPL) 2.0 or later and other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL · Credits