Page MenuHomePhabricator

Upgrade and restart s3 and s7 primary DB master: Thu 7th May
Closed, ResolvedPublic

Description

We need to upgrade and restart s3 and s7 primary database master as part of T239791: DB: perform rolling restart of mariadb daemons to pick up CA changes

The following wikis will go read-only:

s3: https://noc.wikimedia.org/conf/highlight.php?file=dblists/s3.dblist

s7:

arwiki
cawiki
eswiki
fawiki
frwiktionary
hewiki
huwiki
kowiki
metawiki
rowiki
ukwiki
viwiki

s7 also includes centralauth database, which means some operations might fail during the read-only period, such as: GlobalRenames, Changing/Confirming emails, logging into new wikis, password changes...

Impact: Those wikis will go read-only for around 15 minutes (hopefully a lot less). Reads will not be impacted
When: Thu 7th May at 05:00 AM UTC

s3 primary db master: db1123
s7 primary db master: db1086

The read only window request is at: T251157

Event Timeline

Restricted Application added subscribers: Strainu, Base, revi, Aklapper. · View Herald TranscriptApr 27 2020, 3:19 PM
Marostegui triaged this task as Medium priority.Apr 27 2020, 3:19 PM
Marostegui moved this task from Triage to Next on the DBA board.
Marostegui renamed this task from Upgrade and restart s3 and s7 primary DB master to Upgrade and restart s3 and s7 primary DB master: Thu 7th May.Apr 30 2020, 9:10 AM

Added the slot on the deployment's page

Marostegui added a comment.EditedMay 1 2020, 2:20 PM

Day before:

  • Install the 10.1.43-2 package on both masters (db1123 and db1086)

Maintenance day:

  • Silence all hosts in s3 and s7
  • Set read only on s3 and s7:
dbctl --scope eqiad section s3 ro "Maintenance on s3 and s7 T251158" && dbctl --scope eqiad section s7 ro "Maintenance on s3 and s7 T251158"  && dbctl config commit -m "Set s3 and s7 as read-only for maintenance T251158"
  • Confirm read only on the wikis
  • Set read-only on the masters on mysql: db1123 db1086
  • Restart mysql on both hosts: db1123 db1086
  • Run puppet
  • Confirm the slaves are connected
  • Remove read only:
dbctl --scope eqiad section s3 rw  && dbctl --scope eqiad section s7 rw && dbctl config commit -m "Set s3 and s7 as read-only=off for maintenance T251158"
  • Confirm writes can go through
  • Run mysql_upgrade on both hosts: db1123 db1086
  • Close task
Marostegui moved this task from Next to In progress on the DBA board.Tue, May 5, 5:31 AM

Mentioned in SAL (#wikimedia-operations) [2020-05-06T09:12:10Z] <marostegui> Upgrade package on s3 and s7 master (db1123 and db1086) in preparation for tomorrow's restart - T251158

Package upgraded on db1123 and db1086.

Ovedc added a subscriber: Ovedc.Thu, May 7, 4:47 AM

Hello! can you please allow the register users dismiss the notice, once they read it? Thanks!

Hello! can you please allow the register users dismiss the notice, once they read it? Thanks!

Thanks for the message. However we (DBAs) cannot implement that, you might want to comment on T251157 and let them know if that's possible for next iterations.

Thank you!

Mentioned in SAL (#wikimedia-operations) [2020-05-07T05:00:47Z] <marostegui@cumin1001> dbctl commit (dc=all): 'Set s3 and s7 as read-only for maintenance T251158', diff saved to https://phabricator.wikimedia.org/P11166 and previous config saved to /var/cache/conftool/dbconfig/20200507-050046-marostegui.json

Mentioned in SAL (#wikimedia-operations) [2020-05-07T05:04:19Z] <marostegui@cumin1001> dbctl commit (dc=all): 'Set s3 and s7 as read-only=off for maintenance T251158', diff saved to https://phabricator.wikimedia.org/P11167 and previous config saved to /var/cache/conftool/dbconfig/20200507-050419-marostegui.json

Marostegui closed this task as Resolved.Thu, May 7, 5:14 AM

This is done.
RO started: 05:00:47
RO finished: 05:04:19