Page MenuHomePhabricator

Upgrade and restart s5 and s6 primary DB master: Tue 5th May
Closed, ResolvedPublic

Description

We need to upgrade and restart s5 and s6 primary database master as part of T239791: DB: perform rolling restart of mariadb daemons to pick up CA changes

The following wikis will go read-only:

s5:

cebwiki
dewiki
enwikivoyage
mgwiktionary
shwiki
srwiki

s6:

frwiki
jawiki
ruwiki

Impact: Those wikis will go read-only for around 15 minutes (hopefully a lot less). Reads will not be impacted
When: Tuesday 5th May at 05:00 AM UTC

s5 primary db master: db1100
s6 primary db master: db1131

The read only window request is at: T251153

Event Timeline

Restricted Application added subscribers: Petar.petkovic, Aklapper. · View Herald TranscriptApr 27 2020, 3:12 PM
Marostegui triaged this task as Medium priority.Apr 27 2020, 3:12 PM
Marostegui moved this task from Triage to Next on the DBA board.
Marostegui updated the task description. (Show Details)Apr 27 2020, 3:20 PM
Marostegui renamed this task from Upgrade and restart s5 and s6 primary DB master to Upgrade and restart s5 and s6 primary DB master: Tue 5th May.Apr 30 2020, 9:10 AM
Marostegui added a comment.EditedApr 30 2020, 9:21 AM

Day before:

  • Install the 10.1.43-2 package on both masters

Maintenance day:

  • Silence all hosts in s5 and s6
  • Set read only on s5 and s6:
dbctl --scope eqiad section s5 ro "Maintenance on s5 and s6 T251154" && dbctl --scope eqiad section s6 ro "Maintenance on s5 and s6 T251154"  && dbctl config commit -m "Set s5 and s6 as read-only for maintenance T251154"
  • Confirm read only on the wikis
  • Set read-only on the masters on mysql: db1100 db1131
  • Restart mysql on both hosts db1100 db1131
  • Confirm the slaves are connected
  • Remove read only:
dbctl --scope eqiad section s5 rw  && dbctl --scope eqiad section s6 rw && dbctl config commit -m "Set s5 and s6 as read-only=off for maintenance T251154"
  • Confirm writes can go through
  • Run mysql_upgrade db1100 db1131
  • Close task

Added the slot on the deployment's page

Marostegui moved this task from Next to In progress on the DBA board.May 4 2020, 7:24 AM

Mentioned in SAL (#wikimedia-operations) [2020-05-04T07:24:50Z] <marostegui> Install 10.1.43-2 on s5 (db110) and s6 (db1131) masters in preparations for tomorrow's restart - T251154

10.1.43-2 has been installed on both masters (without mysql_upgrade) and they are ready for tomorrow's restart.

Mentioned in SAL (#wikimedia-operations) [2020-05-05T05:19:06Z] <marostegui> Start s5 and s6 maintenance - T251154

Mentioned in SAL (#wikimedia-operations) [2020-05-05T05:20:59Z] <marostegui@cumin1001> dbctl commit (dc=all): 'Set s5 and s6 as read-only for maintenance T251154', diff saved to https://phabricator.wikimedia.org/P11132 and previous config saved to /var/cache/conftool/dbconfig/20200505-052058-marostegui.json

Mentioned in SAL (#wikimedia-operations) [2020-05-05T05:23:34Z] <marostegui@cumin1001> dbctl commit (dc=all): 'Set s5 and s6 as read-only=off for maintenance T251154', diff saved to https://phabricator.wikimedia.org/P11133 and previous config saved to /var/cache/conftool/dbconfig/20200505-052334-marostegui.json

Marostegui closed this task as Resolved.May 5 2020, 5:29 AM

This was done.
We started a bit later than expected due to some on-going issues with another service.
RO started: 05:20:59
RO finished: 05:23:34