Page MenuHomePhabricator

Upgrade and restart s4 (commonswiki) primary database master: Tue 12th May
Closed, ResolvedPublic

Description

We need to upgrade and restart s4 primary database master as part of: T239791: DB: perform rolling restart of mariadb daemons to pick up CA changes

The following wikis will go read-only:

s4:

commonswiki

Impact: That wikis will go read-only for around 15 minutes (hopefully a lot less). Reads will not be impacted
When: Tue 12th May at 05:00 AM UTC

s4 primary db master: db1138

The read only window request is at: T251501

Event Timeline

Restricted Application added a subscriber: Aklapper. · View Herald TranscriptApr 30 2020, 9:15 AM
Marostegui triaged this task as Medium priority.Apr 30 2020, 9:15 AM
Marostegui updated the task description. (Show Details)
Marostegui moved this task from Triage to Next on the DBA board.
Marostegui moved this task from Next to In progress on the DBA board.May 11 2020, 1:19 PM

Mentioned in SAL (#wikimedia-operations) [2020-05-11T13:20:32Z] <marostegui> Upgrade mysql package on s4 master in preparation for tomorrow's maintenance T251502

Package has been upgraded on db1138

Maintenance day:

  • Silence all hosts in s4
  • Set read only on s4:
dbctl --scope eqiad section s4 ro "Maintenance on s4 T251502" && dbctl config commit -m "Set s4 as read-only for maintenance T251502"
  • Confirm read only on commons
  • Set read-only on the master on mysql: db1138
  • Restart mysql on db1138
  • Run puppet
  • Confirm the slaves are connected
  • Remove read only:
dbctl --scope eqiad section s4 rw && dbctl config commit -m "Set s4 as read-only=off for maintenance T251502"
  • Confirm writes can go through
  • Run mysql_upgrade db1138
  • Close task

Window reserved on the Deployment's calendar

Mentioned in SAL (#wikimedia-operations) [2020-05-12T05:00:54Z] <marostegui@cumin1001> dbctl commit (dc=all): 'Set s4 as read-only for maintenance T251502', diff saved to https://phabricator.wikimedia.org/P11179 and previous config saved to /var/cache/conftool/dbconfig/20200512-050054-marostegui.json

Mentioned in SAL (#wikimedia-operations) [2020-05-12T05:03:40Z] <marostegui@cumin1001> dbctl commit (dc=all): 'Set s4 as read-only=off for maintenance T251502', diff saved to https://phabricator.wikimedia.org/P11180 and previous config saved to /var/cache/conftool/dbconfig/20200512-050339-marostegui.json

Marostegui closed this task as Resolved.May 12 2020, 5:06 AM

This has been done.
RO starts: 05:00:54
RO stops: 05:03:40