Page MenuHomePhabricator

Upgrade and restart s1 (enwiki) primary database master: Thu 21th May
Closed, ResolvedPublic

Description

We need to upgrade and restart s1 (enwiki) primary database masters as part of: T239791

The following wikis will go read-only:

enwiki

Impact: enwiki will go read-only for around 30 minutes (hopefully a lot less). Reads will not be impacted
When: Thu 21th May at 05:00 AM UTC

s1 primary db master: db1083

The read only window request is at: T251985

Event Timeline

Restricted Application added a subscriber: Aklapper. · View Herald TranscriptWed, May 6, 7:34 AM
Marostegui triaged this task as Medium priority.Wed, May 6, 7:35 AM
Marostegui moved this task from Triage to Next on the DBA board.
Marostegui moved this task from Next to In progress on the DBA board.Wed, May 20, 9:25 AM

Mentioned in SAL (#wikimedia-operations) [2020-05-20T09:26:16Z] <marostegui> Upgrade db1083 (s1 master) to 10.1.43-2 without restarting T251982

Package upgraded on db1083 to 10.1.43-2 in preparation for tomorrow's maintenance.

Maintenance day:

  • Silence all hosts in s1
  • Set read only on s1:
dbctl --scope eqiad section s1 ro "Maintenance on enwiki T251982" && dbctl config commit -m "Set enwiki as read-only for maintenance T251982"
  • Confirm read only on enwiki
  • Set read-only on the master on mysql: db1083
  • Restart mysql on db1083
  • Run puppet
  • Confirm the slaves are connected
  • Remove read only:
dbctl --scope eqiad section s1 rw && dbctl config commit -m "Set enwiki as read-only=off after maintenance T251982"
  • Confirm writes can go through
  • Run mysql_upgrade db1083
  • Close task
Marostegui updated the task description. (Show Details)Wed, May 20, 10:18 AM

Mentioned in SAL (#wikimedia-operations) [2020-05-21T05:00:30Z] <marostegui@cumin1001> dbctl commit (dc=all): 'Set enwiki as read-only for maintenance T251982', diff saved to https://phabricator.wikimedia.org/P11257 and previous config saved to /var/cache/conftool/dbconfig/20200521-050029-marostegui.json

Mentioned in SAL (#wikimedia-operations) [2020-05-21T05:03:28Z] <marostegui@cumin1001> dbctl commit (dc=all): 'Set enwiki as read-only=off after maintenance T251982', diff saved to https://phabricator.wikimedia.org/P11258 and previous config saved to /var/cache/conftool/dbconfig/20200521-050328-marostegui.json

Marostegui closed this task as Resolved.Thu, May 21, 5:06 AM

This was done.
RO starts: 05:00:30
RO stops: 05:03:28