Upgrade and restart s5 and s6 primary DB master: Tue 5th May
Closed, ResolvedPublic
Actions

Assigned To

Authored By

	Marostegui
	Apr 27 2020, 3:12 PM

Description

We need to upgrade and restart s5 and s6 primary database master as part of T239791: DB: perform rolling restart of mariadb daemons to pick up CA changes

The following wikis will go read-only:

s5:

cebwiki
dewiki
enwikivoyage
mgwiktionary
shwiki
srwiki

s6:

frwiki
jawiki
ruwiki

Impact: Those wikis will go read-only for around 15 minutes (hopefully a lot less). Reads will not be impacted
When: Tuesday 5th May at 05:00 AM UTC

s5 primary db master: db1100
s6 primary db master: db1131

The read only window request is at: T251153

Related Objects
Search...

Status	Assigned	Task
Resolved	jbond	T236277 Extend Puppet CA Expiry date
Resolved	jbond	T237259 Document all uses of the puppetCA certificate
Resolved	RLazarus	T237362 Rolling restart of etcd to pick up the renewed CA public certificate.
Resolved	Marostegui	T239791 DB: perform rolling restart of mariadb daemons to pick up CA changes
Resolved	Marostegui	T251154 Upgrade and restart s5 and s6 primary DB master: Tue 5th May
Resolved	Trizek-WMF	T251153 Read only time window needed for wikis on s5 and s6
Resolved	jbond	T240330 Check if traffic servers need restarting/reloading post CA change

Event Timeline

Marostegui created this task.Apr 27 2020, 3:12 PM

Restricted Application added subscribers: • Petar.petkovic, Aklapper. · View Herald TranscriptApr 27 2020, 3:12 PM

Marostegui triaged this task as Medium priority.Apr 27 2020, 3:12 PM

Marostegui added a subtask: T251153: Read only time window needed for wikis on s5 and s6.

Marostegui moved this task from Triage to Pending comment on the DBA board.

Marostegui updated the task description. (Show Details)Apr 27 2020, 3:20 PM

Marostegui renamed this task from Upgrade and restart s5 and s6 primary DB master to Upgrade and restart s5 and s6 primary DB master: Tue 5th May.Apr 30 2020, 9:10 AM

Day before:

Install the 10.1.43-2 package on both masters

Maintenance day:

Silence all hosts in s5 and s6
Set read only on s5 and s6:

dbctl --scope eqiad section s5 ro "Maintenance on s5 and s6 T251154" && dbctl --scope eqiad section s6 ro "Maintenance on s5 and s6 T251154"  && dbctl config commit -m "Set s5 and s6 as read-only for maintenance T251154"

Confirm read only on the wikis
Set read-only on the masters on mysql: db1100 db1131
Restart mysql on both hosts db1100 db1131
Confirm the slaves are connected
Remove read only:

dbctl --scope eqiad section s5 rw  && dbctl --scope eqiad section s6 rw && dbctl config commit -m "Set s5 and s6 as read-only=off for maintenance T251154"

Confirm writes can go through
Run mysql_upgrade db1100 db1131
Close task

Added the slot on the deployment's page

Agusbou2015 subscribed.May 4 2020, 12:27 AM

Marostegui moved this task from Pending comment to In progress on the DBA board.May 4 2020, 7:24 AM

Mentioned in SAL (#wikimedia-operations) [2020-05-04T07:24:50Z] <marostegui> Install 10.1.43-2 on s5 (db110) and s6 (db1131) masters in preparations for tomorrow's restart - T251154

10.1.43-2 has been installed on both masters (without mysql_upgrade) and they are ready for tomorrow's restart.

Mentioned in SAL (#wikimedia-operations) [2020-05-05T05:19:06Z] <marostegui> Start s5 and s6 maintenance - T251154

Mentioned in SAL (#wikimedia-operations) [2020-05-05T05:20:59Z] <marostegui@cumin1001> dbctl commit (dc=all): 'Set s5 and s6 as read-only for maintenance T251154', diff saved to https://phabricator.wikimedia.org/P11132 and previous config saved to /var/cache/conftool/dbconfig/20200505-052058-marostegui.json

Mentioned in SAL (#wikimedia-operations) [2020-05-05T05:23:34Z] <marostegui@cumin1001> dbctl commit (dc=all): 'Set s5 and s6 as read-only=off for maintenance T251154', diff saved to https://phabricator.wikimedia.org/P11133 and previous config saved to /var/cache/conftool/dbconfig/20200505-052334-marostegui.json

This was done.
We started a bit later than expected due to some on-going issues with another service.
RO started: 05:20:59
RO finished: 05:23:34

Marostegui closed subtask T251153: Read only time window needed for wikis on s5 and s6 as Resolved.May 5 2020, 5:30 AM

Upgrade and restart s5 and s6 primary DB master: Tue 5th MayClosed, ResolvedPublicActions

Description

Related ObjectsSearch...

Event Timeline

Upgrade and restart s5 and s6 primary DB master: Tue 5th May
Closed, ResolvedPublic
Actions

Related Objects
Search...