Upgrade and restart s3 and s7 primary DB master: Thu 7th May
Closed, ResolvedPublic
Actions

Assigned To

Authored By

	• Marostegui
	Apr 27 2020, 3:19 PM

Description

We need to upgrade and restart s3 and s7 primary database master as part of T239791: DB: perform rolling restart of mariadb daemons to pick up CA changes

The following wikis will go read-only:

s3: https://noc.wikimedia.org/conf/highlight.php?file=dblists/s3.dblist

s7:

arwiki
cawiki
eswiki
fawiki
frwiktionary
hewiki
huwiki
kowiki
metawiki
rowiki
ukwiki
viwiki

s7 also includes centralauth database, which means some operations might fail during the read-only period, such as: GlobalRenames, Changing/Confirming emails, logging into new wikis, password changes...

Impact: Those wikis will go read-only for around 15 minutes (hopefully a lot less). Reads will not be impacted
When: Thu 7th May at 05:00 AM UTC

s3 primary db master: db1123
s7 primary db master: db1086

The read only window request is at: T251157

Related Objects
Search...

Status	Assigned	Task
Resolved	jbond	T236277 Extend Puppet CA Expiry date
Resolved	jbond	T237259 Document all uses of the puppetCA certificate
Resolved	RLazarus	T237362 Rolling restart of etcd to pick up the renewed CA public certificate.
Resolved	• Marostegui	T239791 DB: perform rolling restart of mariadb daemons to pick up CA changes
Resolved	• Marostegui	T251158 Upgrade and restart s3 and s7 primary DB master: Thu 7th May
Resolved	Trizek-WMF	T251157 Read only time window needed for wikis on s3 and s7
Resolved	jbond	T240330 Check if traffic servers need restarting/reloading post CA change

Event Timeline

• Marostegui created this task.Apr 27 2020, 3:19 PM

Restricted Application added subscribers: Strainu, Base, revi, Aklapper. · View Herald TranscriptApr 27 2020, 3:19 PM

• Marostegui triaged this task as Medium priority.Apr 27 2020, 3:19 PM

• Marostegui moved this task from Triage to Pending comment on the DBA board.

jcrespo added a subscriber: jcrespo.Apr 27 2020, 3:55 PM

• Marostegui renamed this task from Upgrade and restart s3 and s7 primary DB master to Upgrade and restart s3 and s7 primary DB master: Thu 7th May.Apr 30 2020, 9:10 AM

RhinosF1 added a subscriber: RhinosF1.Apr 30 2020, 9:11 AM

Added the slot on the deployment's page

Day before:

Install the 10.1.43-2 package on both masters (db1123 and db1086)

Maintenance day:

Silence all hosts in s3 and s7
Set read only on s3 and s7:

dbctl --scope eqiad section s3 ro "Maintenance on s3 and s7 T251158" && dbctl --scope eqiad section s7 ro "Maintenance on s3 and s7 T251158"  && dbctl config commit -m "Set s3 and s7 as read-only for maintenance T251158"

Confirm read only on the wikis
Set read-only on the masters on mysql: db1123 db1086
Restart mysql on both hosts: db1123 db1086
Run puppet
Confirm the slaves are connected
Remove read only:

dbctl --scope eqiad section s3 rw  && dbctl --scope eqiad section s7 rw && dbctl config commit -m "Set s3 and s7 as read-only=off for maintenance T251158"

Confirm writes can go through
Run mysql_upgrade on both hosts: db1123 db1086
Close task

Agusbou2015 added a subscriber: Agusbou2015.May 4 2020, 12:27 AM

• Marostegui moved this task from Pending comment to In progress on the DBA board.May 5 2020, 5:31 AM

Mentioned in SAL (#wikimedia-operations) [2020-05-06T09:12:10Z] <marostegui> Upgrade package on s3 and s7 master (db1123 and db1086) in preparation for tomorrow's restart - T251158

Package upgraded on db1123 and db1086.

Hello! can you please allow the register users dismiss the notice, once they read it? Thanks!

In T251158#6115189, @Ovedc wrote:

Hello! can you please allow the register users dismiss the notice, once they read it? Thanks!

Thanks for the message. However we (DBAs) cannot implement that, you might want to comment on T251157 and let them know if that's possible for next iterations.

Thank you!

Mentioned in SAL (#wikimedia-operations) [2020-05-07T05:00:47Z] <marostegui@cumin1001> dbctl commit (dc=all): 'Set s3 and s7 as read-only for maintenance T251158', diff saved to https://phabricator.wikimedia.org/P11166 and previous config saved to /var/cache/conftool/dbconfig/20200507-050046-marostegui.json

Mentioned in SAL (#wikimedia-operations) [2020-05-07T05:04:19Z] <marostegui@cumin1001> dbctl commit (dc=all): 'Set s3 and s7 as read-only=off for maintenance T251158', diff saved to https://phabricator.wikimedia.org/P11167 and previous config saved to /var/cache/conftool/dbconfig/20200507-050419-marostegui.json

This is done.
RO started: 05:00:47
RO finished: 05:04:19

• Marostegui added a subtask: T251157: Read only time window needed for wikis on s3 and s7.May 7 2020, 5:17 AM

Agusbou2015 closed subtask T251157: Read only time window needed for wikis on s3 and s7 as Resolved.May 7 2020, 2:48 PM

Upgrade and restart s3 and s7 primary DB master: Thu 7th MayClosed, ResolvedPublicActions

Description

Related ObjectsSearch...

Event Timeline

Upgrade and restart s3 and s7 primary DB master: Thu 7th May
Closed, ResolvedPublic
Actions

Related Objects
Search...