Page MenuHomePhabricator
Create Task
Maniphest T120122

Perform a rolling restart of all MySQL slaves (masters too for those services with low traffic)
Closed, ResolvedPublic
Actions

Description

Ideally to apply all at once:

  1. Ugrade to jessie
  2. Upgrade to latest mariadb package
  3. enabling ssl
  4. enabling ferm
  5. enabling performance_schema
  6. setting ROW-based replication on codfw

some of these may not be feasible due to resources (reinstalling 150 machines short-term) or have hard blockers- ROW cannot be enabled on eqiad due to labs (maybe nowhere, if we are going to failover soon), performance schema has not been tested yet properly on the busiest servers and we cannot upgrade some API servers. ROW based replication is not a hard blocker for the restart, although it has to be done very carefully.

Details

SubjectRepoBranchLines +/-
Set db1031 as the local eqiad master and set it to ROW binlogoperations/puppetproduction+11 -7
Repool db1021 and db1024, both with low/non critical loadoperations/mediawiki-configmaster+6 -6
Repool of db1022 after maintenanceoperations/mediawiki-configmaster+1 -1
Repool db1022 as regular traffic + APIoperations/mediawiki-configmaster+4 -1
Depool of db1022 for maintenanceoperations/mediawiki-configmaster+1 -1
s2: bye bye coredb; hello mariadb 10 with jessieoperations/puppetproduction+1 -8
Installing jessie on db1024operations/puppetproduction+2 -0
Customize query in gerrit

Related Objects
Search...

View Standalone Graph
This task is connected to more than 200 other tasks. Only direct parents and subtasks are shown here. Use View Standalone Graph to show more of the graph.
StatusSubtypeAssignedTask
· · ·
ResolvedjcrespoT111654 Set up TLS for MariaDB replication
ResolvedjcrespoT105135 Implement mariadb 10.0 masters
ResolvedjcrespoT99485 implement performance_schema for mysql monitoring
ResolvedMoritzMuehlenhoffT104699 Firewall configurations for database hosts
ResolvedNoneT109179 Migrate MySQLs to use ROW-based replication
ResolvedjcrespoT120122 Perform a rolling restart of all MySQL slaves (masters too for those services with low traffic)
Resolved• CmjohnsonT120689 es1019 and its management interface are unresponsive
· · ·

Event Timeline

jcrespo created this task.Dec 2 2015, 7:46 PM
jcrespo claimed this task.
jcrespo raised the priority of this task from to High.
jcrespo updated the task description. (Show Details)
jcrespo added projects: SRE, DBA.
jcrespo added a parent task: T105135: Implement mariadb 10.0 masters.
jcrespo added a parent task: T99485: implement performance_schema for mysql monitoring.
jcrespo added a parent task: T104699: Firewall configurations for database hosts.
jcrespo added subscribers: MoritzMuehlenhoff, Chmarkine, JanZerebecki and 8 others.
jcrespo added a parent task: T109179: Migrate MySQLs to use ROW-based replication.Dec 2 2015, 7:48 PM
jcrespo updated the task description. (Show Details)
jcrespo set Security to None.
jcrespo moved this task from Triage to Backlog on the DBA board.Dec 2 2015, 10:44 PM
jcrespo moved this task from Backlog to In progress on the DBA board.Dec 7 2015, 10:03 AM
jcrespo mentioned this in T120689: es1019 and its management interface are unresponsive.Dec 7 2015, 6:32 PM
jcrespo added a subtask: T120689: es1019 and its management interface are unresponsive.
• Cmjohnson closed subtask T120689: es1019 and its management interface are unresponsive as Resolved.Dec 8 2015, 7:47 PM
jcrespo mentioned this in T124002: Untangle wikitech/labtestwikitech and s7 DBs and networking and mysql grants.Jan 20 2016, 12:52 PM
gerritbot added a comment.Feb 11 2016, 11:16 PM

Change 270133 had a related patch set uploaded (by Jcrespo):
Installing jessie on db1024

https://gerrit.wikimedia.org/r/270133

gerritbot added a project: Patch-For-Review.Feb 11 2016, 11:16 PM
gerritbot added a comment.Feb 11 2016, 11:23 PM

Change 270133 merged by Jcrespo:
Installing jessie on db1024

https://gerrit.wikimedia.org/r/270133

gerritbot added a comment.Feb 11 2016, 11:52 PM

Change 270142 had a related patch set uploaded (by Jcrespo):
s2: bye bye coredb; hello mariadb 10 with jessie

https://gerrit.wikimedia.org/r/270142

gerritbot added a comment.Feb 11 2016, 11:54 PM

Change 270142 merged by Jcrespo:
s2: bye bye coredb; hello mariadb 10 with jessie

https://gerrit.wikimedia.org/r/270142

gerritbot added a comment.Feb 15 2016, 3:28 PM

Change 270752 had a related patch set uploaded (by Volans):
Depool of db1022 for maintenance

https://gerrit.wikimedia.org/r/270752

gerritbot added a comment.Feb 15 2016, 4:03 PM

Change 270752 merged by jenkins-bot:
Depool of db1022 for maintenance

https://gerrit.wikimedia.org/r/270752

gerritbot added a comment.Feb 16 2016, 3:47 PM

Change 270975 had a related patch set uploaded (by Volans):
Repool of db1022 after maintenance

https://gerrit.wikimedia.org/r/270975

gerritbot added a comment.Feb 17 2016, 1:57 PM

Change 271249 had a related patch set uploaded (by Jcrespo):
Repool db1022 as regular traffic API

https://gerrit.wikimedia.org/r/271249

gerritbot added a comment.Feb 17 2016, 2:00 PM

Change 271249 merged by Jcrespo:
Repool db1022 as regular traffic API

https://gerrit.wikimedia.org/r/271249

gerritbot added a comment.Feb 17 2016, 2:08 PM

Change 270975 abandoned by Jcrespo:
Repool of db1022 after maintenance

Reason:
Forgot about this, I already merged https://gerrit.wikimedia.org/r/#/c/271249/

https://gerrit.wikimedia.org/r/270975

gerritbot added a comment.Feb 25 2016, 10:29 AM

Change 273196 had a related patch set uploaded (by Jcrespo):
Repool db1021 and db1024, both with low/non critical load

https://gerrit.wikimedia.org/r/273196

gerritbot added a comment.Feb 25 2016, 11:18 AM

Change 273196 merged by Jcrespo:
Repool db1021 and db1024, both with low/non critical load

https://gerrit.wikimedia.org/r/273196

jcrespo moved this task from In progress to Backlog on the DBA board.Mar 9 2016, 11:09 AM
jcrespo mentioned this in T125028: reimage or decom db servers on precise.Apr 15 2016, 8:36 AM
jcrespo moved this task from Backlog to In progress on the DBA board.Apr 15 2016, 1:14 PM
Dzahn unsubscribed.Apr 15 2016, 7:40 PM
Volans subscribed.Apr 19 2016, 8:46 PM

When rolling restart also check the error log, if too big let's rotate it and compress/delete the old one based on size.

For the current situation of bigger error logs see T127636#2205361

Restricted Application added a subscriber: TerraCodes. · View Herald TranscriptApr 19 2016, 8:46 PM
gerritbot added a comment.Apr 20 2016, 7:57 PM

Change 284516 had a related patch set uploaded (by Jcrespo):
Set db1031 as the local eqiad master and set it to ROW binlog

https://gerrit.wikimedia.org/r/284516

gerritbot added a comment.Apr 20 2016, 8:00 PM

Change 284516 merged by Jcrespo:
Set db1031 as the local eqiad master and set it to ROW binlog

https://gerrit.wikimedia.org/r/284516

jcrespo closed this task as Resolved.Apr 22 2016, 9:58 AM

I am going to go ahead and say this is done, the rest of the servers will be done at a natural pace when upgraded to Jessie. Ferm has been applied to all eqiad core servers, including the masters and tls too (needs reviewing on its ticket). We may or may not reenable ROW on codfw.

Content licensed under Creative Commons Attribution-ShareAlike (CC BY-SA) 4.0 unless otherwise noted; code licensed under GNU General Public License (GPL) 2.0 or later and other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL · Credits