Page MenuHomePhabricator
Create Task
Maniphest T331510

Switchover m1 master (db1164 -> db1101)
Closed, ResolvedPublic
Actions

Description

db1164 is in row B and there will be a switch maintenance with hard downtime

When: Monday 27st March at 08:00 AM UTC
Impact: Read only for a few seconds on the services below:

Services running on m1:

  • bacula
  • cas (and cas staging)
  • backups
  • etherpad
  • librenms
  • pki
  • rt

Switchover steps:

OLD MASTER: db1164

NEW MASTER: db1101

  • Check configuration differences between new and old master pt-config-diff h=db1164.eqiad.wmnet,F=/root/.my.cnf h=db1101.eqiad.wmnet,F=/root/.my.cnf
  • Silence alerts on all hosts
  • Topology changes: move everything under db1101

db-switchover --timeout=1 --only-slave-move db1164.eqiad.wmnet db1101.eqiad.wmnet

run-puppet-agent && cat /etc/haproxy/conf.d/db-master.cfg

  • Start the failover

!log Failover m1 from db1164 to db1101 - T331510

root@cumin1001:~/wmfmariadbpy/wmfmariadbpy# db-switchover --skip-slave-move db1164 db1101
  • Reload haproxies
dbproxy1012:   systemctl reload haproxy && echo "show stat" | socat /run/haproxy/haproxy.sock stdio
dbproxy1014:   systemctl reload haproxy && echo "show stat" | socat /run/haproxy/haproxy.sock stdio
  • kill connections on the old master (db1164)

pt-kill --print --kill --victims all --match-all F=/dev/null,S=/run/mysqld/mysqld.sock

  • Restart puppet on old and new masters (for heartbeat):db1101 and db1164

sudo cumin 'db1164* or db1101*' 'run-puppet-agent -e "primary switchover T331510"'

Details

SubjectRepoBranchLines +/-
backups: Replace db1164 with db1101operations/puppetproduction+2 -2
db1101: Make it masteroperations/puppetproduction+1 -0
mariadb: Promote db1101 to m1 masteroperations/puppetproduction+12 -10
dbproxy1012,dbproxy1014: Test db1101 on proxiesoperations/puppetproduction+4 -4
Customize query in gerrit

Related Objects
Search...

Event Timeline

Marostegui created this task.Mar 8 2023, 8:57 AM
Restricted Application added a subscriber: Aklapper. · View Herald TranscriptMar 8 2023, 8:57 AM
Marostegui claimed this task.Mar 8 2023, 8:58 AM
Marostegui triaged this task as Medium priority.
Marostegui added a project: DBA.
Marostegui updated the task description. (Show Details)
Marostegui mentioned this in T331381: decommission db1101.eqiad.wmnet.
Marostegui mentioned this in T330165: eqiad row B switches upgrade.
Marostegui added a parent task: T330165: eqiad row B switches upgrade.
Marostegui mentioned this in T330977: Move db1183 to m1.Mar 8 2023, 9:01 AM
Marostegui added a subtask: T331511: Move db1101 to m1.Mar 8 2023, 9:06 AM
Marostegui moved this task from Triage to In progress on the DBA board.Mar 8 2023, 9:17 AM
Marostegui closed subtask T331511: Move db1101 to m1 as Resolved.Mar 8 2023, 11:27 AM
Marostegui mentioned this in T331511: Move db1101 to m1.
Marostegui updated the task description. (Show Details)

When: Tuesday 14th March at 07:00 AM UTC

Marostegui updated the task description. (Show Details)Mar 8 2023, 11:28 AM
Marostegui added a subscriber: jcrespo.Mar 13 2023, 6:10 AM

Given that the switch maintenance has been moved to 28th, I am going to move this switchover to the 21st, so @jcrespo is around to handle bacula database (which lives on m1)

Marostegui updated the task description. (Show Details)Mar 13 2023, 6:10 AM
Marostegui updated the task description. (Show Details)
Marostegui mentioned this in T331882: eqiad row C switches upgrade.Mar 15 2023, 1:58 PM
Marostegui updated the task description. (Show Details)
Marostegui added a comment.Mar 15 2023, 2:00 PM

Once this ticket is done, the following day we need to create a "revert" to go back to db1164 as db1101 would be affected by row C maintenance T331882 the following week,

Marostegui updated the task description. (Show Details)Mar 15 2023, 2:00 PM
Marostegui updated the task description. (Show Details)Mar 20 2023, 2:14 PM

When: Monday 27st March at 09:00 AM UTC

gerritbot added a comment.Mar 24 2023, 7:46 AM

Change 902572 had a related patch set uploaded (by Marostegui; author: Marostegui):

[operations/puppet@production] mariadb: Promote db1101 to m1 master

https://gerrit.wikimedia.org/r/902572

gerritbot added a project: Patch-For-Review.Mar 24 2023, 7:46 AM
Marostegui updated the task description. (Show Details)Mar 24 2023, 7:47 AM
gerritbot added a comment.Mar 24 2023, 7:52 AM

Change 902574 had a related patch set uploaded (by Marostegui; author: Marostegui):

[operations/puppet@production] dbproxy1012,dbproxy1014: Test db1101 on proxies

https://gerrit.wikimedia.org/r/902574

gerritbot added a comment.Mar 24 2023, 7:52 AM

Change 902574 merged by Marostegui:

[operations/puppet@production] dbproxy1012,dbproxy1014: Test db1101 on proxies

https://gerrit.wikimedia.org/r/902574

Marostegui added a comment.Mar 24 2023, 7:55 AM

via https://gerrit.wikimedia.org/r/c/operations/puppet/+/902574 I have tested that db1101 is fine at haproxy layer.

Stashbot added a comment.Mar 27 2023, 5:14 AM

Mentioned in SAL (#wikimedia-operations) [2023-03-27T05:14:21Z] <marostegui@cumin1001> START - Cookbook sre.hosts.downtime for 1:00:00 on db[2132,2160].codfw.wmnet,db[1101,1117,1164].eqiad.wmnet with reason: m1 master switch T331510

Stashbot added a comment.Mar 27 2023, 5:14 AM

Mentioned in SAL (#wikimedia-operations) [2023-03-27T05:14:37Z] <marostegui@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:00:00 on db[2132,2160].codfw.wmnet,db[1101,1117,1164].eqiad.wmnet with reason: m1 master switch T331510

Marostegui updated the task description. (Show Details)Mar 27 2023, 5:16 AM
Marostegui updated the task description. (Show Details)

Topology changes made.

Marostegui updated the task description. (Show Details)Mar 27 2023, 7:08 AM
gerritbot added a comment.Mar 27 2023, 7:09 AM

Change 903175 had a related patch set uploaded (by Marostegui; author: Marostegui):

[operations/puppet@production] backups: Replace db1164 with db1101

https://gerrit.wikimedia.org/r/903175

Marostegui updated the task description. (Show Details)Mar 27 2023, 7:09 AM
gerritbot added a comment.Mar 27 2023, 7:32 AM

Change 902572 merged by Marostegui:

[operations/puppet@production] mariadb: Promote db1101 to m1 master

https://gerrit.wikimedia.org/r/902572

Marostegui updated the task description. (Show Details)Mar 27 2023, 7:33 AM
Marostegui updated the task description. (Show Details)
Stashbot added a comment.Mar 27 2023, 7:39 AM

Mentioned in SAL (#wikimedia-operations) [2023-03-27T07:39:50Z] <jynus> disabling puppet and shutding down bacula at backup1001 T331510

Stashbot added a comment.Mar 27 2023, 8:03 AM

Mentioned in SAL (#wikimedia-operations) [2023-03-27T08:03:43Z] <marostegui> Failover m1 from db1164 to db1101 - T331510

gerritbot added a comment.Mar 27 2023, 8:04 AM

Change 903175 merged by Marostegui:

[operations/puppet@production] backups: Replace db1164 with db1101

https://gerrit.wikimedia.org/r/903175

Marostegui updated the task description. (Show Details)Mar 27 2023, 8:05 AM

Switchover is done

gerritbot added a comment.Mar 27 2023, 8:09 AM

Change 903181 had a related patch set uploaded (by Marostegui; author: Marostegui):

[operations/puppet@production] db1101: Make it master

https://gerrit.wikimedia.org/r/903181

gerritbot added a comment.Mar 27 2023, 8:09 AM

Change 903181 merged by Marostegui:

[operations/puppet@production] db1101: Make it master

https://gerrit.wikimedia.org/r/903181

Maintenance_bot removed a project: Patch-For-Review.Mar 27 2023, 8:11 AM
Marostegui mentioned this in T333123: Switchover m1 master (db1101 -> db1164).Mar 27 2023, 8:15 AM
Marostegui updated the task description. (Show Details)
Marostegui closed this task as Resolved.Mar 27 2023, 8:20 AM
Stashbot added a comment.Mar 27 2023, 8:28 AM

Mentioned in SAL (#wikimedia-operations) [2023-03-27T08:28:14Z] <jynus> restarting bacula at backup1001 T331510

Maintenance_bot moved this task from In progress to Done on the DBA board.Mar 27 2023, 8:29 AM
Content licensed under Creative Commons Attribution-ShareAlike (CC BY-SA) 4.0 unless otherwise noted; code licensed under GNU General Public License (GPL) 2.0 or later and other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL