Page MenuHomePhabricator
Create Task
Maniphest T434043

Switchover m1 master (db1213 -> db1164)
Closed, ResolvedPublic
Actions

Description

db1213 needs to be upgraded

Impact: Read only for a few seconds on the services below:

Services running on m1:

  • bacula
  • cas (and cas staging)
  • backups
  • etherpad
  • librenms
  • pki
  • rt

Switchover steps:

OLD MASTER: db1213

NEW MASTER: db1164

  • Check configuration differences between new and old master pt-config-diff h=db1164.eqiad.wmnet,F=/root/.my.cnf h=db1213.eqiad.wmnet,F=/root/.my.cnf
  • Enable notifications on db1164
  • Silence alerts on all hosts: sudo cookbook sre.hosts.downtime --hours 1 -r "Primary switchover m1 T434043" 'A:db-section-m1'
  • Topology changes: move everything under db1164

sudo db-switchover --timeout=1 --only-slave-move db1213.eqiad.wmnet db1164.eqiad.wmnet

run-puppet-agent && cat /etc/haproxy/conf.d/db-master.cfg

  • Start the failover

!log Failover m1 from db1213 to db1164 - T434043

root@cumin1001:~/wmfmariadbpy/wmfmariadbpy# sudo db-switchover --skip-slave-move db1213 db1164
  • Reload haproxies
dbproxy1022:   systemctl reload haproxy && echo "show stat" | socat /run/haproxy/haproxy.sock stdio
dbproxy1024:   systemctl reload haproxy && echo "show stat" | socat /run/haproxy/haproxy.sock stdio
  • kill connections on the old master (db1213)

pt-kill --print --kill --victims all --match-all F=/dev/null,S=/run/mysqld/mysqld.sock

  • Restart puppet on old and new masters (for heartbeat):db1213 and db1164

sudo cumin 'db1164* or db1213*' 'run-puppet-agent -e "primary switchover T434043"'

  • Check services affected (librenms, racktables, etherpad...)
  • Clean orchestrator heartbeat to remove the old masters' one: sudo db-mysql db1164 heartbeat -e "delete from heartbeat where file like 'db1213%';"
  • Merge backup ticket: https://gerrit.wikimedia.org/r/c/operations/puppet/+/1321468
  • Update/resolve phabricator ticket about failover

Details

Related Changes in Gerrit:
SubjectAuthorRepoBranchLines +/-
mariadb: Switchover backup statistics primary db for dbbackupsJcrespooperations/puppetproduction+3 -3
mariadb: Promote db1164 to m1 masterMarosteguioperations/puppetproduction+10 -10
db1164: Enable notificationsMarosteguioperations/puppetproduction+1 -2
m1 proxy: Test db1164 connectivityMarosteguioperations/puppetproduction+4 -4
Customize query in gerrit

Related Objects
Search...

Event Timeline

Marostegui created this task.Wed, Aug 5, 6:55 AM
Marostegui triaged this task as Medium priority.
Marostegui updated the task description. (Show Details)
Marostegui added a subscriber: jcrespo.Wed, Aug 5, 7:00 AM

@jcrespo could you let me know where it would be a BAD moment to do this switchover from a backups point of view so I can avoid that window?
Thanks

Marostegui updated the task description. (Show Details)Wed, Aug 5, 7:03 AM
gerritbot added a comment.Wed, Aug 5, 7:07 AM

Change #1321463 had a related patch set uploaded (by Marostegui; author: Marostegui):

[operations/puppet@production] m1 proxy: Test db1164 connectivity

https://gerrit.wikimedia.org/r/1321463

gerritbot added a project: Patch-For-Review.Wed, Aug 5, 7:07 AM
Marostegui updated the task description. (Show Details)Wed, Aug 5, 7:09 AM
gerritbot added a comment.Wed, Aug 5, 7:09 AM

Change #1321463 merged by Marostegui:

[operations/puppet@production] m1 proxy: Test db1164 connectivity

https://gerrit.wikimedia.org/r/1321463

Marostegui added a comment.Wed, Aug 5, 7:13 AM

Connectivity between proxies and db1164 has been tested.

jcrespo added a comment.Wed, Aug 5, 7:29 AM
In T434043#12185981, @Marostegui wrote:

@jcrespo could you let me know where it would be a BAD moment to do this switchover from a backups point of view so I can avoid that window?

Backups usually run from 20h UTC until 11am in the worst case scenario (normally, until 6 am).

Marostegui added a comment.Wed, Aug 5, 7:30 AM

Ok thanks! I will adjust

Maintenance_bot removed a project: Patch-For-Review.Wed, Aug 5, 7:30 AM
gerritbot added a comment.Wed, Aug 5, 7:32 AM

Change #1321468 had a related patch set uploaded (by Jcrespo; author: Jcrespo):

[operations/puppet@production] mariadb: Switchover backup statistics for backup services

https://gerrit.wikimedia.org/r/1321468

gerritbot added a project: Patch-For-Review.Wed, Aug 5, 7:33 AM
jcrespo updated the task description. (Show Details)Wed, Aug 5, 7:33 AM
Marostegui added a comment.Thu, Aug 6, 7:23 AM

@jcrespo I will schedule this for Tuesday in the end, I'll do it after checking the backups table to makes sure nothing is in progress. The RO time would be around 5-10 seconds and I will merge your patch inmediately after.

jcrespo added a comment.Thu, Aug 6, 7:24 AM

Thank you.

Marostegui moved this task from Triage to In progress on the DBA board.Fri, Aug 7, 10:25 AM
gerritbot added a comment.Tue, Aug 11, 7:09 AM

Change #1324207 had a related patch set uploaded (by Marostegui; author: Marostegui):

[operations/puppet@production] db1164: Enable notifications

https://gerrit.wikimedia.org/r/1324207

gerritbot added a comment.Tue, Aug 11, 7:09 AM

Change #1324207 merged by Marostegui:

[operations/puppet@production] db1164: Enable notifications

https://gerrit.wikimedia.org/r/1324207

Marostegui updated the task description. (Show Details)Tue, Aug 11, 7:10 AM
gerritbot added a comment.Tue, Aug 11, 7:14 AM

Change #1324209 had a related patch set uploaded (by Marostegui; author: Marostegui):

[operations/puppet@production] mariadb: Promote db1164 to m1 master

https://gerrit.wikimedia.org/r/1324209

Marostegui updated the task description. (Show Details)Tue, Aug 11, 7:14 AM
Stashbot added a comment.Tue, Aug 11, 8:25 AM

Mentioned in SAL (#wikimedia-operations) [2026-08-11T08:25:04Z] <marostegui@cumin1003> DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:00:00 on db[2160,2232].codfw.wmnet,db[1164,1213,1217].eqiad.wmnet with reason: Primary switchover m1 T434043

gerritbot added a comment.Tue, Aug 11, 8:27 AM

Change #1324209 merged by Marostegui:

[operations/puppet@production] mariadb: Promote db1164 to m1 master

https://gerrit.wikimedia.org/r/1324209

Marostegui updated the task description. (Show Details)Tue, Aug 11, 8:29 AM
Stashbot added a comment.Tue, Aug 11, 8:29 AM

Mentioned in SAL (#wikimedia-operations) [2026-08-11T08:29:58Z] <marostegui> Failover m1 from db1213 to db1164 - T434043

gerritbot added a comment.Tue, Aug 11, 8:31 AM

Change #1321468 merged by Marostegui:

[operations/puppet@production] mariadb: Switchover backup statistics primary db for dbbackups

https://gerrit.wikimedia.org/r/1321468

Maintenance_bot removed a project: Patch-For-Review.Tue, Aug 11, 8:31 AM
Marostegui closed this task as Resolved.Tue, Aug 11, 8:32 AM
Marostegui updated the task description. (Show Details)

This was done

Maintenance_bot moved this task from In progress to Done on the DBA board.Tue, Aug 11, 9:29 AM
Content licensed under Creative Commons Attribution-ShareAlike (CC BY-SA) 4.0 unless otherwise noted; code licensed under GNU General Public License (GPL) 2.0 or later and other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL · Credits