Page MenuHomePhabricator
Create Task
Maniphest T288720

Failover m5 master (db1128) to db1132 to upgrade its kernel
Closed, ResolvedPublic
Actions

Description

db1128 (m5 - master) needs its kernel upgraded, let's failover it to db1132

  • Reimage db1132
  • Move db1132 to m5 as slave

Databases on m5 (excluding labswiki, as hopefully it won't be there once this is ready to happen):

labsdbaccounts
mailman3
mailman3web
striker
test_labsdbaccounts
toolhub

When: Tuesday 23rd Nov at 14:00 UTC

Failover process

OLD MASTER: db1128

NEW MASTER: db1132

  • Decrease m5-master TTL to 1M
  • Check configuration differences between new and old master

$ pt-config-diff h=db1128.eqiad.wmnet,F=/root/.my.cnf h=db1132.eqiad.wmnet,F=/root/.my.cnf

  • Silence alerts on all hosts
  • Topology changes: move everything under db1132

db-switchover --timeout=15 --only-slave-move db1128.eqiad.wmnet db1132.eqiad.wmnet

puppet agent -tv && cat /etc/haproxy/conf.d/db-master.cfg

  • Start the failover: !log Failover m5 from db1128 to db1132 - T288720
  • DB switchover

root@cumin1001:~/wmfmariadbpy/wmfmariadbpy# db-switchover --skip-slave-move db1128 db1132

  • Reload haproxies
dbproxy1017:   systemctl reload haproxy && echo "show stat" | socat /run/haproxy/haproxy.sock stdio
dbproxy1021:   systemctl reload haproxy && echo "show stat" | socat /run/haproxy/haproxy.sock stdio
  • kill connections on the old master (db1128)

pt-kill --print --kill --victims all --match-all F=/dev/null,S=/run/mysqld/mysql.sock

Details

SubjectRepoBranchLines +/-
wmnet: Restore TTL back to 5M for m5-masteroperations/dnsmaster+1 -1
dbproxy10{17,21}: Change m5 standby hostoperations/puppetproduction+4 -4
mariadb: Promote db1132 to m5 masteroperations/puppetproduction+14 -15
db1132: Enable notificationsoperations/puppetproduction+0 -1
mariadb: Move db1132 to m5.operations/puppetproduction+7 -6
Customize query in gerrit

Related Objects
Search...

Event Timeline

There are a very large number of changes, so older changes are hidden. Show Older Changes
Restricted Application added a subscriber: Aklapper. · View Herald TranscriptAug 12 2021, 6:11 AM
• Marostegui triaged this task as Medium priority.Aug 12 2021, 6:11 AM
• Marostegui moved this task from Triage to Blocked on the DBA board.
• Marostegui mentioned this in T288197: Failover m3 (phabricator) master (db1132) to a different host to upgrade its kernel.
• Marostegui added a comment.Aug 12 2021, 6:14 AM

This switchover needs to be done AFTER we have moved wikitech out (T167973) of m5.
So I don't expect this switchover to be done before October.

• Marostegui added a project: cloud-services-team.Aug 12 2021, 6:15 AM
• Marostegui updated the task description. (Show Details)
• Marostegui added subscribers: Ladsgroup, Legoktm.
Restricted Application edited projects, added cloud-services-team (Kanban); removed cloud-services-team. · View Herald TranscriptAug 12 2021, 6:15 AM
• Marostegui added a comment.Aug 12 2021, 6:18 AM

@Legoktm @Ladsgroup cloud-services-team adding you to this tag as you have services running here that will be affected by this failover.
The failover means around 1-2 minutes of read only time (the DNS TTL for m5-master.eqiad.wmnet will be 1 minute).
Keep in mind that right now m5 doesn't use the proxies, and that's why it needs to be done via DNS change. I am in process of re-enabling back the proxies on m5 (T288093) but it cannot happen until wikitech is moved to s6, and it won't happen in between that and this anyways.

The wikitech move is scheduled for 16th of Sept, and after that I am planning to take long holidays so I expect this not to happen before October. I will try to define a date closer to the time.

Legoktm added a comment.Aug 12 2021, 6:29 AM

Sounds good. We probably want to restart Mailman so it reconnects to the new database server. I don't really know how Mailman will behave when receiving emails if the DB is read-only, we definitely have enough time to test it in Cloud beforehand. Worst case we just turn off Mailman for a minute or two.

• Marostegui added a comment.Aug 12 2021, 8:05 AM

Keep in mind that after the switch, I will kill all the connections on the old host, so it might reconnect to the new one itself. But yeah, a controlled restart doesn't hurt if it can be done.

gerritbot added a comment.Aug 13 2021, 5:07 AM

Change 712849 had a related patch set uploaded (by Marostegui; author: Marostegui):

[operations/puppet@production] mariadb: Move db1132 to m5.

https://gerrit.wikimedia.org/r/712849

gerritbot added a project: Patch-For-Review.Aug 13 2021, 5:07 AM
ops-monitoring-bot added a comment.Aug 13 2021, 5:09 AM

Script wmf-auto-reimage was launched by marostegui on cumin1001.eqiad.wmnet for hosts:

['db1132.eqiad.wmnet']

The log can be found in /var/log/wmf-auto-reimage/202108130509_marostegui_17104.log.

ops-monitoring-bot added a comment.Aug 13 2021, 5:33 AM

Completed auto-reimage of hosts:

['db1132.eqiad.wmnet']

and were ALL successful.

• Marostegui updated the task description. (Show Details)Aug 13 2021, 5:47 AM
• Marostegui updated the task description. (Show Details)Aug 13 2021, 6:27 AM

db1132 is now replicating from db1128, if all goes ok, I will change it on the haproxy config, in case db1128 goes down, we'll get a "free" swap instead of failing over to db1117:3325

gerritbot added a comment.Aug 13 2021, 7:38 AM

Change 712849 merged by Marostegui:

[operations/puppet@production] mariadb: Move db1132 to m5.

https://gerrit.wikimedia.org/r/712849

Maintenance_bot removed a project: Patch-For-Review.Aug 13 2021, 8:10 AM
gerritbot added a comment.Aug 16 2021, 4:57 AM

Change 713090 had a related patch set uploaded (by Marostegui; author: Marostegui):

[operations/puppet@production] db1132: Enable notifications

https://gerrit.wikimedia.org/r/713090

gerritbot added a project: Patch-For-Review.Aug 16 2021, 4:57 AM

Change 713090 merged by Marostegui:

[operations/puppet@production] db1132: Enable notifications

https://gerrit.wikimedia.org/r/713090

Maintenance_bot removed a project: Patch-For-Review.Aug 16 2021, 5:10 AM
• Marostegui added a comment.Aug 16 2021, 6:08 AM

Added db1132 as standby host for m5 proxies (unused for now)

Ladsgroup added a comment.Aug 16 2021, 7:45 AM

Let me know when the date and time is set so I write a quick announcement and do a restart of mailman services.

• Marostegui added a comment.Aug 16 2021, 7:49 AM

Will do! Thank you

• Marostegui added a comment.Oct 22 2021, 6:44 AM

Let's do this once T288093: Place m5 proxies in codfw and eqiad is completed, as it will be a lot easier and with less downtime.

• Marostegui added a subtask: T288093: Place m5 proxies in codfw and eqiad.Oct 27 2021, 7:43 AM
• Marostegui closed subtask T288093: Place m5 proxies in codfw and eqiad as Resolved.Nov 10 2021, 2:14 PM
• Marostegui moved this task from Blocked to Ready on the DBA board.Nov 10 2021, 2:32 PM
• Marostegui updated the task description. (Show Details)Nov 11 2021, 8:11 AM

db1132 needs restarting as it has the old my.cnf values.

Stashbot added a comment.Nov 11 2021, 8:13 AM

Mentioned in SAL (#wikimedia-operations) [2021-11-11T08:13:06Z] <marostegui> Restart db1132 T288720

Stashbot added a comment.Nov 11 2021, 8:17 AM

Mentioned in SAL (#wikimedia-operations) [2021-11-11T08:17:10Z] <marostegui> Upgrade db2078 T288720

• Marostegui mentioned this in T295524: Upgrade m5 hosts to 10.4.21.Nov 11 2021, 8:22 AM
• Marostegui updated the task description. (Show Details)Nov 11 2021, 8:26 AM
• Marostegui closed subtask T295524: Upgrade m5 hosts to 10.4.21 as Resolved.Nov 11 2021, 8:51 AM
• Marostegui added subscribers: Andrew, bd808.

@bd808 @Andrew @Legoktm @Ladsgroup now that we have the proxy in place for m5 we can proceed with this. It should only be a few seconds of read-only time. I don't know if you want/need to be present for this maintenance. I was thinking about Tuesday 23rd at 14:00 UTC?

Let me know your thoughts!
Thanks.

Ladsgroup added a comment.Nov 11 2021, 2:44 PM

I can be around for that timee

• Marostegui claimed this task.Nov 12 2021, 7:06 AM
• Marostegui moved this task from Ready to In progress on the DBA board.
bd808 added a comment.Nov 12 2021, 2:48 PM
In T288720#7497969, @Marostegui wrote:

I was thinking about Tuesday 23rd at 14:00 UTC?

This date/time works for me too.

Andrew added a comment.Nov 13 2021, 11:37 PM
In T288720#7500354, @bd808 wrote:
In T288720#7497969, @Marostegui wrote:

I was thinking about Tuesday 23rd at 14:00 UTC?

This date/time works for me too.

works for me!

Legoktm added a comment.Nov 15 2021, 7:47 PM

I'll let Amir take my spot this time, and sleep in a bit :)

• Marostegui added a comment.Nov 17 2021, 5:33 AM

Thank you all! I will get this scheduled!

• Marostegui updated the task description. (Show Details)Nov 17 2021, 5:33 AM
• Marostegui updated the task description. (Show Details)Nov 18 2021, 6:30 AM
• Marostegui updated the task description. (Show Details)
• Marostegui updated the task description. (Show Details)Nov 18 2021, 6:32 AM
• Marostegui updated the task description. (Show Details)
• Marostegui updated the task description. (Show Details)
• Marostegui updated the task description. (Show Details)
• Marostegui updated the task description. (Show Details)Nov 18 2021, 8:03 AM
Stashbot added a comment.Nov 19 2021, 6:55 AM

Mentioned in SAL (#wikimedia-operations) [2021-11-19T06:55:33Z] <marostegui> Reboot db1132 to pick up new kernel T288720

• Marostegui updated the task description. (Show Details)Nov 19 2021, 6:56 AM
• Marostegui added a comment.Nov 19 2021, 7:00 AM

Rebooted db1132 (future master) to pick up the latest kernel.

• Marostegui updated the task description. (Show Details)Nov 22 2021, 7:57 AM
gerritbot added a comment.Nov 23 2021, 6:10 AM

Change 740714 had a related patch set uploaded (by Marostegui; author: Marostegui):

[operations/puppet@production] mariadb: Promote db1132 to m5 master

https://gerrit.wikimedia.org/r/740714

gerritbot added a project: Patch-For-Review.Nov 23 2021, 6:10 AM
• Marostegui updated the task description. (Show Details)Nov 23 2021, 6:32 AM
• Marostegui updated the task description. (Show Details)Nov 23 2021, 1:07 PM
• Marostegui updated the task description. (Show Details)Nov 23 2021, 1:11 PM
• Marostegui updated the task description. (Show Details)
gerritbot added a comment.Nov 23 2021, 1:13 PM

Change 740714 merged by Marostegui:

[operations/puppet@production] mariadb: Promote db1132 to m5 master

https://gerrit.wikimedia.org/r/740714

• Marostegui updated the task description. (Show Details)Nov 23 2021, 1:14 PM
• Marostegui updated the task description. (Show Details)Nov 23 2021, 1:50 PM
Stashbot added a comment.Nov 23 2021, 2:00 PM

Mentioned in SAL (#wikimedia-operations) [2021-11-23T14:00:33Z] <marostegui> Failover m5 from db1128 to db1132 - T288720

• Marostegui updated the task description. (Show Details)Nov 23 2021, 2:04 PM
• Marostegui added a comment.Nov 23 2021, 2:07 PM

The switchover was done
RO started: 14:01:00
RO finished: 14:01:17

Total RO time: 17 seconds

• Marostegui added a comment.Nov 23 2021, 2:08 PM

Thanks @Andrew @bd808 and @Ladsgroup for the support!

• Marostegui updated the task description. (Show Details)Nov 23 2021, 2:08 PM
Maintenance_bot removed a project: Patch-For-Review.Nov 23 2021, 2:10 PM
gerritbot added a comment.Nov 23 2021, 2:17 PM

Change 740839 had a related patch set uploaded (by Marostegui; author: Marostegui):

[operations/puppet@production] dbproxy10{17,21}: Change m5 standby host

https://gerrit.wikimedia.org/r/740839

gerritbot added a project: Patch-For-Review.Nov 23 2021, 2:17 PM
• Marostegui updated the task description. (Show Details)Nov 23 2021, 2:19 PM
gerritbot added a comment.Nov 24 2021, 5:58 AM

Change 740839 merged by Marostegui:

[operations/puppet@production] dbproxy10{17,21}: Change m5 standby host

https://gerrit.wikimedia.org/r/740839

• Marostegui updated the task description. (Show Details)Nov 24 2021, 6:01 AM
gerritbot added a comment.Nov 24 2021, 6:03 AM

Change 740964 had a related patch set uploaded (by Marostegui; author: Marostegui):

[operations/dns@master] wmnet: Restore TTL back to 5M for m5-master

https://gerrit.wikimedia.org/r/740964

• Marostegui updated the task description. (Show Details)Nov 24 2021, 6:03 AM
gerritbot added a comment.Nov 24 2021, 6:04 AM

Change 740964 merged by Marostegui:

[operations/dns@master] wmnet: Restore TTL back to 5M for m5-master

https://gerrit.wikimedia.org/r/740964

• Marostegui updated the task description. (Show Details)Nov 24 2021, 6:04 AM
Stashbot added a comment.Nov 24 2021, 6:05 AM

Mentioned in SAL (#wikimedia-operations) [2021-11-24T06:05:54Z] <marostegui> Upgrade db1128's kernel T288720

• Marostegui closed this task as Resolved.Nov 24 2021, 6:08 AM
• Marostegui updated the task description. (Show Details)

This is all done!

• Marostegui mentioned this in T295965: Test MariaDB 10.4 with Bullseye.Nov 24 2021, 6:11 AM
Content licensed under Creative Commons Attribution-ShareAlike (CC BY-SA) 4.0 unless otherwise noted; code licensed under GNU General Public License (GPL) 2.0 or later and other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL