Page MenuHomePhabricator

Upgrade and restart m5 master (db1133)
Closed, ResolvedPublic

Description

In order to continue with T239791: DB: perform rolling restart of mariadb daemons to pick up CA changes we'd need to restart MySQL on m5 host (db1133).
As we have to restart MySQL, we will also upgrade it to the newer version.

These are the databases m5 currently store:

root@cumin1001:/home/marostegui# mysql.py -hdb1133 -e "show databases"
+---------------------+
| Database            |
+---------------------+
| designate           |
| glance              |
| heartbeat           |
| information_schema  |
| keystone            |
| labsdbaccounts      |
| labswiki            |
| mysql               |
| neutron             |
| nova                |
| nova_api            |
| nova_api_eqiad1     |
| nova_cell0_eqiad1   |
| nova_eqiad1         |
| ops                 |
| performance_schema  |
| striker             |
| sys                 |
| test_labsdbaccounts |
| testreduce_0715     |
| testreduce_vd       |
+---------------------+

This restart shouldn't take more than a few seconds, but obviously during those seconds, all the databases (including wikitech) will not be available.
cloud-services-team when would it be a good moment to perform this restart?

Event Timeline

Restricted Application added a subscriber: Aklapper. · View Herald TranscriptFeb 4 2020, 9:41 AM
Marostegui triaged this task as Medium priority.Feb 4 2020, 9:41 AM
Marostegui moved this task from Triage to Next on the DBA board.
Andrew added a comment.Feb 4 2020, 2:19 PM

For a few seconds interruption I wouldn't expect this to be very disruptive. If you schedule it in my morning (e.g. 15:00 UTC) then I can send out notice to users &c. and be around in case unexpected things happen.

For a few seconds interruption I wouldn't expect this to be very disruptive. If you schedule it in my morning (e.g. 15:00 UTC) then I can send out notice to users &c. and be around in case unexpected things happen.

Thank you!
What about Monday 10th at 15:00 UTC?

Andrew added a comment.Feb 4 2020, 2:27 PM

Thank you!
What about Monday 10th at 15:00 UTC?

Works for me! I'll put it on the team calendar.

Thank you!
What about Monday 10th at 15:00 UTC?

Works for me! I'll put it on the team calendar.

Excellent! Do you want me to email wikitech-l or you'll do it?

Andrew added a comment.Feb 4 2020, 2:33 PM

I'll do it now.

Andrew added a comment.Feb 4 2020, 2:40 PM
  • email sent to wikitech-l and cloud-announce
Marostegui moved this task from Next to In progress on the DBA board.Feb 5 2020, 9:39 AM

Mentioned in SAL (#wikimedia-operations) [2020-02-10T14:32:48Z] <marostegui> Downtime m5 hosts for the upcoming maintenance - T244209

Mentioned in SAL (#wikimedia-operations) [2020-02-10T14:41:36Z] <marostegui> Full-upgrade db1133 (without restarting mysql) - T244209

Mentioned in SAL (#wikimedia-operations) [2020-02-10T15:00:58Z] <marostegui> Restart mysql on m5 master (wikitech will go down) - T244209

Mentioned in SAL (#wikimedia-operations) [2020-02-10T15:06:53Z] <marostegui> Reload haproxy on dbproxy1017 and dbproxy1017 - T244209

Marostegui closed this task as Resolved.Feb 10 2020, 3:07 PM

This has been done.
Downtime was:
15:01:13 - 15:02:27

mysql:root@localhost [(none)]> select @@version;
+-----------------+
| @@version       |
+-----------------+
| 10.1.43-MariaDB |
+-----------------+
1 row in set (0.00 sec)

Thanks @Andrew for your support! :)