Page MenuHomePhabricator

Upgrade and restart m5 master (db1133)
Closed, ResolvedPublic

Description

In order to continue with T239791: DB: perform rolling restart of mariadb daemons to pick up CA changes we'd need to restart MySQL on m5 host (db1133).
As we have to restart MySQL, we will also upgrade it to the newer version.

These are the databases m5 currently store:

root@cumin1001:/home/marostegui# mysql.py -hdb1133 -e "show databases"
+---------------------+
| Database            |
+---------------------+
| designate           |
| glance              |
| heartbeat           |
| information_schema  |
| keystone            |
| labsdbaccounts      |
| labswiki            |
| mysql               |
| neutron             |
| nova                |
| nova_api            |
| nova_api_eqiad1     |
| nova_cell0_eqiad1   |
| nova_eqiad1         |
| ops                 |
| performance_schema  |
| striker             |
| sys                 |
| test_labsdbaccounts |
| testreduce_0715     |
| testreduce_vd       |
+---------------------+

This restart shouldn't take more than a few seconds, but obviously during those seconds, all the databases (including wikitech) will not be available.
cloud-services-team when would it be a good moment to perform this restart?

Event Timeline

Marostegui triaged this task as Medium priority.Feb 4 2020, 9:41 AM
Marostegui moved this task from Triage to Pending comment on the DBA board.

For a few seconds interruption I wouldn't expect this to be very disruptive. If you schedule it in my morning (e.g. 15:00 UTC) then I can send out notice to users &c. and be around in case unexpected things happen.

For a few seconds interruption I wouldn't expect this to be very disruptive. If you schedule it in my morning (e.g. 15:00 UTC) then I can send out notice to users &c. and be around in case unexpected things happen.

Thank you!
What about Monday 10th at 15:00 UTC?

Thank you!
What about Monday 10th at 15:00 UTC?

Works for me! I'll put it on the team calendar.

Thank you!
What about Monday 10th at 15:00 UTC?

Works for me! I'll put it on the team calendar.

Excellent! Do you want me to email wikitech-l or you'll do it?

  • email sent to wikitech-l and cloud-announce

Mentioned in SAL (#wikimedia-operations) [2020-02-10T14:32:48Z] <marostegui> Downtime m5 hosts for the upcoming maintenance - T244209

Mentioned in SAL (#wikimedia-operations) [2020-02-10T14:41:36Z] <marostegui> Full-upgrade db1133 (without restarting mysql) - T244209

Mentioned in SAL (#wikimedia-operations) [2020-02-10T15:00:58Z] <marostegui> Restart mysql on m5 master (wikitech will go down) - T244209

Mentioned in SAL (#wikimedia-operations) [2020-02-10T15:06:53Z] <marostegui> Reload haproxy on dbproxy1017 and dbproxy1017 - T244209

This has been done.
Downtime was:
15:01:13 - 15:02:27

mysql:root@localhost [(none)]> select @@version;
+-----------------+
| @@version       |
+-----------------+
| 10.1.43-MariaDB |
+-----------------+
1 row in set (0.00 sec)

Thanks @Andrew for your support! :)