Upgrade and restart m5 master (db1133)
Closed, ResolvedPublic
Actions

Assigned To

Authored By

	Marostegui
	Feb 4 2020, 9:41 AM

Description

In order to continue with T239791: DB: perform rolling restart of mariadb daemons to pick up CA changes we'd need to restart MySQL on m5 host (db1133).
As we have to restart MySQL, we will also upgrade it to the newer version.

These are the databases m5 currently store:

root@cumin1001:/home/marostegui# mysql.py -hdb1133 -e "show databases"
+---------------------+
| Database            |
+---------------------+
| designate           |
| glance              |
| heartbeat           |
| information_schema  |
| keystone            |
| labsdbaccounts      |
| labswiki            |
| mysql               |
| neutron             |
| nova                |
| nova_api            |
| nova_api_eqiad1     |
| nova_cell0_eqiad1   |
| nova_eqiad1         |
| ops                 |
| performance_schema  |
| striker             |
| sys                 |
| test_labsdbaccounts |
| testreduce_0715     |
| testreduce_vd       |
+---------------------+

This restart shouldn't take more than a few seconds, but obviously during those seconds, all the databases (including wikitech) will not be available.
cloud-services-team when would it be a good moment to perform this restart?

Related Objects
Search...

Status	Assigned	Task
Resolved	jbond	T236277 Extend Puppet CA Expiry date
Resolved	jbond	T237259 Document all uses of the puppetCA certificate
Resolved	RLazarus	T237362 Rolling restart of etcd to pick up the renewed CA public certificate.
Resolved	Marostegui	T239791 DB: perform rolling restart of mariadb daemons to pick up CA changes
Resolved	Marostegui	T244209 Upgrade and restart m5 master (db1133)
Resolved	jbond	T240330 Check if traffic servers need restarting/reloading post CA change

Event Timeline

Marostegui created this task.Feb 4 2020, 9:41 AM

Restricted Application added a subscriber: Aklapper. · View Herald TranscriptFeb 4 2020, 9:41 AM

Marostegui triaged this task as Medium priority.Feb 4 2020, 9:41 AM

Marostegui moved this task from Triage to Pending comment on the DBA board.

For a few seconds interruption I wouldn't expect this to be very disruptive. If you schedule it in my morning (e.g. 15:00 UTC) then I can send out notice to users &c. and be around in case unexpected things happen.

In T244209#5848154, @Andrew wrote:

For a few seconds interruption I wouldn't expect this to be very disruptive. If you schedule it in my morning (e.g. 15:00 UTC) then I can send out notice to users &c. and be around in case unexpected things happen.

Thank you!
What about Monday 10th at 15:00 UTC?

In T244209#5848195, @Marostegui wrote:

Thank you!
What about Monday 10th at 15:00 UTC?

Works for me! I'll put it on the team calendar.

In T244209#5848200, @Andrew wrote:

In T244209#5848195, @Marostegui wrote:

Thank you!
What about Monday 10th at 15:00 UTC?

Works for me! I'll put it on the team calendar.

Excellent! Do you want me to email wikitech-l or you'll do it?

I'll do it now.

Thank you!

email sent to wikitech-l and cloud-announce

Marostegui moved this task from Pending comment to In progress on the DBA board.Feb 5 2020, 9:39 AM

Mentioned in SAL (#wikimedia-operations) [2020-02-10T14:32:48Z] <marostegui> Downtime m5 hosts for the upcoming maintenance - T244209

Mentioned in SAL (#wikimedia-operations) [2020-02-10T14:41:36Z] <marostegui> Full-upgrade db1133 (without restarting mysql) - T244209

Mentioned in SAL (#wikimedia-operations) [2020-02-10T15:00:58Z] <marostegui> Restart mysql on m5 master (wikitech will go down) - T244209

Mentioned in SAL (#wikimedia-operations) [2020-02-10T15:06:53Z] <marostegui> Reload haproxy on dbproxy1017 and dbproxy1017 - T244209

This has been done.
Downtime was:
15:01:13 - 15:02:27

mysql:root@localhost [(none)]> select @@version;
+-----------------+
| @@version       |
+-----------------+
| 10.1.43-MariaDB |
+-----------------+
1 row in set (0.00 sec)

Thanks @Andrew for your support! :)

Marostegui mentioned this in T244566: Upgrade and restart m3 (phabricator) master (db1128).Feb 10 2020, 3:40 PM

Marostegui mentioned this in T244238: Upgrade and restart m1 master (db1135).

Upgrade and restart m5 master (db1133)Closed, ResolvedPublicActions

Description

Related ObjectsSearch...

Event Timeline

Upgrade and restart m5 master (db1133)
Closed, ResolvedPublic
Actions

Related Objects
Search...