Make sre.mysql.clone handling pooling/depooling
Closed, ResolvedPublic
Actions

Assigned To

Authored By

	Marostegui
	Jul 1 2024, 8:12 AM

Project Tags

Referenced Files

	F58619308: Screenshot 2025-03-06 at 14-29-44 MySQL - SRE Data Persistence Team - Dashboards - Grafana.png
	Mar 6 2025, 1:30 PM

Subscribers

Description

Right now when we use sre.mysql.clone we need to depool the source host.
Ideally we should let sre.mysql.clone to do that, and then once the host is back in sync with the master, slowly repool it back.

If the source isn't pooled originally at the start of the clone, the cookbook should not repool it.

Let's not pool the target host as there could be multiple reasons no to do this right away: eg (new host that was was cloned for the first time, new version testing, etc)

Ensure that host is depooled upon clone, do it if necessary (see this conversation: https://gerrit.wikimedia.org/r/c/operations/cookbooks/+/1071155/comment/d9b49037_6fa7c7e6/)

Details

Related Changes in Gerrit:

Subject	Author	Repo	Branch	Lines +/-
sre.mysql.upgrade: Switch to Host, apt-get and mysql helpers	Arnaudb	operations/cookbooks	master	+114 -96
mariadb: clone cookbook maintenance	Arnaudb	operations/cookbooks	master	+184 -155
mysql_legacy: fix list_host_instances	Arnaudb	operations/software/spicerack	master	+39 -36
mysql_legacy: reuse old code pattern to avoid potential issue	Arnaudb	operations/software/spicerack	master	+3 -0

Customize query in gerrit

Related Objects
Search...

Status	Assigned	Task
		Unknown Object (Task)
Resolved	Jhancock.wm	T342174 Q1:rack/setup/install db21[88-95]
Resolved	ABran-WMF	T343674 Productionize db21[88-95]
In Progress	FCeratto-WMF	T384810 MariaDB lifetime management system
Open	FCeratto-WMF	T423841 Add section-level locking to scripts and cookbooks
Open	FCeratto-WMF	T239814 Automate DB upgrades
Resolved	Volans	T362893 Spicerack support for dbctl
Resolved	ABran-WMF	T368881 Make sre.mysql.clone handling pooling/depooling
Resolved	None	T374191 upgrade clone cookbook
Declined	ABran-WMF	T376712 mysql_legacy: SQL query quote escape
Resolved	ABran-WMF	T376701 mysql_legacy data_directory getter
Resolved	ABran-WMF	T377129 mariadb: systemctl status accessor in mysql_legacy

Event Timeline

Marostegui created this task.Jul 1 2024, 8:12 AM

Restricted Application added a subscriber: Aklapper. · View Herald TranscriptJul 1 2024, 8:12 AM

This will be unlocked by T362893

ABran-WMF claimed this task.Jul 3 2024, 6:17 AM

ABran-WMF triaged this task as Medium priority.

ABran-WMF moved this task from Triage to Ready on the DBA board.

ABran-WMF added a project: Data-Persistence-Automations.Aug 21 2024, 7:06 AM

ABran-WMF mentioned this in T374191: upgrade clone cookbook.Sep 6 2024, 6:58 AM

ABran-WMF removed ABran-WMF as the assignee of this task.Sep 11 2024, 9:00 AM

ABran-WMF added a subtask: T374191: upgrade clone cookbook.Oct 8 2024, 1:36 PM

ABran-WMF updated the task description. (Show Details)Oct 8 2024, 2:24 PM

ABran-WMF changed the status of subtask T374191: upgrade clone cookbook from In Progress to Open.Oct 15 2024, 9:27 AM

Change #1080718 had a related patch set uploaded (by Arnaudb; author: Arnaudb):

[operations/cookbooks@master] sre.mysql.upgrade: add depool/pool logic

https://gerrit.wikimedia.org/r/1080718

gerritbot added a project: Patch-For-Review.Oct 16 2024, 2:02 PM

ABran-WMF moved this task from Todo to External dep/In review on the Data-Persistence-Automations board.Oct 17 2024, 7:24 AM

Change #1071155 had a related patch set uploaded (by Arnaudb; author: Arnaudb):

[operations/cookbooks@master] mariadb: clone cookbook maintenance

https://gerrit.wikimedia.org/r/1071155

ABran-WMF claimed this task.Oct 17 2024, 9:28 AM

ABran-WMF added a subtask: T377129: mariadb: systemctl status accessor in mysql_legacy.

ABran-WMF added a subtask: T376701: mysql_legacy data_directory getter.

ABran-WMF moved this task from Ready to Blocked on the DBA board.

Change #1082469 had a related patch set uploaded (by Arnaudb; author: Arnaudb):

[operations/software/spicerack@master] mysql_legacy: reuse old code pattern to avoid potential issue

https://gerrit.wikimedia.org/r/1082469

Change #1082469 abandoned by Arnaudb:

[operations/software/spicerack@master] mysql_legacy: reuse old code pattern to avoid potential issue

Reason:

as discussed, this will be replaced by a comment on the cookbook where it was found, so if it fails it is identifiable.

https://gerrit.wikimedia.org/r/1082469

code is implemented, needs to be tested under T374191

ABran-WMF mentioned this in T376712: mysql_legacy: SQL query quote escape.Oct 23 2024, 1:43 PM

ABran-WMF closed subtask T377129: mariadb: systemctl status accessor in mysql_legacy as Resolved.Oct 23 2024, 1:47 PM

Change #1084132 had a related patch set uploaded (by Arnaudb; author: Arnaudb):

[operations/software/spicerack@master] mysql_legacy: fix _list_host_instances

https://gerrit.wikimedia.org/r/1084132

Change #1084132 merged by jenkins-bot:

[operations/software/spicerack@master] mysql_legacy: fix list_host_instances

https://gerrit.wikimedia.org/r/1084132

Marostegui closed subtask T374191: upgrade clone cookbook as Resolved.Jan 16 2025, 11:24 AM

I tested the CR at the current version (patchset 26) doing a real update on db1197 and it worked without manual intervention.

https://grafana.wikimedia.org/d/000000273/mysql?orgId=1&var-job=All&var-server=db1197&var-port=9104&from=1741261829208&to=1741267475523

The script is able to depool, optionally, if the host is configured on dbctl and repool.
Repooling can be disabled.
Right now it's designed to upgrade one host at a time and I'd recommend doing rolling updates rather than batches to minimize the number of pooled-out hosts at a time and the impact if the update fails.
In future the cookbook could be updated to support multi-instance hosts.

Right now very little testing is done after the upgrade and this could be improved.

Also, the output of the upgrade in terms of installed packages and versions should be compared across hosts to detect unexpected differences. It would be still far from an atomic process but much bettter than unmanaged.

Change #1080718 abandoned by Arnaudb:

[operations/cookbooks@master] sre.mysql.upgrade: Switch to Host, apt-get and mysql helpers

https://gerrit.wikimedia.org/r/1080718

Change #1071155 abandoned by Arnaudb:

[operations/cookbooks@master] mariadb: clone cookbook maintenance

https://gerrit.wikimedia.org/r/1071155

Maintenance_bot removed a project: Patch-For-Review.Jul 28 2025, 2:33 PM

Change #1080718 restored by Federico Ceratto:

[operations/cookbooks@master] sre.mysql.upgrade: Switch to Host, apt-get and mysql helpers

https://gerrit.wikimedia.org/r/1080718

gerritbot added a project: Patch-For-Review.Aug 11 2025, 3:58 PM

Change #1080718 abandoned by Federico Ceratto: