Abstract away different database depooling mechanisms into a cookbook
Closed, ResolvedPublic
Actions

Assigned To

Authored By

	Ladsgroup
	Jan 14 2025, 1:49 PM

Project Tags

Referenced Files

None

Subscribers

Description

Currently, we have different ways to depool a host:

core dbs use dbctl depool and then wait for drain
parsercache/ms use dbctl set-weight for the whole section and then weight for drain
wikireplicas use confctl: https://wikitech.wikimedia.org/wiki/Portal:Data_Services/Admin/Runbooks/Depool_wikireplicas
Masters shouldn't be depooled
Backup sources don't need to be depooled but should be avoided when a backup is running.
misc cluster should use haproxy
...

It makes sense to actually let all of these be handled via a cookbook instead. So an SRE oncall simply would run the cookbook instead.

The idea is by @jcrespo

Details

Related Changes in Gerrit:

	Subject	Author	Repo	Branch	Lines +/-
	mysql: rename newpool cookbook to pool	Federico Ceratto	operations/cookbooks	master	+717 -1 K
	sre.mysql.newpool: [de]pool various section kinds	Federico Ceratto	operations/cookbooks	master	+1 K -24

Customize query in gerrit

Related Objects
Search...

Status	Assigned	Task
In Progress	None	T384810 MariaDB lifetime management system
Resolved	FCeratto-WMF	T383674 Abstract away different database depooling mechanisms into a cookbook
Duplicate	FCeratto-WMF	T411573 sre.mysql.parsercache: make it work with msX sections

Event Timeline

Ladsgroup created this task.Jan 14 2025, 1:49 PM

Restricted Application added a subscriber: Aklapper. · View Herald TranscriptJan 14 2025, 1:49 PM

For further context, as this is a huge scope. I think at a first step only 1 or 2 of those need to be automated. I think it is more important that there is a script that identifies the "category" and provides even just a link to docs than 100% automation.

E.g.:

db-depool masterdb1001.eqiad.wmnet
ABORTED: Depool not done- this is a primary mediawiki server so use this procedure instead: <link to master switchover>

The idea is, as more categories and automations are created, having an easy entry point for non-DBAs.

In a ideal scenario I'd love this to be:
depooling_cookbook $HOST and all the logic and checks are done underneath. So the operator wouldn't need to care whether it is safe to depool a host or not (or a section), if it belongs to MW or misc etc.

Marostegui triaged this task as Medium priority.Jan 14 2025, 2:32 PM

Marostegui moved this task from Triage to Refine on the DBA board.

depooling_cookbook $HOST and all the logic and checks are done underneath. So the operator wouldn't need to care whether it is safe to depool a host or not (or a section), if it belongs to MW or misc etc.

The start of this is the recently added sre.mysql.pool and sre.mysql.depool cookbooks for core DBs that could be expanded to cover the additional use cases.

Btw, I am going to use the same idea for backups, with a single command to $ recover db1001 and abstract the different methods into an easy to use script wrapper (but that is out of scope of this).

FCeratto-WMF updated the task description. (Show Details)Jan 28 2025, 3:15 PM

FCeratto-WMF subscribed.

FCeratto-WMF claimed this task.Jan 30 2025, 8:37 AM

FCeratto-WMF added a parent task: T384810: MariaDB lifetime management system.

FCeratto-WMF changed the task status from Open to In Progress.Mar 12 2025, 5:47 PM

FCeratto-WMF removed a project: Data-Persistence-Automations.Mar 25 2025, 9:23 AM

FCeratto-WMF removed FCeratto-WMF as the assignee of this task.Apr 16 2025, 12:38 PM

Marostegui updated the task description. (Show Details)Nov 19 2025, 8:05 AM

Marostegui moved this task from Refine to Ready on the DBA board.

Marostegui mentioned this in T411573: sre.mysql.parsercache: make it work with msX sections.Dec 3 2025, 6:23 AM

Marostegui added a subtask: T411573: sre.mysql.parsercache: make it work with msX sections.

FCeratto-WMF claimed this task.Dec 5 2025, 9:48 AM

FCeratto-WMF moved this task from Ready to In progress on the DBA board.

Change #1215575 had a related patch set uploaded (by Federico Ceratto; author: Federico Ceratto):

[operations/cookbooks@master] sre.mysql.pool: [de]pool various section kinds

https://gerrit.wikimedia.org/r/1215575

gerritbot added a project: Patch-For-Review.Dec 5 2025, 11:48 AM

FCeratto-WMF moved this task from In progress to Blocked on the DBA board.Dec 9 2025, 1:49 PM

Mentioned in SAL (#wikimedia-operations) [2025-12-22T11:04:48Z] <fceratto@cumin1003> START - Cookbook sre.mysql.newdepool es2051 - test T383674

Mentioned in SAL (#wikimedia-operations) [2025-12-22T11:12:38Z] <fceratto@cumin1003> END (PASS) - Cookbook sre.mysql.newdepool (exit_code=0) es2051 - test T383674

Mentioned in SAL (#wikimedia-operations) [2025-12-22T12:30:31Z] <fceratto@cumin1003> START - Cookbook sre.mysql.newpool es2051 gradually with 4 steps - test T383674

Mentioned in SAL (#wikimedia-operations) [2025-12-22T13:15:59Z] <fceratto@cumin1003> END (PASS) - Cookbook sre.mysql.newpool (exit_code=0) es2051 gradually with 4 steps - test T383674

Marostegui merged a task: T411573: sre.mysql.parsercache: make it work with msX sections.Dec 23 2025, 7:01 AM

FCeratto-WMF moved this task from Blocked to Pending comment on the DBA board.Jan 9 2026, 3:41 PM

Change #1215575 merged by Federico Ceratto:

[operations/cookbooks@master] sre.mysql.newpool: [de]pool various section kinds

https://gerrit.wikimedia.org/r/1215575