Page MenuHomePhabricator

Abstract away different database depooling mechanisms into a cookbook
Open, In Progress, MediumPublic

Description

Currently, we have different ways to depool a host:

It makes sense to actually let all of these be handled via a cookbook instead. So an SRE oncall simply would run the cookbook instead.

The idea is by @jcrespo

Details

Event Timeline

For further context, as this is a huge scope. I think at a first step only 1 or 2 of those need to be automated. I think it is more important that there is a script that identifies the "category" and provides even just a link to docs than 100% automation.

E.g.:

db-depool masterdb1001.eqiad.wmnet
ABORTED: Depool not done- this is a primary mediawiki server so use this procedure instead: <link to master switchover>

The idea is, as more categories and automations are created, having an easy entry point for non-DBAs.

In a ideal scenario I'd love this to be:
depooling_cookbook $HOST and all the logic and checks are done underneath. So the operator wouldn't need to care whether it is safe to depool a host or not (or a section), if it belongs to MW or misc etc.

Marostegui moved this task from Triage to Refine on the DBA board.

depooling_cookbook $HOST and all the logic and checks are done underneath. So the operator wouldn't need to care whether it is safe to depool a host or not (or a section), if it belongs to MW or misc etc.

+1

The start of this is the recently added sre.mysql.pool and sre.mysql.depool cookbooks for core DBs that could be expanded to cover the additional use cases.

Btw, I am going to use the same idea for backups, with a single command to $ recover db1001 and abstract the different methods into an easy to use script wrapper (but that is out of scope of this).

FCeratto-WMF changed the task status from Open to In Progress.Mar 12 2025, 5:47 PM
Marostegui moved this task from Refine to Ready on the DBA board.
FCeratto-WMF moved this task from Ready to In progress on the DBA board.

Change #1215575 had a related patch set uploaded (by Federico Ceratto; author: Federico Ceratto):

[operations/cookbooks@master] sre.mysql.pool: [de]pool various section kinds

https://gerrit.wikimedia.org/r/1215575