Avoid depooling hosts if the schema change has been applied before
Open, MediumPublic
Actions

Assigned To

Authored By

	Ladsgroup
	Jan 18 2022, 8:15 PM

Project Tags

Referenced Files

None

Subscribers

Description

It should simply detect+skip that host before attempting to depool it.

Details

Related Changes in GitLab:

Title	Reference	Author	Source Branch	Dest Branch
schema_change_helper: filter TODO replicas	repos/sre/schema-changes!65	fceratto	T299441-2	T423841
Tidy up imports	repos/data_persistence/dbtools/auto_schema!20	fceratto	T299441-5	main
Support setting replicas from CLI	repos/data_persistence/dbtools/auto_schema!19	fceratto	T299441-4	main
Draft: Depool hosts only when needed based on check()	repos/data_persistence/dbtools/auto_schema!14	fceratto	T299441-3	main
Add planning function	repos/data_persistence/dbtools/auto_schema!7	fceratto	T299441	main

Customize query in GitLab

Related Objects

Mentioned Here: P85838 dbctl commit (dc=all): 'Depooling db1187 (T299441)'
P85636 dbctl commit (dc=all): 'Depooling db1187 (T299441)'
P85617 dbctl commit (dc=all): 'Depooling db1187 (T299441)'
P85509 dbctl commit (dc=all): 'Depooling db1187 (T299441)'
P85504 dbctl commit (dc=all): 'Depooling db1187 (T299441)'
P85388 dbctl commit (dc=all): 'Depooling db1187 (T299441)'

Event Timeline

Ladsgroup created this task.Jan 18 2022, 8:15 PM

Restricted Application added a subscriber: Aklapper. · View Herald TranscriptJan 18 2022, 8:15 PM

Ladsgroup triaged this task as Low priority.Jan 18 2022, 8:15 PM

Ladsgroup moved this task from Triage to Ready on the DBA board.

I am going to give this medium priority instead of low as it can be very useful if a schema change needs to be retried on an entire section (ie: some hosts failed to depool, you might just want to simply run the schema change on the entire section instead of having to check+target specific hosts)

• Kormat subscribed.Feb 15 2022, 10:24 AM

ABran-WMF subscribed.May 24 2024, 1:16 PM

@FCeratto-WMF this is something I'd like you to tackle also once @Ladsgroup has gotten you into the wonderful world of schema changes.

FCeratto-WMF claimed this task.Jan 23 2025, 1:24 PM

ABran-WMF unsubscribed.Jan 23 2025, 1:28 PM

@FCeratto-WMF currently, the schema change script works this way:

Depool a host
Check if the schema change is applied
- if not -> run in -> repool the host in 5 steps
- if yes -> skip schema change -> repool the in 5 steps.

Ideally, the script should check if the schema change is applied (this is a safe operation and won't imply any peformance penalty) and if it is not applied, then depool, run and repool.
Let me know if this gives you enough context or you'd need more details.

FCeratto-WMF moved this task from Ready to In progress on the DBA board.Mar 13 2025, 8:43 AM

As requested by @Ladsgroup I moved the tool to https://gitlab.wikimedia.org/repos/data_persistence/dbtools/auto_schema - we also discussed adding tests, type hints, and fixing any bug that comes up that might impact the depool/repool process as we don't have any way to do real end-to-end testing.

fceratto opened https://gitlab.wikimedia.org/repos/data_persistence/dbtools/auto_schema/-/merge_requests/7

Skip depooling hosts where schema change has been already performed

FCeratto-WMF moved this task from In progress to Blocked on the DBA board.Mar 18 2025, 9:08 AM

ladsgroup merged https://gitlab.wikimedia.org/repos/data_persistence/dbtools/auto_schema/-/merge_requests/7

Add planning function

Maintenance_bot removed a project: Patch-For-Review.Jun 17 2025, 2:32 PM

FCeratto-WMF moved this task from Blocked to Ready on the DBA board.Jun 23 2025, 2:12 PM

FCeratto-WMF moved this task from Ready to Blocked on the DBA board.Jul 22 2025, 10:31 AM

fceratto opened https://gitlab.wikimedia.org/repos/data_persistence/dbtools/auto_schema/-/merge_requests/14

Draft: Depool hosts only when needed based on check()

As discussed during the meeting - let's do:

Use the wrapper at https://gitlab.wikimedia.org/repos/sre/schema-changes/-/merge_requests/42 (which needs to be merged)
Document the way to run schema changes with the helper (which will populate zarcillo)
Add a flag for the current auto_schema to pass the list of hosts that need the schema change

Marostegui moved this task from Blocked to In progress on the DBA board.Nov 18 2025, 8:58 AM

Mentioned in SAL (#wikimedia-operations) [2025-11-19T13:33:59Z] <marostegui@cumin1003> dbctl commit (dc=all): 'Depooling db1187 (T299441)', diff saved to https://phabricator.wikimedia.org/P85388 and previous config saved to /var/cache/conftool/dbconfig/20251119-133358-marostegui.json

Mentioned in SAL (#wikimedia-operations) [2025-11-24T11:21:11Z] <marostegui@cumin1003> dbctl commit (dc=all): 'Depooling db1187 (T299441)', diff saved to https://phabricator.wikimedia.org/P85504 and previous config saved to /var/cache/conftool/dbconfig/20251124-112111-marostegui.json

Mentioned in SAL (#wikimedia-operations) [2025-11-24T11:28:50Z] <marostegui@cumin1003> dbctl commit (dc=all): 'Depooling db1187 (T299441)', diff saved to https://phabricator.wikimedia.org/P85509 and previous config saved to /var/cache/conftool/dbconfig/20251124-112850-marostegui.json

FCeratto-WMF moved this task from In progress to Ready on the DBA board.Nov 25 2025, 11:27 AM

Mentioned in SAL (#wikimedia-operations) [2025-11-25T14:08:10Z] <fceratto@cumin1003> dbctl commit (dc=all): 'Depooling db1187 (T299441)', diff saved to https://phabricator.wikimedia.org/P85617 and previous config saved to /var/cache/conftool/dbconfig/20251125-140809-fceratto.json

Mentioned in SAL (#wikimedia-operations) [2025-11-25T15:20:32Z] <fceratto@cumin1003> dbctl commit (dc=all): 'Depooling db1187 (T299441)', diff saved to https://phabricator.wikimedia.org/P85636 and previous config saved to /var/cache/conftool/dbconfig/20251125-152031-fceratto.json

Mentioned in SAL (#wikimedia-operations) [2025-11-27T10:39:02Z] <fceratto@cumin1003> dbctl commit (dc=all): 'Depooling db1187 (T299441)', diff saved to https://phabricator.wikimedia.org/P85838 and previous config saved to /var/cache/conftool/dbconfig/20251127-103901-fceratto.json