Page MenuHomePhabricator

Migrate largest ToolsDB users to Trove
Open, MediumPublic

Description

ToolsDB is a heavily loaded shared database. Combining many small users to a shared database makes some sense, but we should move some of its larger/heavier users to separate instances. Doing it via OpenStack Trove seems like the best solution.

Still to sort out: backups, replication?, how to enable self-provisioning of a new db by a Toolforge maintainer

ToolsDB databases ordered by size: https://tool-db-usage.toolforge.org/

Related Objects

StatusSubtypeAssignedTask
ResolvedBstorm
ResolvedBstorm
Resolvedfnegri
Resolveddcaro
OpenNone
Openfnegri
ResolvedAndrew
Resolvedaborrero
DuplicateNone
Resolvedaborrero
Resolvedaborrero
Resolvedaborrero
ResolvedAndrew
ResolvedAndrew
OpenNone
ResolvedAndrew
ResolvedAndrew
Resolvedfnegri
ResolvedTBurmeister
Resolvedfnegri
ResolvedTheresNoTime
Resolvedfnegri
Openfnegri

Event Timeline

Ideally we would migrate all users off of toolsdb onto Trove or something else. Anyone who isn't using any esoteric "features" of ToolsDB and who can migrate should do so.

I would like to Trove to have replication and/or backup solutions available before we move much user data there. The existing toolsdb has both backups AND replication so it's a step-backwards in terms of resiliency until we have Trove spruced up.

aborrero triaged this task as Medium priority.Oct 13 2021, 3:17 PM
aborrero moved this task from Needs discussion to Blocked on the cloud-services-team (Kanban) board.

I opened T326754: Clarify Trove and Toolsdb usage within WMCS, as per the description, I think what was suggested here has largely come to pass. Thoughts?

I think this task can still be useful as a parent task for any migration we want to do from ToolsDB to Trove.

I propose the scope for this task becomes:

  • define which databases we absolutely want to migrate because of size/load
  • migrate them!
fnegri renamed this task from Trove for some ToolsDB users(?) to Trove for some ToolsDB users.Jan 31 2023, 3:58 PM

As of today, data from https://tool-db-usage.toolforge.org/

Total number of tools is 475.

Size under10 MB100 MB500 MB1 GB5 GB10 GB25 GB50 GB
Number of Tools235337391407433450464468
Percentage of Tools49.47%70.95%82.32%85.68%91.16%94.74%97.68%98.53%

As the table suggest, more than 90% of tools fall under the 5GB limit, and almost 95% fall under the 10GB Limit.

Choosing a 10GB cutoff would mean a migration of 25 tools to trove. A 25GB cutoff would mean a migration of 11 tools to trove.

25 more trove databases sounds like a lot to me, let's start with the 25GB cutoff.

fnegri changed the subtype of this task from "Task" to "Goal".
fnegri changed the subtype of this task from "Goal" to "Task".Apr 19 2023, 2:22 PM
fnegri renamed this task from Trove for some ToolsDB users to Migrate largest ToolsDB users to Trove.Jan 22 2024, 5:22 PM
fnegri updated the task description. (Show Details)