Page MenuHomePhabricator

Trove for some ToolsDB users
Open, MediumPublic


We've talked about this in the past, but never actually made a task. The upcoming Stretch deprecation seems like a good opportunity to do it.

ToolsDB is a heavily loaded shared database. Combining many small users to a shared database makes some sense, but we should move some of its larger/heavier users to separate instances. Doing it via OpenStack Trove seems like the best solution.

Still to sort out: backups, replication?, how to enable self-provisioning of a new db by a Toolforge maintainer

Related Objects

Event Timeline

Ideally we would migrate all users off of toolsdb onto Trove or something else. Anyone who isn't using any esoteric "features" of ToolsDB and who can migrate should do so.

I would like to Trove to have replication and/or backup solutions available before we move much user data there. The existing toolsdb has both backups AND replication so it's a step-backwards in terms of resiliency until we have Trove spruced up.

aborrero triaged this task as Medium priority.Oct 13 2021, 3:17 PM
aborrero moved this task from Needs discussion to Blocked on the cloud-services-team (Kanban) board.

I opened T326754: Clarify Trove and Toolsdb usage within WMCS, as per the description, I think what was suggested here has largely come to pass. Thoughts?

I think this task can still be useful as a parent task for any migration we want to do from ToolsDB to Trove.

I propose the scope for this task becomes:

  • define which databases we absolutely want to migrate because of size/load
  • migrate them!
fnegri renamed this task from Trove for some ToolsDB users(?) to Trove for some ToolsDB users.Jan 31 2023, 3:58 PM

As of today, data from

Total number of tools is 475.

Size under10 MB100 MB500 MB1 GB5 GB10 GB25 GB50 GB
Number of Tools235337391407433450464468
Percentage of Tools49.47%70.95%82.32%85.68%91.16%94.74%97.68%98.53%

As the table suggest, more than 90% of tools fall under the 5GB limit, and almost 95% fall under the 10GB Limit.

Choosing a 10GB cutoff would mean a migration of 25 tools to trove. A 25GB cutoff would mean a migration of 11 tools to trove.

25 more trove databases sounds like a lot to me, let's start with the 25GB cutoff.

fnegri changed the subtype of this task from "Task" to "Goal".
fnegri changed the subtype of this task from "Goal" to "Task".Apr 19 2023, 2:22 PM