Page MenuHomePhabricator

Request increased quota for wmf-research-tools Cloud VPS project
Closed, ResolvedPublic

Description

Project Name: wmf-research-tools (original creation task: T186519)

Type of quota increase requested:

  • CPUs: +16
  • RAM: +48GB
  • Instance count: +8
  • Floating IP: none

Reason: We continue to prototype out new tools and fill up our quota. I've gone through the current set of tools on the project and they're still all active prototypes that we share out. Specifially, we've been testing out more advanced machine learning models like article description generation (T318384; currently hosted on separate Cloud VPS project), vandalism detection (T314386; now exploring productionization on LiftWing), and citation verification as parts of APIs. These often require more RAM per instance -- e.g., 4-8GB -- to run.

Prior request: T266180

Event Timeline

48GB is quite a lot of RAM, but given that we have it available and the use case is interesting, +1 to the request.

currently hosted on separate Cloud VPS project

@Isaac does it mean that we could potentially reduce the quota on that separate project?

@Isaac does it mean that we could potentially reduce the quota on that separate project?

I think not -- I probably won't actually move that one over, was just using it as an example of some of the larger machine learning models we've been testing (hence the RAM needs). More generally, I think we're probably an atypical user of Cloud VPS in that we have two types of projects:

  • A small number of long-term, easier-to-plan-for tools that we create like the article description generation model mentioned. We mostly try to keep them on the recommendation-api project. I assume this is how most Cloud VPS users operate.
  • A large number of shorter-term tools that we are constantly prototyping out via Cloud VPS. If they're internal, we do them on this project. If they're with external collaborators, we do them on research-collaborations-api (T274400).

Because of the latter, our resource usage fluctuates a good bit depending on the projects we're working on. We've been bumping against the resource limit for a bit (which isn't bad because it makes sure we clean up older, no-longer-needed instances), but with a few bigger models under discussion and no obvious way to make space at the moment, it felt like a good time to expand. I also suspect we'll want to be able to make space if needed on research-collaborations-api for the Hackathon in a few months, which this request will help with. I'll keep in mind though that we can return resources and try to make sure to check semi-annually to see if we're consistently under-utilizing the projects.

Mentioned in SAL (#wikimedia-cloud-feed) [2023-01-12T14:07:28Z] <wm-bot2> Increased quotas by 16 cores, 8 instances, 49152 ram (T326338) - cookbook ran by arturo@nostromo

aborrero claimed this task.

@Isaac thanks for the detailed answer, really appreciated. :)