Page MenuHomePhabricator

[toolforge,storage,swift,s3] Object store?
Open, HighPublic

Description

For a few tools, I find it necessary/helpful to store large numbers of medium-sized blobs (scraped HTML pages, JSON text, the like). I am currently using MariaDB tables for this, but some have grown to significant size, and I had to truncate one table recently, after a warning about the database size.

Is there/could there be an object store available on Toolforge/Labs, so that I can just record the Object URI/ID/whatnot in the database, and store the blob in the object store?

Not urgent or breaking, but might be nice to have...

Now that we have swift on cloudvps, we can use that (with some extra setup/api/system to allow interacting with it).

Event Timeline

A major focus for the cloud-services-team in the July 2019-June 2020 fiscal year will be improving and enhancing the storage systems that we use both "in the backend" for managing Cloud VPS instance virtual disks and in the more visible areas such as our current NFS system. The backend project will be worked on first, but we hope that work will lead to us being able to provide other features like an S3-like bucket+object storage interface in the later stages of the project. We are not ready to make any timeline commitments yet for when this might be available in Toolforge and other Cloud VPS projects.

dcaro renamed this task from Object store? to [toolforge,storage,swift,s3] Object store?.Feb 22 2024, 4:11 PM
dcaro triaged this task as High priority.
dcaro updated the task description. (Show Details)

As a concrete example cluebotng-trainer currently uses NFS for storing artefacts from training runs; this would easy fit into an object store model allowing 0 NFS usage in that namespace (currently 1 container provides read/write access to NFS for other containers)