Possible user-facing experience:

toolforge storage list
toolforge storage create --type=s3 --size=4G <storagename>
toolforge storage show <storagename>
toolforge storage delete <storagename>

# tbd, some storage might not need mounting, like s3
toolforge jobs run --mount-storage=<storagename>:/path/to/mount/on ...

Main use cases:

• Generate reports (needed for jobs using buildpack base images)
• Generate some results/extract data from services

Possible storage providers:

• cinder
• s3/swift
• NFS

From T337192: Toolforge: consider introducing some semantics for persistent storage

During Wikimedia-Hackathon-2023 it was mentioned a couple of times how convenient it would be to have something similar to cinder volumes but for toolforge tools.

We could have a command line interface to create/manage different storage kinds. This has been previously suggested, some kind of toolforge storage [..] interface, with options to manage different backends:

• Ceph RBD volumes (similar to openstack cinder)
• NFS-like volumes (for filesharing)
• scratch-like storage (disposable)
• some kind of backups, TBD
• etc

With particular semantics that could be like:

• Read only storages for retrieval of public data (dumps, mediawiki), this would be parallel to the replicas as information sources.
• Read/write persistent shared storage (current scratch), this would be for sharing between users/projects.
• Read/write persistent private storage (current project home), this would be for your own tool, private
• Read/write ephemeral private storage (current /tmp, home project), this would be for temporary big file processing and such, I think currently people just use the home directory.