Page MenuHomePhabricator

Setup proof of concept storage and retrieval from WMCS object storage
Closed, ResolvedPublicFeature

Description

Follow up T420532: Request creation of etherpads3 VPS project by creating a storage bucket, loading it with some archived pads, and reading them back.

  • create service user to hold write credentials for the bucket
  • create a bucket
  • load some files from /data/project/etherpad-backup/public_html/p into the bucket
  • fetch some files from the bucket
  • profit!

Event Timeline

bd808 changed the task status from Open to In Progress.Apr 10 2026, 5:13 PM
bd808 claimed this task.
bd808 triaged this task as Medium priority.

The new service user is https://ldap.toolforge.org/user/etherpadbackupbot / https://meta.wikimedia.org/wiki/User:EtherpadBackupBot. I have the passwords for this Developer / SUL account pair in my local password manager. The accounts are connected to the etherpad-backup dot bot at toolforge dot org email address which will message every maintainer of the etherpad-backup tool. This should allow ongoing use of the account when and if I am unable to continue to provide needed access.

I followed the directions from https://wikitech.wikimedia.org/wiki/Help:Object_storage_user_guide#S3_API to create credentials for the etherpadbackupbot account. These are currently stored as a note in my local password manager.

I then continued on by following the "s3cmd example" instructions that @taavi nicely left showing how to test the credentials and play with the storage. The result: https://object.eqiad1.wikimediacloud.org/84baa6f9fe8d41afb4b7ca99891161f3:etherpads/README.md

I tried using s3cmd ws-create s3://html to make a bucket that would act as a webservice and automatically serve up an index.html file. Unfortunately the command failed with ERROR: S3 error: 405 (MethodNotAllowed). I'm not sure if this is deliberate, or a sign that there is some config that needs to be added on the rados gateway. I'll try to remember to ask somebody about that.

I meant to stay in this rabbit hole a bit longer with testing things, but my wiggles got the best of me over the weekend and I jumped past testing and into building a webservice. I then found other rabbit holes to fall into (oauth, better local dev), but I am making progress. This is a "volunteer time" project, so I probably won't get back to it seriously before the weekend.

All of the files in the tool's $HOME/public_html/p as of 2026-04-19T22:41Z have been uploaded to the storage bucket. I did this the sort of boring way of creating a tarball on Toolforge, scp'ing that to my laptop, and uploading to the bucket using a local install of s3cmd.

Mentioned in SAL (#wikimedia-cloud) [2026-05-01T10:17:49Z] <wmbot~bd808@tools-bastion-14> Built and deployed image from ecf5892c (T422958)

Mentioned in SAL (#wikimedia-cloud) [2026-05-01T13:03:15Z] <wmbot~bd808@tools-bastion-14> Built image from bab09762 and deployed (T422958)