This sounds like you might need some storage of significant size for temp space right? Is that why you aren't aiming to put it on Toolforge or is that because of the forked Pillow?

@Bstorm both. Re storage space: potentially yes, but only for the odd edge case if it runs across a gigapixel image (also why Pillow is forked to add handling for niche use case). Being space conscious, each worker only downloads/works with one file at a time and deletes the temp copy once done. I tend to run variable numbers of workers depending on the work load.

Program runs with a watcher/worker model, where the watcher shoves all relevant details (mostly just title, hash to verify download) on a text based redis queue, the workers then pop, download, process, do whatever is needed as result, delete. This is used since images are uploaded faster than can be processed in real time and was a suggested implementation at the commons approval.

I was asking about storage because the latest round of flavors come with only 20GB a piece. That suggests you may want to use cinder storage (usually for important data), something like scratch NFS or similar. If you aren't likely to need that at first, at least, when we can just make the project and figure that out later.

@Bstorm would it be something that could be easily attached or would that require the instance recreated? If the latter, we should probably do now. If it can be added seamlessly, then later is fine. Just wondering...if later, I’d just file a ticket here requesting or?

It can be added easily later. We'll move this along.

Mentioned in SAL (#wikimedia-cloud) [2021-04-15T12:36:13Z] <arturo> create project T279246