Page MenuHomePhabricator

Request increased quota for hoiscript Toolforge tool
Closed, DeclinedPublic

Description

Tool Name: hoiscript
Quota increase requested: +10GiBRAM
Reason:
I am running automated upload tasks to Commons with this tool account. It downloads images and combines them into a PDF with the Python package img2pdf. Sometimes, there are too many images to process, resulting in the task being killed due to exhausting memory.

Typically, such a set of images belongs to a single book volume. So they have to be packed into a single PDF at once. <del>Considering the maximum file size limit of Commons is 4GiB, the reasonable working set memory requirements would be 4GiB (images to be processed) + 4GiB (generated PDF in memory) + some necessary redundancy (assuming 2GiB).</del>

There are some bigger PDFs in my other upload tasks. Some of them exceed 6GiB[1]. The upload script has to split them and upload them as multiple files, which requires more memory.

I would like to request to increase the memory limit per container and quota per account to <del>10GiB</del> 16GiB.

[1] https://hoiscript.toolforge.org/nlc/nlc511-34673.pdf

Thanks.

Event Timeline

Some Toolforge worker machines have 16GB RAM, but others (the majority) have 8GB RAM.

I think the usage you are describing may result in kubernetes having some difficulties to schedule your tool's pods.

If the software can't be rearranged in a different architecture to better fit the current memory limits of Toolforge Kubernetes, we may offer you a Cloud VPS project.

What do you think?

I am afraid it is not easy to rearrange the upload script since it depends on some PDF libraries that are infeasible to modify.

A Cloud VPS would be great. Actually, I was trying to run the upload script on the bastion machine. But some packages cannot be installed there.

• aborrero triaged this task as Medium priority.
• aborrero moved this task from Inbox to Discussion needed on the Toolforge (Quota-requests) board.

Please request a Cloud VPS project then: https://phabricator.wikimedia.org/project/view/2875/