Page MenuHomePhabricator

Request to add hoiscript.toolforge.org to wgCopyUploadsDomains for batch uploading
Closed, DeclinedPublic

Description

Hello.

I am uploading tons of pdf files to Commons with the bot account [[User:PencakeBot]]. Those files are scanned copies of ancient books from the National Library of China (read.nlc.cn).

To automate the upload process, I write a script in Python with mwclient. But just at the very first, I encounter frustrating problems. Some files are of 100+MiBs (typically less than 150MiB). For these (not so) big files, chunked-uploading returns a timeout error (typically 504 upstream request timeout) almost all the time for the final chunk. After spending hours of effort to retry it again and again in different network environments and alternative bot frameworks, I still cannot manage to upload them.

I was told by kind people in #wikimedia-tech that there are long-term stability problems with chunked-uploading. I realize uploading by URLs seems an ideal alternative way. nlc.cn does not provide static URLs for all of its files. So I am planning to let Commons fetch files from my server.

Could you please add my domain hoiscript.toolforge.org to the wgCopyUploadsDomains list so that I can upload files directly from my domain/server to Commons?

Thanks!

Related Objects

View Standalone Graph
This task is connected to more than 200 other tasks. Only direct parents and subtasks are shown here. Use View Standalone Graph to show more of the graph.

Event Timeline

Aklapper renamed this task from Request to add my domain to wgCopyUploadsDomains for batch uploading to Request to add pencakebot.lotust.xyz to wgCopyUploadsDomains for batch uploading.Oct 9 2021, 9:49 PM
Urbanecm subscribed.

The site returns an error now. Reopen if this is still desirable.

Hoi renamed this task from Request to add pencakebot.lotust.xyz to wgCopyUploadsDomains for batch uploading to Request to add hoiscript.toolforge.org to wgCopyUploadsDomains for batch uploading.Jun 7 2023, 6:38 AM
Hoi updated the task description. (Show Details)