This is a tracking task to collect all subtasks related to issues with large file uploads to commons.
Description
Description
| Status | Subtype | Assigned | Task | ||
|---|---|---|---|---|---|
| Invalid | None | T292954 [epic] large file uploads to commons | |||
| Open | None | T278104 Unable to upload to Commons: uploadstash-file-not-found: Key "187kyl5ozj74.xtav8j.51508.djvu" not found in stash | |||
| Resolved | Legoktm | T294689 Please move file from en.wp to commons | |||
| Resolved | aborrero | T288481 Unexpected upload speed to commons | |||
| Resolved | None | T280926 Various errors when trying to upload large files (Could not acquire lock, Service Temporarily Unavailable, 503 Backend fetch failed, 502 Next Hop Connection Failed) | |||
| Resolved | None | T254459 Large PDF upload issue | |||
| Duplicate | PRODUCTION ERROR | None | T278389 File upload to Commons via UploadWizard fails with "Caught exception of type Wikimedia\Rdbms\DBQueryError" | ||
| Open | None | T201090 Intermittent transcode failure 'An unknown error occurred in storage backend "local-swift-codfw".' | |||
| Resolved | BUG REPORT | None | T288382 Difficulty uploading video file to Commons via multiple methods | ||
| Resolved | Legoktm | T275752 Jobrunner timeouts on cross-DC file uploads because of HTTP/2 |
Event Timeline
There are a very large number of changes, so older changes are hidden. Show Older Changes
Comment Actions
Neverending tracking tasks should not be created as it means that they'll never get resolved. Dedicated project tags should be used instead. (In case this is Tracking-Neverending; if it is only Epic and can be resolved at some point please ignore this comment.)
Comment Actions
Ah. "Large uploads to Commons are broken and need to be fixed" seems like a good Epic, when it manifests in multiple ways (see subtasks) and will require multiple interventions to resolve (moving imginfo to bulk storage, moving OCR out of metadata, "do something" so the job queue doesn't choke on publishing these out of stash, etc.). That satisfies the "can be fixed, even if it takes time" criterion.
I presume the intent is to actually try to get this fixed, and not just document the problems' existence?
Comment Actions
Technically these issues can be resolved but my impression is that nobody is capable and interested in working on them. It affects quite some people directly (the uploaders) and ultimately hurts the wikis as educational content (like a big PDF, very high resolution image or a video) can be difficult or impossible to upload. Perhaps it'll never be resolved and these subtasks have been open for way too long, but that's not because it's unresolvable. The goals are clear.
Comment Actions
Reporting on behalf of my colleagues from WMDE Policies team
WMDE is also affected by the big size file problem. It seems it might be related to some of already related sub-tasks, but we’d leave it for the experts at the WMF to identify.
In our cooperation with the German public broadcaster Terra X (see the session at Wikimania here for more information: https://www.youtube.com/watch?v=XOrcvJjoOwI), the community uttered the demand of clips at around 15 seconds up to 2 minutes. The ZDF (broadcaster) Team has uploaded about 200 files so far, all high quality, and would continue to provide content in 2k / as high quality as possible.
You can see the files and file usage here: https://mvc.toolforge.org/index.php?category=Videos_by_Terra_X×pan=now-30&rangestart=&rangeend=&limit=200
The community feedback is very good. The files have been used around multiple WP language versions and been featured as media of the day repeatedly. Therefore we intend to continue with the cooperation with the broadcaster, hoping to also include longer videos, and in higher quality.
At the moment Terra-X editors tend to send clips only, so they have time to solve the problem. But if they switch to UHD quality, and produce longer interview sequences, we would face a real problem in our cooperation, as no uploads would really be possible.
Few observations from our side, also based on the input from Wikipedia and Commons editors who work with us on our project:
- Upload from about 600 MiB still allows entering info about the file, but fails later on
- Error from file size of more than 1 GiB appears directly after the actual upload -- the upload does not even start?
- Uploaded video quality: 4K, at a data rate of 100 MBit/s. We would reach the critical file size limit (of 600 MiB) after a film length of about 75 seconds -- we certainly intend to include longer videos too.
- Error always appears in our manual attempts (since estimated April 2021)
- Upload method, upload OS and browser all lead to the same result and error
- Error can be reproduced with any sufficiently large file
- We’ve heard from Wikipedia/Commons volunteers that besides pictures and videos also PDFs are affected
- Apparently only the file size is the problem: gigapixel images, which are small enough in file size, can be uploaded and published
- database "local-swift-codfw" seems to be a central problem
- Before (earlier this year?), a big fileupload seemed to have been problematic only via the "usual" upload interface (Upload Wizard). We're told that recently system admins seem to have problems via the server-side upload as well
Having functional upload is essential for WMDE’s pilot programme with German public broadcaster. Therefore we’d appreciate looking into this problem.
Comment Actions
Again a failed upload at https://commons.wikimedia.org/wiki/File:Nouveau_Larousse_illustr%C3%A9,_1898,_IV.djvu (upload-by-url) from https://archive.org/download/nouveaularoussei04laro/nouveaularoussei04laro.djvu ( https://archive.org/details/nouveaularoussei04laro ).
DjVu file, 125 MB.
Note that other uploads methods also failed.
Request from 90.112.25.87 via cp3054 cp3054, Varnish XID 154633797 Error: 503, Backend fetch failed at Sun, 24 Oct 2021 08:49:27 GMT
Comment Actions
Again a failed upload from https://archive.org/details/TheLostWorld1925/The+Lost+World+1925.mp4 (upload-by-url).
Direct link: https://archive.org/download/TheLostWorld1925/The%20Lost%20World%201925.ogv
Request from 90.112.25.87 via cp3054 cp3054, Varnish XID 661985376 Error: 503, Backend fetch failed at Sun, 24 Oct 2021 13:28:02 GMT
Comment Actions
Upload even failed for a 77 MB TIFF file from https://archive.org/details/clevelandart-1988.58-child-s-throne
DIrect link: https://archive.org/download/clevelandart-1988.58-child-s-throne/1988.58_full.tif
Request from 90.112.25.87 via cp3062 cp3062, Varnish XID 196697078 Error: 503, Backend fetch failed at Tue, 26 Oct 2021 21:15:55 GMT
Comment Actions
Upload failed for 491 MB TIFF file from https://archive.org/details/clevelandart-1947.196-low-tide-at-pourvill
Direct link: https://archive.org/download/clevelandart-1947.196-low-tide-at-pourvill/1947.196_full.tif
Request from 90.112.25.87 via cp3062 cp3062, Varnish XID 330016222 Error: 503, Backend fetch failed at Wed, 27 Oct 2021 16:26:44 GMT
Comment Actions
Upload failed for 459 MB TIFF file from https://archive.org/details/clevelandart-1958.32-two-poplars-in-the-a
Direct link: https://archive.org/download/clevelandart-1958.32-two-poplars-in-the-a/1958.32_full.tif
Request from 90.112.25.87 via cp3062 cp3062, Varnish XID 438573168 Error: 503, Backend fetch failed at Wed, 27 Oct 2021 17:48:58 GMT
Comment Actions
watching, we are also experiencing problems its preventing uploading of all the recorded session from Wikimania 2021
Comment Actions
Hi, we think this issue is partially a symptom of a regression we've been trying to track down for some time and that happened after our last round of operating system updates on the MediaWiki servers, T275752. We finally tracked it down (thanks to @Xover for noticing the smoking gun there) and we hope to get a fix written today and deployed to production next week.
That peformance degradation exposes a larger issue, IMHO, which is wrapping file uploads in a database transaction; but fixing this regression should move the needle of the maximum file you can upload quite a bit, and thus "fix" the issue for most practical cases, we hope.
Comment Actions
Upload failed for a 114 MB TIFF file from https://archive.org/details/clevelandart-1929.465-the-cliff-at-etretat
Direct link: https://archive.org/download/clevelandart-1929.465-the-cliff-at-etretat/1929.465_full.tif
Request from 90.112.25.87 via cp3056 cp3056, Varnish XID 319322344 Error: 503, Backend fetch failed at Fri, 29 Oct 2021 20:56:49 GMT
Comment Actions
Upload failed for a 70 MB JPEG file from https://archive.org/details/clevelandart-1916.1044-gardener-s-house-at
Direct link: https://archive.org/download/clevelandart-1916.1044-gardener-s-house-at/1916.1044_full.jpg
Request from 90.112.25.87 via cp3056 cp3056, Varnish XID 409960770 Error: 503, Backend fetch failed at Fri, 29 Oct 2021 21:06:31 GMT
Comment Actions
Can you provide a bit more detail so I can find this in the logs? Specifically, which wiki did you try uploading to? What tool were you using, the classic Special:Upload or UploadWizard, or something else? At what point in the upload process did you see this error?
Comment Actions
Using bigChunkedUpload.js I uploaded a 921MB video:
01006: 224/231> Chunk uploaded 01006: 225/231> in progress Upload: 99% 01011: 225/231> Chunk uploaded 01011: 226/231> in progress Upload: 100% 01016: 226/231> Chunk uploaded 01016: 227/231> in progress Upload: 99% 01020: 227/231> Chunk uploaded 01020: 228/231> in progress Upload: 100% 01024: 228/231> Chunk uploaded 01025: 229/231> in progress Upload: 100% 01029: 229/231> Chunk uploaded 01029: 230/231> in progress Upload: 98% 01033: 230/231> Chunk uploaded 01033: 231/231> in progress Upload: 93% 01043: 231/231> upload is stuck 01044: 231/231> Connection seems to be okay. Waiting one more time... 01048: 231/231> upload is stuck 01049: 231/231> Connection seems to be okay. Waiting one more time... 01053: 231/231> upload is stuck 01054: 231/231> Connection seems to be okay. Waiting one more time... 01058: 231/231> upload is stuck 01059: 231/231> Connection seems to be okay. Waiting one more time... 01063: 231/231> upload is stuck 01064: 231/231> Connection seems to be okay. Waiting one more time... 01068: 231/231> upload is stuck 01069: 231/231> Connection seems to be okay. Waiting one more time... 01073: 231/231> upload is stuck 01074: 231/231> Connection seems to be okay. Waiting one more time... 01078: 231/231> upload is stuck 01079: 231/231> Connection seems to be okay. Waiting one more time... 01083: 231/231> upload is stuck 01084: 231/231> Connection seems to be okay. Waiting one more time... 01088: 231/231> upload is stuck 01089: 231/231> Connection seems to be okay. Waiting one more time... 01093: 231/231> upload is stuck 01094: 231/231> Server error 0 after uploading chunk: Response: 01094: 231/231> Connection seems to be okay. Re-sending this request. 01094: 231/231> Connection seems to be okay. Re-sending this request. Upload: 96% 01095: FAILED: uploadstash-file-not-found: Key "18qfe2nwiafg.dx1sfn.2521907.webm" not found in stash.
Special:UploadStash showed:
18qfff9yoto0.mcayum.2521907.webm (view thumbnail)
I entered this:
var a = new mw.Api(); a.postWithToken( 'edit', { action: 'upload', filekey: '18qfff9yoto0.mcayum.2521907.webm', 'filename': 'David Pakman - Politics, media and reality tunnels.webm' } );And got an empty response. The file was successfully published to File:David Pakman - Politics, media and reality tunnels.webm. I've since moved it to File:David Pakman talks to Rebel Wisdom about politics and media.webm because I discovered the similarly named File:David Pakman - Politics, Media and Reality Tunnels.webm already existed.
Why the UploadStash key changed? Beats me
Comment Actions
Just noting that I had a lot of trouble uploading a large file: y2commons didn't work, ChunkedUpload didn't work, publishing from my stash didn't work (a half-dozen times), and one last try at ChunkedUpload/stash failed so badly that the special page for my stash wouldn't load. I have no clue what is up. Another user helped me by uploading the file to en.wp but it needs to be ported over to Commons and overwrite the malformed version there: https://en.wikipedia.org/wiki/File:David_Pakman_talks_to_Rebel_Wisdom_about_politics_and_media.webm
Can anyone do that?
*Edit*: Oh, the above.
Comment Actions
Can you provide a bit more detail so I can find this in the logs? Specifically, which wiki did you try uploading to? What tool were you using, the classic Special:Upload or UploadWizard, or something else? At what point in the upload process did you see this error?
And that is the problem: As a commons user, you do not get this details. Only Rillke's chunked upload script does actually give the details. The UploadWizard is specifically bad at that. A year ago it included "$1" in error messages, obviously a variable that was supposed to filled in with some information, but didn't. The fix chosen by developers was to remove the "$1". Over the years I opened a number of tasks, some of them including information from the Rillke script and IDs from Special:UploadStash, only to get them closed again and again for not being reproducable. And now here is a request for "more detail". And from Joe the information, that it is a known problem (known to deveopers), while there is neither a message in UploadWizard ("The Wizard is broken: Do not even try to upload files bigger than 50MB, you will only waste your time"), nor can this info be found at any help page in Forum, Village Pump or were ever.
If you really want to find out: T286532 T286481 T286430 have a large number of videos that need revsion upload because it was not possible to upload the complete file. You can pick any of them and try to upload, than you will get "more detail" as much as you want or may ever need.
Comment Actions
To provide context I should have done earlier, at ~21:20 UTC yesterday, we fixed *one* of the problems causing large file uploads to timeout. So far I know uploads that previous failed that were ~800MB have succeeded now. I'm relatively confident that most large uploads that previously would have worked, will work now. Stuff closer to the 4GB limit might still need server-side uploads.
Whatever information the tool you're using spits out is fine, screenshots are OK too.
Over the years I opened a number of tasks, some of them including information from the Rillke script and IDs from Special:UploadStash, only to get them closed again and again for not being reproducable. And now here is a request for "more detail".
I can only speak for myself and not what people did in the past, I'm willing to spend time digging through the logs to attempt to identify what went wrong in specific uploads, in the hopes of identifying the problem. There are basically no developers responsible for file uploads, it's definitely not my responsibility, but since I spent the past few days figuring out how this works I'm offering to help with related issues.
And from Joe the information, that it is a known problem (known to deveopers), while there is neither a message in UploadWizard ("The Wizard is broken: Do not even try to upload files bigger than 50MB, you will only waste your time"), nor can this info be found at any help page in Forum, Village Pump or were ever.
It was only in the past few weeks that T275752 was linked to the large file problem, but yes, you're mostly right.
If you really want to find out: T286532 T286481 T286430 have a large number of videos that need revsion upload because it was not possible to upload the complete file. You can pick any of them and try to upload, than you will get "more detail" as much as you want or may ever need.
Thanks, I gave one of the ~600MB files a try and it uploaded just fine using Rillke's script: https://commons.wikimedia.org/wiki/File:CSD_Hannover_2021_07.webm - I suspect your other files will work too now.
Comment Actions
Seems like T278104: Unable to upload to Commons: uploadstash-file-not-found: Key "187kyl5ozj74.xtav8j.51508.djvu" not found in stash. I'll try to follow-up on that on Monday
Comment Actions
I figured someone with access to FileImporter could do that, but I was mistaken. Personally I lost the FileExporter link a long time ago and never used it before I got blocked on Commons, so I hadn't even checked if it would produce any error. It's supposed to be an option in https://en.wikipedia.org/wiki/Special:Preferences#mw-prefsection-betafeatures but for me it isn't. I figured out how to create the link manually and as it turns out, FileImporter can't handle it: The file you are currently trying to import exceeds the file size limit the import can handle. So you really do need someone to do this server-side. I guess you should open a new ticket for that.
I'm curious, I'll keep an eye on it.
Comment Actions
In T292954#7469506, @C.Suthorn wrote:
And that is the problem: As a commons user, you do not get this details. Only Rillke's chunked upload script does actually give the details. The UploadWizard is specifically bad at that. A year ago it included "$1" in error messages, obviously a variable that was supposed to filled in with some information, but didn't. The fix chosen by developers was to remove the "$1".
That looks to be T208186: UploadWizard: Unknown error: "$1".
Comment Actions
Ok, reupload of a previously problematic file—c:File:Frank Leslie's Illustrated Newspaper Vol. 18.djvu—using Rillke's BCU.
Upload time (client to… whatever component it's talking to): 844s (~14 minutes)
Total time (inc. reassembly and publish): 966s (~16 minutes)
Given the file is 653,585,468 bytes, that gives an actual network transfer speed of 756 kBps (which is not great, but within the reasonable range for public internet) and a net throughput of 660 kBps including all associated overhead (per-chunk headers etc.; reassembly of chunks; etc. etc.). Most crucially in this context, almost all the time was spent in the public-internet network transfer: only 2 out of 16 total minutes was spent server-side.
Previous reupload attempts of this file have also emitted error messages after each chunk ("Upload appears to be stuck."; probably Rillke's code timing out waiting for an ack from somewhere) that were not in evidence now. Previous upload attempts have hard-failed somewhere in that reassemble/publish phase with various generic messages roughly consistent with hitting the 300s DB timeout. Representative example from a different file back in May:
Based on this (one) sample it looks like the most critical problem is fixed, and the typical "large upload" cases that have been broken since (probably) the Buster migration will now work. But those 2 minutes server-side are worryingly long, and it seems likely that at somewhere above 1GB uploads will start hitting that 300s DB timeout. I've had ~2 real-world cases of files in that size class that I can recall so from my perspective that's relatively rare, but something like T283045 that takes that timeout out of the equation would still be welcome (well, not T283045 exactly, since that's addressing a different aspect I think, but something that doesn't wrap this inherently time-consuming task inside a DB transaction).
However, there seems to now be a (probably unrelated) issue: for this reupload, the thumbnails of the previous and current revisions are swapped. The previous version of the file was a heavily compressed bitonal version, and the new upload was a full-colour version. The currently displayed thumbnails show the old version in colour and the new version as black and white. This could be due to the old broken revisions of the file (the 0x0 / 20MB revs) making something (Thumbor?) confused. Those old borked versions were manually published from the stash after a failed upload with BCU, so somewhere there's a code path for a failed upload to end up partially in the stash when these timeouts occur.
Comment Actions
I was able to upload one 3.8GiB file with the Rillke script.
Then I tried to upload 8 files with a different tool. 5 failed (not found in upload stash), 3 uploaded (the smallest files?)
Comment Actions
What was the other tool? Does it have any logs anywhere? What were the file names you attempted to upload to (it can help the developers find relevant log entries)? And was this to Commons or directly to another wiki?
I note that @AlexisJazz also had an issue with the stash key changing. Could it be that there is a separate issue that affects the upload stash? @C.Suthorn Do the 5 files show up if you open your UploadStash? If so, could you try using EnhancedStash to publish them (don't forget to set a sensible filename: EnhancedStash uses a nonsense file name by default)?
Comment Actions
Adding more and more random problems as comments makes this an even more unactionable catch-all ticket. Please don't do this. Thank you. Edit: Looks like this is mostly debugging; so I apologize for my previous comment here.
Comment Actions
Ok, just reuploaded a 1.67GB (1,792,272,952 bytes) file that failed a week and a half ago (it took literally 24 hours to regenerate it from scans: I really need to get a faster laptop): c:File:A Latin Dictionary (1984).djvu
Just upload took 2,088 seconds (838 kBps); reassembling chunks took another 148 seconds; and publishing the file another 178 seconds. Total upload time was 2,414 seconds, with a net throughput of 725 kBps. However, the server-side part of that had an overall throughput of 5,2 MBps (for the networking wonks, that's ~42Mbps average throughput), including disk IO, any network transfers (e.g. jobrunner->swift), CPU, and, I'm pretty sure, DB transactions to insert metadata (which for DjVu includes stuffing all 2k pages worth of OCR into imginfo).
From my perspective this represents a "torture test" scenario, and if it handled this upload with essentially "flying colours" then reliability and performance generally is back to where it was before. I'm not super impressed with the performance in absolute terms, but it's definitely not "broken" either.
Looking through the connected tasks, all of them could be the same issue; but some of them could also be separate issues. I've poked a couple of them to retest.
@Aklapper No worries. We appreciate you keeping us all in line so Phabricator doesn't descend into chaos and anarchy. :)
Comment Actions
I tried again to upload a 491 MB TIFF file from https://archive.org/details/clevelandart-1916.1044-gardener-s-house-at and it failed
Direct link: https://archive.org/download/clevelandart-1916.1044-gardener-s-house-at/1916.1044_full.tif
(upload-by-url on Commons)
Request from 90.112.25.87 via cp3064 cp3064, Varnish XID 452004418 Error: 503, Backend fetch failed at Sun, 31 Oct 2021 20:15:12 GMT
Comment Actions
I managed this one with video2commons: https://commons.wikimedia.org/wiki/File:The_Lost_World_(1925),_full.ogv
Comment Actions
just noting, I had success with 925 MB https://commons.wikimedia.org/wiki/File:Wikimedia_Australia,_Wikimania_2021.webm uploaded using upload wizard, at about 1430utc 31 October. I did skip the structured data step
Comment Actions
This one is still failing:
Request from 90.112.25.87 via cp3064 cp3064, Varnish XID 741901089 Error: 503, Backend fetch failed at Sun, 31 Oct 2021 23:56:48 GMT
Comment Actions
T283045 does propose to stop wrapping file operations in a DB transaction. I have some time to work on it now.
Comment Actions
I was able to do ~250 successful revision uploads, but for 150 more files the upload is not possible (curid):
74160702, 74305338, 74998198, 74998202, 74998207, 74998229, 74998236, 75622881, 75762675, 76320869, 76604256, 76822922, 77651537, 79066814, 79262554, 79262558, 80015487, 82427597, 82891884, 82891887, 82891912, 82891923, 82895325, 83356907, 83739265, 83741205, 84406251, 84420489, 84420539, 84420562,
84505710, 84696096, 84735599, 84735603, 84787937, 84787940, 84790317, 84790326, 84798280, 84809610, 84809612, 84841545, 84923003, 84923593, 84923595, 85015693, 85026363, 85026373, 85043207, 85043217, 85043226, 85043239, 85043249, 85043260, 85092579, 85153961, 85612817, 85612818, 85796109, 85796113,
85796115, 86032395, 86032400, 86032401, 86032405, 86032411, 86032421, 86032429, 86278842, 87111884, 87111886, 87111898, 87111899, 87111929, 87111934, 87155652, 87155656, 87371038, 87444333, 87552630, 87552632, 87552635, 87859718, 87859719, 87859761, 89994603, 89994608, 89994612, 89994615, 89994623,
89994628, 89994630, 91019765, 91232251, 91232257, 91232260, 91232263, 91232265, 91303877, 91487119, 91487120, 91487124, 91487153, 91487163, 91487256, 91684325, 91684333, 91693717, 91693722, 91693760, 91849584, 91918042, 91918106, 91918184, 91924057, 91924197, 91953165, 91953196, 92168243, 92168271,
92170612, 92170613, 92170620, 92306453, 92435328, 92435331, 92435519, 92435526, 92435533, 92435626, 92435638, 92435677, 92454212, 92454278, 92454312, 92635789, 92635809, 92635859, 92635866, 92635904, 92668410, 93018756, 93018767, 93018875, 93018890, 93018991, 93019034, 93019055, 93019250, 93019360,
93019361
All are from Task T282755 , @Urbanecm copied them to /data/scratch/urbanecm-T282755
Comment Actions
Hi, While trying large file upload on test.wikipedia.org via upload-by-url, I came into 2 issues:
- Uploading from Internet Archive is not possible: Copy uploads are not available from this domain.
- Uploading from Commons, I get This action has been automatically identified as harmful, and therefore disallowed. If you believe your action was constructive, please inform an administrator of what you were trying to do. A brief description of the abuse rule which your action matched is: 0 copyvios.
Comment Actions
MediaWiki has a 180 second (3 minutes) timeout that it has to download the file in. It took me ~4 minutes to download the file at home, and AFAICT the bandwidth limitation is not on my side. On a production appserver it's even slower, currently saying it's going to take 15 minutes (I wonder if they're rate limiting us?) - see P17649. Now, even if the file does download in 180 seconds, there's a 200 second overall MediaWiki timeout. So it would need to upload to Swift, extract metadata, and update the databases in 20 seconds, which is cutting it pretty close. Unfortunately there's no chunked-upload-by-url system I'm aware of, in the meantime I think the best solution is to download the file locally or to Toolforge and then chunked upload to Commons.
Comment Actions
And the reason you're getting a cryptic, generic error message instead of one that actually tells you the real problem is because the Varnish/ATS timeout appears to be 180 seconds, so that kicks in before MediaWiki's own timeout kicks in. I filed T294800: Reconcile MediaWiki POST timeout and Varnish/ATS timeouts for that.
Comment Actions
I think an even better solution is to significantly raise the timeout until some chunked-upload-by-url system is created.
Archive.org is not whitelisted in InitialiseSettings.php (wgCopyUploadsDomains) for testwiki, but upload.wikimedia.org is. The second error seems to be hitting the 0 copyvios abuse filter. You have only 5 edits on testwiki so you just need to spam more to get 10.
Comment Actions
https://wikisource-bot.toolforge.org/ssu_request/1916.1044_full.tif
@Yann that should download much faster.
The IA downloads are incredibly slow for me: sometimes it an hour to get a sub-1GB ZIP.
Comment Actions
Chunked upload is fine, but the problem that happens to me is that after a file is uploaded and the chunks are assembled, the publishing fails. So the timeout will happen wether the upload itself is chunked or not. Rillke's tool may sometimes succeed - i guess - because it polls the publishing process from time to time, thereby reseting the mediawiki timeout??
Comment Actions
I've also filed:
- T295007: Upload by URL should use the job queue, possibly chunked with range requests
- T295008: MediaWiki uploads files to Swift in eqiad and codfw in serial, not parallel
- T295009: Improve download speed from archive.org on appservers
I think we are reaching the end of the usefulness of this debugging ticket soon, AFAICT all of the high level issues have been identified, they just lack someone to work on them.
Comment Actions
I think we are reaching the end of the usefulness of this debugging ticket soon
Bolding closing, It's been over a year since the last action a sub task and nothing new has been attached to this tracking task.
Going with invalis more than resolved as this task was closiong to a tracking task, and overall Large uploads probably won't ever be fully solved but seperate tasks for each issue will work better.