Page MenuHomePhabricator

[epic] large file uploads to commons
Open, Needs TriagePublic

Description

This is a tracking task to collect all subtasks related to issues with large file uploads to commons.

Related Objects

Event Timeline

There are a very large number of changes, so older changes are hidden. Show Older Changes

Neverending tracking tasks should not be created as it means that they'll never get resolved. Dedicated project tags should be used instead. (In case this is Tracking-Neverending; if it is only Epic and can be resolved at some point please ignore this comment.)

right... it was so nice to see all the subtasks in the graph :-(

Feel free to close!

Ah. "Large uploads to Commons are broken and need to be fixed" seems like a good Epic, when it manifests in multiple ways (see subtasks) and will require multiple interventions to resolve (moving imginfo to bulk storage, moving OCR out of metadata, "do something" so the job queue doesn't choke on publishing these out of stash, etc.). That satisfies the "can be fixed, even if it takes time" criterion.

I presume the intent is to actually try to get this fixed, and not just document the problems' existence?

Neverending tracking tasks should not be created as it means that they'll never get resolved. Dedicated project tags should be used instead. (In case this is Tracking-Neverending; if it is only Epic and can be resolved at some point please ignore this comment.)

Technically these issues can be resolved but my impression is that nobody is capable and interested in working on them. It affects quite some people directly (the uploaders) and ultimately hurts the wikis as educational content (like a big PDF, very high resolution image or a video) can be difficult or impossible to upload. Perhaps it'll never be resolved and these subtasks have been open for way too long, but that's not because it's unresolvable. The goals are clear.

Reporting on behalf of my colleagues from WMDE Policies team
WMDE is also affected by the big size file problem. It seems it might be related to some of already related sub-tasks, but we’d leave it for the experts at the WMF to identify.
In our cooperation with the German public broadcaster Terra X (see the session at Wikimania here for more information: https://www.youtube.com/watch?v=XOrcvJjoOwI), the community uttered the demand of clips at around 15 seconds up to 2 minutes. The ZDF (broadcaster) Team has uploaded about 200 files so far, all high quality, and would continue to provide content in 2k / as high quality as possible.
You can see the files and file usage here: https://mvc.toolforge.org/index.php?category=Videos_by_Terra_X&timespan=now-30&rangestart=&rangeend=&limit=200
The community feedback is very good. The files have been used around multiple WP language versions and been featured as media of the day repeatedly. Therefore we intend to continue with the cooperation with the broadcaster, hoping to also include longer videos, and in higher quality.
At the moment Terra-X editors tend to send clips only, so they have time to solve the problem. But if they switch to UHD quality, and produce longer interview sequences, we would face a real problem in our cooperation, as no uploads would really be possible.

Few observations from our side, also based on the input from Wikipedia and Commons editors who work with us on our project:

  • Upload from about 600 MiB still allows entering info about the file, but fails later on
  • Error from file size of more than 1 GiB appears directly after the actual upload -- the upload does not even start?
  • Uploaded video quality: 4K, at a data rate of 100 MBit/s. We would reach the critical file size limit (of 600 MiB) after a film length of about 75 seconds -- we certainly intend to include longer videos too.
  • Error always appears in our manual attempts (since estimated April 2021)
  • Upload method, upload OS and browser all lead to the same result and error
  • Error can be reproduced with any sufficiently large file
  • We’ve heard from Wikipedia/Commons volunteers that besides pictures and videos also PDFs are affected
  • Apparently only the file size is the problem: gigapixel images, which are small enough in file size, can be uploaded and published
  • database "local-swift-codfw" seems to be a central problem
  • Before (earlier this year?), a big fileupload seemed to have been problematic only via the "usual" upload interface (Upload Wizard). We're told that recently system admins seem to have problems via the server-side upload as well

Having functional upload is essential for WMDE’s pilot programme with German public broadcaster. Therefore we’d appreciate looking into this problem.

Again a failed upload at https://commons.wikimedia.org/wiki/File:Nouveau_Larousse_illustr%C3%A9,_1898,_IV.djvu (upload-by-url) from https://archive.org/download/nouveaularoussei04laro/nouveaularoussei04laro.djvu ( https://archive.org/details/nouveaularoussei04laro ).
DjVu file, 125 MB.
Note that other uploads methods also failed.

Request from 90.112.25.87 via cp3054 cp3054, Varnish XID 154633797
Error: 503, Backend fetch failed at Sun, 24 Oct 2021 08:49:27 GMT

Again a failed upload from https://archive.org/details/TheLostWorld1925/The+Lost+World+1925.mp4 (upload-by-url).
Direct link: https://archive.org/download/TheLostWorld1925/The%20Lost%20World%201925.ogv

Request from 90.112.25.87 via cp3054 cp3054, Varnish XID 661985376
Error: 503, Backend fetch failed at Sun, 24 Oct 2021 13:28:02 GMT

Upload even failed for a 77 MB TIFF file from https://archive.org/details/clevelandart-1988.58-child-s-throne
DIrect link: https://archive.org/download/clevelandart-1988.58-child-s-throne/1988.58_full.tif

Request from 90.112.25.87 via cp3062 cp3062, Varnish XID 196697078
Error: 503, Backend fetch failed at Tue, 26 Oct 2021 21:15:55 GMT

Upload failed for 491 MB TIFF file from https://archive.org/details/clevelandart-1947.196-low-tide-at-pourvill
Direct link: https://archive.org/download/clevelandart-1947.196-low-tide-at-pourvill/1947.196_full.tif

Request from 90.112.25.87 via cp3062 cp3062, Varnish XID 330016222
Error: 503, Backend fetch failed at Wed, 27 Oct 2021 16:26:44 GMT

Upload failed for 459 MB TIFF file from https://archive.org/details/clevelandart-1958.32-two-poplars-in-the-a
Direct link: https://archive.org/download/clevelandart-1958.32-two-poplars-in-the-a/1958.32_full.tif

Request from 90.112.25.87 via cp3062 cp3062, Varnish XID 438573168
Error: 503, Backend fetch failed at Wed, 27 Oct 2021 17:48:58 GMT

watching, we are also experiencing problems its preventing uploading of all the recorded session from Wikimania 2021

Hi, we think this issue is partially a symptom of a regression we've been trying to track down for some time and that happened after our last round of operating system updates on the MediaWiki servers, T275752. We finally tracked it down (thanks to @Xover for noticing the smoking gun there) and we hope to get a fix written today and deployed to production next week.

That peformance degradation exposes a larger issue, IMHO, which is wrapping file uploads in a database transaction; but fixing this regression should move the needle of the maximum file you can upload quite a bit, and thus "fix" the issue for most practical cases, we hope.

Upload failed for a 114 MB TIFF file from https://archive.org/details/clevelandart-1929.465-the-cliff-at-etretat
Direct link: https://archive.org/download/clevelandart-1929.465-the-cliff-at-etretat/1929.465_full.tif

Request from 90.112.25.87 via cp3056 cp3056, Varnish XID 319322344
Error: 503, Backend fetch failed at Fri, 29 Oct 2021 20:56:49 GMT

Upload failed for a 70 MB JPEG file from https://archive.org/details/clevelandart-1916.1044-gardener-s-house-at
Direct link: https://archive.org/download/clevelandart-1916.1044-gardener-s-house-at/1916.1044_full.jpg

Request from 90.112.25.87 via cp3056 cp3056, Varnish XID 409960770
Error: 503, Backend fetch failed at Fri, 29 Oct 2021 21:06:31 GMT

Upload failed for a 70 MB JPEG file from https://archive.org/details/clevelandart-1916.1044-gardener-s-house-at
Direct link: https://archive.org/download/clevelandart-1916.1044-gardener-s-house-at/1916.1044_full.jpg

Request from 90.112.25.87 via cp3056 cp3056, Varnish XID 409960770
Error: 503, Backend fetch failed at Fri, 29 Oct 2021 21:06:31 GMT

Can you provide a bit more detail so I can find this in the logs? Specifically, which wiki did you try uploading to? What tool were you using, the classic Special:Upload or UploadWizard, or something else? At what point in the upload process did you see this error?

Using bigChunkedUpload.js I uploaded a 921MB video:

01006: 224/231> Chunk uploaded
01006: 225/231> in progress Upload: 99%
01011: 225/231> Chunk uploaded
01011: 226/231> in progress Upload: 100%
01016: 226/231> Chunk uploaded
01016: 227/231> in progress Upload: 99%
01020: 227/231> Chunk uploaded
01020: 228/231> in progress Upload: 100%
01024: 228/231> Chunk uploaded
01025: 229/231> in progress Upload: 100%
01029: 229/231> Chunk uploaded
01029: 230/231> in progress Upload: 98%
01033: 230/231> Chunk uploaded
01033: 231/231> in progress Upload: 93%
01043: 231/231> upload is stuck
01044: 231/231> Connection seems to be okay. Waiting one more time...
01048: 231/231> upload is stuck
01049: 231/231> Connection seems to be okay. Waiting one more time...
01053: 231/231> upload is stuck
01054: 231/231> Connection seems to be okay. Waiting one more time...
01058: 231/231> upload is stuck
01059: 231/231> Connection seems to be okay. Waiting one more time...
01063: 231/231> upload is stuck
01064: 231/231> Connection seems to be okay. Waiting one more time...
01068: 231/231> upload is stuck
01069: 231/231> Connection seems to be okay. Waiting one more time...
01073: 231/231> upload is stuck
01074: 231/231> Connection seems to be okay. Waiting one more time...
01078: 231/231> upload is stuck
01079: 231/231> Connection seems to be okay. Waiting one more time...
01083: 231/231> upload is stuck
01084: 231/231> Connection seems to be okay. Waiting one more time...
01088: 231/231> upload is stuck
01089: 231/231> Connection seems to be okay. Waiting one more time...
01093: 231/231> upload is stuck
01094: 231/231> Server error 0 after uploading chunk: 
Response: 
01094: 231/231> Connection seems to be okay. Re-sending this request.
01094: 231/231> Connection seems to be okay. Re-sending this request. Upload: 96%
01095: FAILED: uploadstash-file-not-found: Key "18qfe2nwiafg.dx1sfn.2521907.webm" not found in stash.

Special:UploadStash showed:

18qfff9yoto0.mcayum.2521907.webm (view thumbnail)

I entered this:

var a = new mw.Api(); a.postWithToken( 'edit', { action: 'upload', filekey: '18qfff9yoto0.mcayum.2521907.webm', 'filename': 'David Pakman - Politics, media and reality tunnels.webm' } );

And got an empty response. The file was successfully published to File:David Pakman - Politics, media and reality tunnels.webm. I've since moved it to File:David Pakman talks to Rebel Wisdom about politics and media.webm because I discovered the similarly named File:David Pakman - Politics, Media and Reality Tunnels.webm already existed.

Why the UploadStash key changed? Beats me

Just noting that I had a lot of trouble uploading a large file: y2commons didn't work, ChunkedUpload didn't work, publishing from my stash didn't work (a half-dozen times), and one last try at ChunkedUpload/stash failed so badly that the special page for my stash wouldn't load. I have no clue what is up. Another user helped me by uploading the file to en.wp but it needs to be ported over to Commons and overwrite the malformed version there: https://en.wikipedia.org/wiki/File:David_Pakman_talks_to_Rebel_Wisdom_about_politics_and_media.webm

Can anyone do that?

*Edit*: Oh, the above.

Can you provide a bit more detail so I can find this in the logs? Specifically, which wiki did you try uploading to? What tool were you using, the classic Special:Upload or UploadWizard, or something else? At what point in the upload process did you see this error?

And that is the problem: As a commons user, you do not get this details. Only Rillke's chunked upload script does actually give the details. The UploadWizard is specifically bad at that. A year ago it included "$1" in error messages, obviously a variable that was supposed to filled in with some information, but didn't. The fix chosen by developers was to remove the "$1". Over the years I opened a number of tasks, some of them including information from the Rillke script and IDs from Special:UploadStash, only to get them closed again and again for not being reproducable. And now here is a request for "more detail". And from Joe the information, that it is a known problem (known to deveopers), while there is neither a message in UploadWizard ("The Wizard is broken: Do not even try to upload files bigger than 50MB, you will only waste your time"), nor can this info be found at any help page in Forum, Village Pump or were ever.

If you really want to find out: T286532 T286481 T286430 have a large number of videos that need revsion upload because it was not possible to upload the complete file. You can pick any of them and try to upload, than you will get "more detail" as much as you want or may ever need.

To provide context I should have done earlier, at ~21:20 UTC yesterday, we fixed *one* of the problems causing large file uploads to timeout. So far I know uploads that previous failed that were ~800MB have succeeded now. I'm relatively confident that most large uploads that previously would have worked, will work now. Stuff closer to the 4GB limit might still need server-side uploads.

Can you provide a bit more detail so I can find this in the logs? Specifically, which wiki did you try uploading to? What tool were you using, the classic Special:Upload or UploadWizard, or something else? At what point in the upload process did you see this error?

And that is the problem: As a commons user, you do not get this details. Only Rillke's chunked upload script does actually give the details. The UploadWizard is specifically bad at that. A year ago it included "$1" in error messages, obviously a variable that was supposed to filled in with some information, but didn't. The fix chosen by developers was to remove the "$1".

Whatever information the tool you're using spits out is fine, screenshots are OK too.

Over the years I opened a number of tasks, some of them including information from the Rillke script and IDs from Special:UploadStash, only to get them closed again and again for not being reproducable. And now here is a request for "more detail".

I can only speak for myself and not what people did in the past, I'm willing to spend time digging through the logs to attempt to identify what went wrong in specific uploads, in the hopes of identifying the problem. There are basically no developers responsible for file uploads, it's definitely not my responsibility, but since I spent the past few days figuring out how this works I'm offering to help with related issues.

And from Joe the information, that it is a known problem (known to deveopers), while there is neither a message in UploadWizard ("The Wizard is broken: Do not even try to upload files bigger than 50MB, you will only waste your time"), nor can this info be found at any help page in Forum, Village Pump or were ever.

It was only in the past few weeks that T275752 was linked to the large file problem, but yes, you're mostly right.

If you really want to find out: T286532 T286481 T286430 have a large number of videos that need revsion upload because it was not possible to upload the complete file. You can pick any of them and try to upload, than you will get "more detail" as much as you want or may ever need.

Thanks, I gave one of the ~600MB files a try and it uploaded just fine using Rillke's script: https://commons.wikimedia.org/wiki/File:CSD_Hannover_2021_07.webm - I suspect your other files will work too now.

Using bigChunkedUpload.js I uploaded a 921MB video:

<snip>
01095: FAILED: uploadstash-file-not-found: Key "18qfe2nwiafg.dx1sfn.2521907.webm" not found in stash.

Seems like T278104: Unable to upload to Commons: uploadstash-file-not-found: Key "187kyl5ozj74.xtav8j.51508.djvu" not found in stash. I'll try to follow-up on that on Monday

Just noting that I had a lot of trouble uploading a large file: y2commons didn't work, ChunkedUpload didn't work, publishing from my stash didn't work (a half-dozen times), and one last try at ChunkedUpload/stash failed so badly that the special page for my stash wouldn't load. I have no clue what is up. Another user helped me by uploading the file to en.wp but it needs to be ported over to Commons and overwrite the malformed version there: https://en.wikipedia.org/wiki/File:David_Pakman_talks_to_Rebel_Wisdom_about_politics_and_media.webm

Can anyone do that?

*Edit*: Oh, the above.

I figured someone with access to FileImporter could do that, but I was mistaken. Personally I lost the FileExporter link a long time ago and never used it before I got blocked on Commons, so I hadn't even checked if it would produce any error. It's supposed to be an option in https://en.wikipedia.org/wiki/Special:Preferences#mw-prefsection-betafeatures but for me it isn't. I figured out how to create the link manually and as it turns out, FileImporter can't handle it: The file you are currently trying to import exceeds the file size limit the import can handle. So you really do need someone to do this server-side. I guess you should open a new ticket for that.

Using bigChunkedUpload.js I uploaded a 921MB video:

<snip>
01095: FAILED: uploadstash-file-not-found: Key "18qfe2nwiafg.dx1sfn.2521907.webm" not found in stash.

Seems like T278104: Unable to upload to Commons: uploadstash-file-not-found: Key "187kyl5ozj74.xtav8j.51508.djvu" not found in stash. I'll try to follow-up on that on Monday

I'm curious, I'll keep an eye on it.

In T292954#7469506, @C.Suthorn wrote:
And that is the problem: As a commons user, you do not get this details. Only Rillke's chunked upload script does actually give the details. The UploadWizard is specifically bad at that. A year ago it included "$1" in error messages, obviously a variable that was supposed to filled in with some information, but didn't. The fix chosen by developers was to remove the "$1".

That looks to be T208186: UploadWizard: Unknown error: "$1".

Ok, reupload of a previously problematic file—c:File:Frank Leslie's Illustrated Newspaper Vol. 18.djvu—using Rillke's BCU.

Upload time (client to… whatever component it's talking to): 844s (~14 minutes)
Total time (inc. reassembly and publish): 966s (~16 minutes)

Given the file is 653,585,468 bytes, that gives an actual network transfer speed of 756 kBps (which is not great, but within the reasonable range for public internet) and a net throughput of 660 kBps including all associated overhead (per-chunk headers etc.; reassembly of chunks; etc. etc.). Most crucially in this context, almost all the time was spent in the public-internet network transfer: only 2 out of 16 total minutes was spent server-side.

Previous reupload attempts of this file have also emitted error messages after each chunk ("Upload appears to be stuck."; probably Rillke's code timing out waiting for an ack from somewhere) that were not in evidence now. Previous upload attempts have hard-failed somewhere in that reassemble/publish phase with various generic messages roughly consistent with hitting the 300s DB timeout. Representative example from a different file back in May:

Screenshot 2020-05-05 at 12.35.18.jpg (1×2 px, 504 KB)

Based on this (one) sample it looks like the most critical problem is fixed, and the typical "large upload" cases that have been broken since (probably) the Buster migration will now work. But those 2 minutes server-side are worryingly long, and it seems likely that at somewhere above 1GB uploads will start hitting that 300s DB timeout. I've had ~2 real-world cases of files in that size class that I can recall so from my perspective that's relatively rare, but something like T283045 that takes that timeout out of the equation would still be welcome (well, not T283045 exactly, since that's addressing a different aspect I think, but something that doesn't wrap this inherently time-consuming task inside a DB transaction).

However, there seems to now be a (probably unrelated) issue: for this reupload, the thumbnails of the previous and current revisions are swapped. The previous version of the file was a heavily compressed bitonal version, and the new upload was a full-colour version. The currently displayed thumbnails show the old version in colour and the new version as black and white. This could be due to the old broken revisions of the file (the 0x0 / 20MB revs) making something (Thumbor?) confused. Those old borked versions were manually published from the stash after a failed upload with BCU, so somewhere there's a code path for a failed upload to end up partially in the stash when these timeouts occur.

I was able to upload one 3.8GiB file with the Rillke script.

Then I tried to upload 8 files with a different tool. 5 failed (not found in upload stash), 3 uploaded (the smallest files?)

Then I tried to upload 8 files with a different tool. 5 failed (not found in upload stash), 3 uploaded (the smallest files?)

What was the other tool? Does it have any logs anywhere? What were the file names you attempted to upload to (it can help the developers find relevant log entries)? And was this to Commons or directly to another wiki?

I note that @AlexisJazz also had an issue with the stash key changing. Could it be that there is a separate issue that affects the upload stash? @C.Suthorn Do the 5 files show up if you open your UploadStash? If so, could you try using EnhancedStash to publish them (don't forget to set a sensible filename: EnhancedStash uses a nonsense file name by default)?

Adding more and more random problems as comments makes this an even more unactionable catch-all ticket. Please don't do this. Thank you. Edit: Looks like this is mostly debugging; so I apologize for my previous comment here.

Ok, just reuploaded a 1.67GB (1,792,272,952 bytes) file that failed a week and a half ago (it took literally 24 hours to regenerate it from scans: I really need to get a faster laptop): c:File:A Latin Dictionary (1984).djvu

Just upload took 2,088 seconds (838 kBps); reassembling chunks took another 148 seconds; and publishing the file another 178 seconds. Total upload time was 2,414 seconds, with a net throughput of 725 kBps. However, the server-side part of that had an overall throughput of 5,2 MBps (for the networking wonks, that's ~42Mbps average throughput), including disk IO, any network transfers (e.g. jobrunner->swift), CPU, and, I'm pretty sure, DB transactions to insert metadata (which for DjVu includes stuffing all 2k pages worth of OCR into imginfo).

From my perspective this represents a "torture test" scenario, and if it handled this upload with essentially "flying colours" then reliability and performance generally is back to where it was before. I'm not super impressed with the performance in absolute terms, but it's definitely not "broken" either.

Looking through the connected tasks, all of them could be the same issue; but some of them could also be separate issues. I've poked a couple of them to retest.

@Aklapper No worries. We appreciate you keeping us all in line so Phabricator doesn't descend into chaos and anarchy. :)

@Legoktm All my reports above were about uploads by upload-by-url on Commons.

I tried again to upload a 491 MB TIFF file from https://archive.org/details/clevelandart-1916.1044-gardener-s-house-at and it failed
Direct link: https://archive.org/download/clevelandart-1916.1044-gardener-s-house-at/1916.1044_full.tif
(upload-by-url on Commons)

Request from 90.112.25.87 via cp3064 cp3064, Varnish XID 452004418
Error: 503, Backend fetch failed at Sun, 31 Oct 2021 20:15:12 GMT

Upload failed for a 70 MB JPEG file from https://archive.org/details/clevelandart-1916.1044-gardener-s-house-at
Direct link: https://archive.org/download/clevelandart-1916.1044-gardener-s-house-at/1916.1044_full.jpg

Request from 90.112.25.87 via cp3056 cp3056, Varnish XID 409960770
Error: 503, Backend fetch failed at Fri, 29 Oct 2021 21:06:31 GMT

I finally managed for this one: https://commons.wikimedia.org/wiki/File:Gardener%27s_House_at_Antibes,_by_Claude_Monet,_Cleveland_Museum_of_Art,_1916.1044.jpg

Again a failed upload from https://archive.org/details/TheLostWorld1925/The+Lost+World+1925.mp4 (upload-by-url).
Direct link: https://archive.org/download/TheLostWorld1925/The%20Lost%20World%201925.ogv

Request from 90.112.25.87 via cp3054 cp3054, Varnish XID 661985376
Error: 503, Backend fetch failed at Sun, 24 Oct 2021 13:28:02 GMT

I managed this one with video2commons: https://commons.wikimedia.org/wiki/File:The_Lost_World_(1925),_full.ogv

Upload even failed for a 77 MB TIFF file from https://archive.org/details/clevelandart-1988.58-child-s-throne
DIrect link: https://archive.org/download/clevelandart-1988.58-child-s-throne/1988.58_full.tif

Request from 90.112.25.87 via cp3062 cp3062, Varnish XID 196697078
Error: 503, Backend fetch failed at Tue, 26 Oct 2021 21:15:55 GMT

I managed this one: https://commons.wikimedia.org/wiki/File:Pierre-Marie_Balny_le_Jeune_-_Child%27s_Throne_-_1988.58_-_Cleveland_Museum_of_Art.tif

just noting, I had success with 925 MB https://commons.wikimedia.org/wiki/File:Wikimedia_Australia,_Wikimania_2021.webm uploaded using upload wizard, at about 1430utc 31 October. I did skip the structured data step

Upload failed for a 114 MB TIFF file from https://archive.org/details/clevelandart-1929.465-the-cliff-at-etretat
Direct link: https://archive.org/download/clevelandart-1929.465-the-cliff-at-etretat/1929.465_full.tif

Request from 90.112.25.87 via cp3056 cp3056, Varnish XID 319322344
Error: 503, Backend fetch failed at Fri, 29 Oct 2021 20:56:49 GMT

This one is still failing:

Request from 90.112.25.87 via cp3064 cp3064, Varnish XID 741901089
Error: 503, Backend fetch failed at Sun, 31 Oct 2021 23:56:48 GMT

something like T283045 that takes that timeout out of the equation would still be welcome (well, not T283045 exactly, since that's addressing a different aspect I think, but something that doesn't wrap this inherently time-consuming task inside a DB transaction).

T283045 does propose to stop wrapping file operations in a DB transaction. I have some time to work on it now.

I was able to do ~250 successful revision uploads, but for 150 more files the upload is not possible (curid):

74160702, 74305338, 74998198, 74998202, 74998207, 74998229, 74998236, 75622881, 75762675, 76320869, 76604256, 76822922, 77651537, 79066814, 79262554, 79262558, 80015487, 82427597, 82891884, 82891887, 82891912, 82891923, 82895325, 83356907, 83739265, 83741205, 84406251, 84420489, 84420539, 84420562,
84505710, 84696096, 84735599, 84735603, 84787937, 84787940, 84790317, 84790326, 84798280, 84809610, 84809612, 84841545, 84923003, 84923593, 84923595, 85015693, 85026363, 85026373, 85043207, 85043217, 85043226, 85043239, 85043249, 85043260, 85092579, 85153961, 85612817, 85612818, 85796109, 85796113,
85796115, 86032395, 86032400, 86032401, 86032405, 86032411, 86032421, 86032429, 86278842, 87111884, 87111886, 87111898, 87111899, 87111929, 87111934, 87155652, 87155656, 87371038, 87444333, 87552630, 87552632, 87552635, 87859718, 87859719, 87859761, 89994603, 89994608, 89994612, 89994615, 89994623,
89994628, 89994630, 91019765, 91232251, 91232257, 91232260, 91232263, 91232265, 91303877, 91487119, 91487120, 91487124, 91487153, 91487163, 91487256, 91684325, 91684333, 91693717, 91693722, 91693760, 91849584, 91918042, 91918106, 91918184, 91924057, 91924197, 91953165, 91953196, 92168243, 92168271,
92170612, 92170613, 92170620, 92306453, 92435328, 92435331, 92435519, 92435526, 92435533, 92435626, 92435638, 92435677, 92454212, 92454278, 92454312, 92635789, 92635809, 92635859, 92635866, 92635904, 92668410, 93018756, 93018767, 93018875, 93018890, 93018991, 93019034, 93019055, 93019250, 93019360,
93019361

All are from Task T282755 , @Urbanecm copied them to /data/scratch/urbanecm-T282755

Hi, While trying large file upload on test.wikipedia.org via upload-by-url, I came into 2 issues:

  • Uploading from Internet Archive is not possible: Copy uploads are not available from this domain.
  • Uploading from Commons, I get This action has been automatically identified as harmful, and therefore disallowed. If you believe your action was constructive, please inform an administrator of what you were trying to do. A brief description of the abuse rule which your action matched is: 0 copyvios.

I tried again to upload a 491 MB TIFF file from https://archive.org/details/clevelandart-1916.1044-gardener-s-house-at and it failed
Direct link: https://archive.org/download/clevelandart-1916.1044-gardener-s-house-at/1916.1044_full.tif
(upload-by-url on Commons)

Request from 90.112.25.87 via cp3064 cp3064, Varnish XID 452004418
Error: 503, Backend fetch failed at Sun, 31 Oct 2021 20:15:12 GMT

MediaWiki has a 180 second (3 minutes) timeout that it has to download the file in. It took me ~4 minutes to download the file at home, and AFAICT the bandwidth limitation is not on my side. On a production appserver it's even slower, currently saying it's going to take 15 minutes (I wonder if they're rate limiting us?) - see P17649. Now, even if the file does download in 180 seconds, there's a 200 second overall MediaWiki timeout. So it would need to upload to Swift, extract metadata, and update the databases in 20 seconds, which is cutting it pretty close. Unfortunately there's no chunked-upload-by-url system I'm aware of, in the meantime I think the best solution is to download the file locally or to Toolforge and then chunked upload to Commons.

Request from 90.112.25.87 via cp3064 cp3064, Varnish XID 452004418
Error: 503, Backend fetch failed at Sun, 31 Oct 2021 20:15:12 GMT

MediaWiki has a 180 second (3 minutes) timeout that it has to download the file in. It took me ~4 minutes to download the file at home, and AFAICT the bandwidth limitation is not on my side. On a production appserver it's even slower, currently saying it's going to take 15 minutes (I wonder if they're rate limiting us?) - see P17649. Now, even if the file does download in 180 seconds, there's a 200 second overall MediaWiki timeout.

And the reason you're getting a cryptic, generic error message instead of one that actually tells you the real problem is because the Varnish/ATS timeout appears to be 180 seconds, so that kicks in before MediaWiki's own timeout kicks in. I filed T294800: Reconcile MediaWiki POST timeout and Varnish/ATS timeouts for that.

I tried again to upload a 491 MB TIFF file from https://archive.org/details/clevelandart-1916.1044-gardener-s-house-at and it failed
Direct link: https://archive.org/download/clevelandart-1916.1044-gardener-s-house-at/1916.1044_full.tif
(upload-by-url on Commons)

Request from 90.112.25.87 via cp3064 cp3064, Varnish XID 452004418
Error: 503, Backend fetch failed at Sun, 31 Oct 2021 20:15:12 GMT

MediaWiki has a 180 second (3 minutes) timeout that it has to download the file in. It took me ~4 minutes to download the file at home, and AFAICT the bandwidth limitation is not on my side. On a production appserver it's even slower, currently saying it's going to take 15 minutes (I wonder if they're rate limiting us?) - see P17649. Now, even if the file does download in 180 seconds, there's a 200 second overall MediaWiki timeout. So it would need to upload to Swift, extract metadata, and update the databases in 20 seconds, which is cutting it pretty close. Unfortunately there's no chunked-upload-by-url system I'm aware of, in the meantime I think the best solution is to download the file locally or to Toolforge and then chunked upload to Commons.

I think an even better solution is to significantly raise the timeout until some chunked-upload-by-url system is created.

Hi, While trying large file upload on test.wikipedia.org via upload-by-url, I came into 2 issues:

  • Uploading from Internet Archive is not possible: Copy uploads are not available from this domain.
  • Uploading from Commons, I get This action has been automatically identified as harmful, and therefore disallowed. If you believe your action was constructive, please inform an administrator of what you were trying to do. A brief description of the abuse rule which your action matched is: 0 copyvios.

Archive.org is not whitelisted in InitialiseSettings.php (wgCopyUploadsDomains) for testwiki, but upload.wikimedia.org is. The second error seems to be hitting the 0 copyvios abuse filter. You have only 5 edits on testwiki so you just need to spam more to get 10.

in the meantime I think the best solution is to download the file locally or to Toolforge and then chunked upload to Commons.

https://wikisource-bot.toolforge.org/ssu_request/1916.1044_full.tif

@Yann that should download much faster.

The IA downloads are incredibly slow for me: sometimes it an hour to get a sub-1GB ZIP.

Chunked upload is fine, but the problem that happens to me is that after a file is uploaded and the chunks are assembled, the publishing fails. So the timeout will happen wether the upload itself is chunked or not. Rillke's tool may sometimes succeed - i guess - because it polls the publishing process from time to time, thereby reseting the mediawiki timeout??

in the meantime I think the best solution is to download the file locally or to Toolforge and then chunked upload to Commons.

https://wikisource-bot.toolforge.org/ssu_request/1916.1044_full.tif

@Yann that should download much faster.

The IA downloads are incredibly slow for me: sometimes it an hour to get a sub-1GB ZIP.

Yes, it worked from that URL.

I've also filed:

I think we are reaching the end of the usefulness of this debugging ticket soon, AFAICT all of the high level issues have been identified, they just lack someone to work on them.