When a domain is whitelisted for server upload, one of the user current expectation is to be able to handle larger files easily.
Yet, a 300-500 Mb files often fail with a "HTTP request timed out" message.
This has been experienced both from archive.org at T106293 and before with other servers.
How to reproduce?
| Status | Subtype | Assigned | Task | ||
|---|---|---|---|---|---|
| Open | None | T118887 Upload by URL doesn't work well for large files: HTTP request timed out. | |||
| Open | None | T295007 Upload by URL should use the job queue, possibly chunked with range requests |
@csteipp Would you know if there is some security configuration in place to timeout fairly quickly an upload task to prevent DoS?
In mediawiki, no. Moritz may know if our outbound proxy is setup for that explicitly.
Actually, yes we do:
*we have a connect timeout of 5 seconds (must connect to server in under 5 seconds. This timeout isprobably not being reached)
*The entire file must be downloadable in less than 25 seconds
*on beta cluster, the entire file must be downloaded in under 10 seconds
*gwtoolset increases this limit to 20 minutes. It messes with globals to do this, so it could affect other things, but is unlikely.
*url-downloader.wikimedia.org has a timeout of 5 minutes (i think. There are some other timeouts, but i dont think they apply to this situation)
Tl;dr: upload by url has a timeout of 25 secondd on commons, except for gwtoolset which has a timeout of 5 minutes
If we want to change this, the config is $wgCopyUploadTimeout. But if we increase it too much we'll probably end up running into php's max execution time.
Change 254700 had a related patch set uploaded (by Brian Wolff):
Increae $wgCopyUploadTimeout to 90 seconds (from default 25)
This patch would triple the time limit. But be aware that even if that patch gets approved, upload by url is still going to suck for super large files, and that's inherent with the current design where we have to fetch the file synchronously with the current request.
This 90 seconds delay seems worth to try: if the remote server can transfer at 30 Mbps, it would allow 300 Mo files.
Change 254700 merged by jenkins-bot:
Increase $wgCopyUploadTimeout to 90 seconds (from default 25)
This one (https://archive.org/details/AlfredHitchcockExtendedInterview 170 MB) went through, but this one (https://archive.org/details/AttackOfTheGiantLeeches 253 MB failed).
Would it make sense to increase $wgCopyUploadTimeout to 300 or are we beyond the PHP execution time limit?
The issue was briefly discussed on #wikimedia-operations channel and @brion highlighted a risk to increase DDoS attack surface as the main drawback to increase $wgCopyUploadTimeout.
The risk is somewhat mitigated by the restriction to $wgCopyUploadDomain.
@csteipp Could you help us to find a value which would be the best compromise between ability to download 1-2 Gb large files and a mitigation of this security risk?
For reference, the current value is 90 s while and a 2 Gb file at 100 Mbps takes 164 s to upload.
It should be mentioned that most free videos available on the Net are not under a free format acceptable for Commons, so uploading videos directly is never an issue. The only exception is old movies available on Internet Archive which are 200 to 450 MB.
Other files which would benefit from this are PDF and DJVU files for Wikisource. These are rarely more than 200 MB.
So setting a value allowing any file up to 450 MB would cover at least 99% of cases.
I beg to differ, we have currently on the site-request project an increase of requests for the 1-2 Gb size. Sources could be WMF or chapters, Commons user shooting videos for the purpose to upload them on Commons.
And then there is all this collection of the videos processed by video2commons: https://commons.wikimedia.org/wiki/Category:Uploaded_with_video2commons/Server-side_uploads where I see some 1.69 Gb and 2.24 Gb.
@csteipp Could you help us to find a value which would be the best compromise between ability to download 1-2 Gb large files and a mitigation of this security risk?
I don't think there's a specific number that will answer this. Since we can move the throttle, lets increase it to something we think will work. If other networks report that we're dosing them, or we're saturating our connection, then we'll drop it back down.
Change 281823 had a related patch set uploaded (by Dereckson):
Raise upload-by-URL request timeout
Mentioned in SAL [2016-04-07T23:13:36Z] <dereckson@tin> Synchronized wmf-config/InitialiseSettings.php: Raise upload-by-URL request timeout (T118887) (duration: 00m 41s)
Mentioned in SAL [2016-04-07T23:42:07Z] <dereckson@tin> Synchronized wmf-config/InitialiseSettings.php: Raise upload-by-URL request timeout (task T118887) (duration: 00m 32s)
Tested: I've a 503 Service Temporarily Unavailable.
But then, when I download it on one of my machine, I've a speed of 392KB/s, so a 7 minutes download. That's greater than the 180 secondes allowed.
No, because QA is required.
This must be tested and only closed if tested (and when tested, if it doesn't work, moved to the most appropriate column).
I'm not confident at all we're good in this task, especially with IA so slow link.
I would probably recommend retooling the upload via URL to run via the job queue, like long-running video transcodes do. This would allow for downloads to take longer if needed, without the possibility of dosing the app servers. This is a bigger task though...
If ddosing ourselves is the concern, it could probably be partially mitigated with poolcounter
Change #1007344 had a related patch set uploaded (by Giuseppe Lavagetto; author: Giuseppe Lavagetto):
[mediawiki/core@master] Switch Special:Upload to use async upload-by-url
Change #1007344 merged by jenkins-bot:
[mediawiki/core@master] Switch Special:Upload to use async upload-by-url