Upload by URL doesn't work well for large files: HTTP request timed out.
Open, LowPublic

Description

When a domain is whitelisted for server upload, one of the user current expectation is to be able to handle larger files easily.

Yet, a 300-500 Mb files often fail with a "HTTP request timed out" message.

This has been experienced both from archive.org at T106293 and before with other servers.

How to reproduce?

  1. Go to https://commons.wikimedia.org/wiki/Special:Upload
  2. Enter the URL for a 300-400 Mb files (150 Mb seems to always work, 200 Mb often works)
Dereckson updated the task description. (Show Details)
Dereckson raised the priority of this task from to Normal.
Dereckson added subscribers: Dereckson, Yann, csteipp.
Restricted Application added projects: Multimedia, Internet-Archive. · View Herald TranscriptNov 17 2015, 6:51 PM
Restricted Application added subscribers: Steinsplitter, Aklapper. · View Herald Transcript

@csteipp Would you know if there is some security configuration in place to timeout fairly quickly an upload task to prevent DoS?

Restricted Application added a subscriber: Matanya. · View Herald TranscriptNov 17 2015, 6:52 PM
Dereckson updated the task description. (Show Details)Nov 17 2015, 6:54 PM
Dereckson set Security to None.

@csteipp Would you know if there is some security configuration in place to timeout fairly quickly an upload task to prevent DoS?

In mediawiki, no. Moritz may know if our outbound proxy is setup for that explicitly.

Actually, yes we do:

*we have a connect timeout of 5 seconds (must connect to server in under 5 seconds. This timeout isprobably not being reached)
*The entire file must be downloadable in less than 25 seconds
*on beta cluster, the entire file must be downloaded in under 10 seconds
*gwtoolset increases this limit to 20 minutes. It messes with globals to do this, so it could affect other things, but is unlikely.
*url-downloader.wikimedia.org has a timeout of 5 minutes (i think. There are some other timeouts, but i dont think they apply to this situation)

Tl;dr: upload by url has a timeout of 25 secondd on commons, except for gwtoolset which has a timeout of 5 minutes

If we want to change this, the config is $wgCopyUploadTimeout. But if we increase it too much we'll probably end up running into php's max execution time.

In mediawiki, no. Moritz may know if our outbound proxy is setup for that explicitly.

Not that I'm aware of, no.

Change 254700 had a related patch set uploaded (by Brian Wolff):
Increae $wgCopyUploadTimeout to 90 seconds (from default 25)

https://gerrit.wikimedia.org/r/254700

Change 254700 had a related patch set uploaded (by Brian Wolff):
Increae $wgCopyUploadTimeout to 90 seconds (from default 25)

https://gerrit.wikimedia.org/r/254700

This patch would triple the time limit. But be aware that even if that patch gets approved, upload by url is still going to suck for super large files, and that's inherent with the current design where we have to fetch the file synchronously with the current request.

This 90 seconds delay seems worth to try: if the remote server can transfer at 30 Mbps, it would allow 300 Mo files.

Change 254700 merged by jenkins-bot:
Increase $wgCopyUploadTimeout to 90 seconds (from default 25)

https://gerrit.wikimedia.org/r/254700

*url-downloader.wikimedia.org has a timeout of 5 minutes (i think. There are some other timeouts, but i dont think they apply to this situation)

If we want to change this, the config is $wgCopyUploadTimeout. But if we increase it too much we'll probably end up running into php's max execution time.

Would it make sense to increase $wgCopyUploadTimeout to 300 or are we beyond the PHP execution time limit?

Dereckson added a subscriber: brion.Apr 1 2016, 2:28 PM

The issue was briefly discussed on #wikimedia-operations channel and @brion highlighted a risk to increase DDoS attack surface as the main drawback to increase $wgCopyUploadTimeout.

The risk is somewhat mitigated by the restriction to $wgCopyUploadDomain.

@csteipp Could you help us to find a value which would be the best compromise between ability to download 1-2 Gb large files and a mitigation of this security risk?

For reference, the current value is 90 s while and a 2 Gb file at 100 Mbps takes 164 s to upload.

Yann added a comment.Apr 1 2016, 7:51 PM

It should be mentioned that most free videos available on the Net are not under a free format acceptable for Commons, so uploading videos directly is never an issue. The only exception is old movies available on Internet Archive which are 200 to 450 MB.
Other files which would benefit from this are PDF and DJVU files for Wikisource. These are rarely more than 200 MB.
So setting a value allowing any file up to 450 MB would cover at least 99% of cases.

Dereckson added a comment.EditedApr 1 2016, 7:54 PM

I beg to differ, we have currently on the site-request project an increase of requests for the 1-2 Gb size. Sources could be WMF or chapters, Commons user shooting videos for the purpose to upload them on Commons.

And then there is all this collection of the videos processed by video2commons: https://commons.wikimedia.org/wiki/Category:Uploaded_with_video2commons/Server-side_uploads where I see some 1.69 Gb and 2.24 Gb.

@csteipp Could you help us to find a value which would be the best compromise between ability to download 1-2 Gb large files and a mitigation of this security risk?

I don't think there's a specific number that will answer this. Since we can move the throttle, lets increase it to something we think will work. If other networks report that we're dosing them, or we're saturating our connection, then we'll drop it back down.

Change 281823 had a related patch set uploaded (by Dereckson):
Raise upload-by-URL request timeout

https://gerrit.wikimedia.org/r/281823

Restricted Application added a subscriber: JEumerus. · View Herald TranscriptApr 6 2016, 5:10 PM

Mentioned in SAL [2016-04-07T23:13:36Z] <dereckson@tin> Synchronized wmf-config/InitialiseSettings.php: Raise upload-by-URL request timeout (T118887) (duration: 00m 41s)

Change 281823 merged by jenkins-bot:
Raise upload-by-URL request timeout

https://gerrit.wikimedia.org/r/281823

Mentioned in SAL [2016-04-07T23:42:07Z] <dereckson@tin> Synchronized wmf-config/InitialiseSettings.php: Raise upload-by-URL request timeout (task T118887) (duration: 00m 32s)

Dereckson added a comment.EditedApr 8 2016, 1:17 AM

Tested: I've a 503 Service Temporarily Unavailable.

But then, when I download it on one of my machine, I've a speed of 392KB/s, so a 7 minutes download. That's greater than the 180 secondes allowed.

If this task is done, why is still opened? Should we close it or not?

Dereckson added a comment.EditedApr 14 2016, 12:02 PM

No, because QA is required.

This must be tested and only closed if tested (and when tested, if it doesn't work, moved to the most appropriate column).

I'm not confident at all we're good in this task, especially with IA so slow link.

Ok. I was wondering that there is unclosed task in done column.

brion added a comment.Apr 24 2016, 1:09 PM

I would probably recommend retooling the upload via URL to run via the job queue, like long-running video transcodes do. This would allow for downloads to take longer if needed, without the possibility of dosing the app servers. This is a bigger task though...

The issue was briefly discussed on #wikimedia-operations channel and @brion highlighted a risk to increase DDoS attack surface as the main drawback to increase $wgCopyUploadTimeout.

The risk is somewhat mitigated by the restriction to $wgCopyUploadDomain.

If ddosing ourselves is the concern, it could probably be partially mitigated with poolcounter

MarkTraceur lowered the priority of this task from Normal to Low.Dec 5 2016, 10:15 PM
MarkTraceur moved this task from Untriaged to Triaged on the Multimedia board.