Maybe check to see if it works with a small test file, just to rule out the size of the file triggering something (Yeah, I know, 403 would be the wrong error code for file too big, but just in case).

I'm idly wondering whether you're setting a User-Agent and whether you can perform other actions such as edits.

From the url-downloader config:

reply_body_max_size 534773760 allow all

Looks like upload by url has a max file size of 510 mb. (Also I feel like even without that, things might time out before a 900 mb video got uploaded)

(In reply to MZMcBride from comment #2)

Ignore me! I mis-read the bug report and Bawolff set me straight.

(In reply to Bawolff (Brian Wolff) from comment #3)

From the url-downloader config:

reply_body_max_size 534773760 allow all

Looks like upload by url has a max file size of 510 mb. (Also I feel like
even without that, things might time out before a 900 mb video got uploaded)

https://git.wikimedia.org/blob/operations%2Fpuppet.git/df4b24132abb856323e69a0decf0ebcde65b67fa/modules%2Furl_downloader%2Ftemplates%2Fsquid.conf.erb#L81

Yeah, there are a few references to 510 MB in there.

Change 172120 had a related patch set uploaded by Brian Wolff:
Increase max file size of url downloader proxy to 1010mb

https://gerrit.wikimedia.org/r/172120

Change 172120 merged by Faidon Liambotis:
Increase max file size of url downloader proxy to 1010mb

https://gerrit.wikimedia.org/r/172120

Puppet patch has been merged.

If the problem was indeed the file being too large, the API should have a less misleading way of reporting it, so I don't think this is fully fixed.

Marcin, can you verify whether the upload works now?

Right now I get "HTTP request timed out.", so we are one stage ahead. It didn't wait very long before timeout, few seconds maybe.

In reply to Tisza Gergő from comment #9)

If the problem was indeed the file being too large, the API should have a
less misleading way of reporting it, so I don't think this is fully fixed.

Its a proxy squid server in between the api and the rest of the internet. I dont think mw has any way of knowing it hit a file too big instead of legit 403.

(In reply to Marcin Cieślak from comment #10)

Right now I get "HTTP request timed out.", so we are one stage ahead. It
didn't wait very long before timeout, few seconds maybe.

I think the timeout is currently 30 seconds(?) Im not surprised you hit it. You would have to have some pretty good conectivity to transfer a 900mb file in 30 seconds.

Change 172437 had a related patch set uploaded by saper:
More debug diagnostics for upload by URL

https://gerrit.wikimedia.org/r/172437

The only place where squid (or the target website) tell us more details is the body of the error response.

gerrit change 172437 adds remote server output to the debug log to help troubleshoot such issues.

(In reply to Bawolff (Brian Wolff) from comment #11)

Its a proxy squid server in between the api and the rest of the internet. I
dont think mw has any way of knowing it hit a file too big instead of legit

We are talking about a squid operated by WM-PL, right? AFAIK wikimedia.pl is not WMF-hosted.

That would make this a bug in the squid configuration of WM-PL, presumably. Is there a tracker where this could be upstreamed then?

I think the timeout is currently 30 seconds(?) Im not surprised you hit it.
You would have to have some pretty good connectivity to transfer a 900mb file
in 30 seconds.

The deeper issue is, then, that we don't have a method for upload-by-url that's not limited by the timeout (unlike normal uploads, where we have the chunked upload API)?

(In reply to Tisza Gergő from comment #14)

(In reply to Bawolff (Brian Wolff) from comment #11)
> Its a proxy squid server in between the api and the rest of the internet. I
> dont think mw has any way of knowing it hit a file too big instead of legit
> 403.

We are talking about a squid operated by WM-PL, right? AFAIK wikimedia.pl is
not WMF-hosted.

No, we are talking about url-downloader.wikimedia.org proxy. I believe the proxy is used as a security measure (not 100% sure against what, but isolating user trigerrable http downloads is probably just a goodvidea in general)

The deeper issue is, then, that we don't have a method for upload-by-url
that's not limited by the timeout (unlike normal uploads, where we have the
chunked upload API)?

Yes. We used to have an async mode, but its disabled and possibly has issues (not sure what exactly). We may be able to ease issue by increasing various timeouts or removing the restriction against using tool labs (which afaik has a really fast link to production, being in same data centre and all), but ultimately to truly fix this we would need to reserurecte async mode (otoh i have no idea how that would tie in with current UI. But current ui sucks a lot so i suppose it wouldnt matter much)

what's the status on this btw? did the upload eventually succeeded ?

Last time I tried was when I commented on this bug - it just timed out probably due to transfer time required.

I don't even have the original video now.

thanks @saper, I'm not sure there's anything else from operations @Tgr ? perhaps if download-by-url times out the regular upload can be suggested to the user or sth like that

In T75200#1173463, @Steinsplitter wrote:

The orginal file (http://tools.wikimedia.pl/~odder/whitehouse/41d5129de542285b6c62.webm) has been deleted. Can't reproduce.

Sorry about this, guys; I wasn't aware that a video that I transcoded was being used to test a bug.

I have absolutely no idea what it was (the file name suggests it might have been a Barack Obama speech), and cannot find it right now, but I can upload a new one of around the same size for you to test on if that helps.

Timeout issues (if any) belong to the different bug I think.

An attempt to upload https://archive.org/download/HansMoserUmEineNasenlaenge/Um_eine_Nasenlaenge_1949_Hans_Moser.ogv (356843452 bytes) results in:

{"servedby":"mw1136","error":{"code":"http-timed-out",
   "info":"Error fetching file from remote source",
   "0":"http://tools.wikimedia.pl/~saper/Um_eine_Nasenlaenge_1949_Hans_Moser.ogv",
   "*":"See https://commons.wikimedia.org/w/api.php for API usage"}}