Page MenuHomePhabricator

Commons API and Special:Upload fail to upload file over or almost 100 MB (with cryptic "413 Request Entity Too Large" error)
Open, NormalPublic

Description

Using the labs tool ia-upload, it failed to upload a file to Commons with the error being reported being

"The upload to WikimediaCommons failed: Client error response [status code] 413 [reason phrase] Request Entity Too Large [url] https://commons.wikimedia.org/w/api.php?action=upload&filename=Old_and_New_London%2C_vol._1.djvu&format=json"

The file was 57MB in size. I talked to the toollabs developer, and passing on his assessment of why he believed it failed and I was directed to use the standard upload tool. That becomes quite a circuitous route for a file of that size, and it would be great to identify whether the root cause is the API failing or not. thanks.

Event Timeline

Billinghurst raised the priority of this task from to Needs Triage.
Billinghurst updated the task description. (Show Details)
Billinghurst added a subscriber: Billinghurst.
Billinghurst set Security to None.
Aklapper renamed this task from Commons API fails to upload file within 100MB threshold to Commons API fails (413 error) to upload file within 100MB threshold.Jan 12 2015, 1:43 PM
Aklapper triaged this task as Normal priority.

Wondering if either anomie could take a look at this, or if that's already Multimedia team territory...?

Neither the API nor anything else in MediaWiki that I can find will raise an HTTP 413 error. Some software handling the request (e.g. varnish, apache, hhvm, or possibly even the library that tool is using to make the request) is throwing that error before MediaWiki ever sees it, because the request is exceeding the maximum size it is configured to allow. I don't know which piece of the infrastructure is responsible or what the configured maximum for this might be.

The developer of that tool may find chunked uploading helpful to avoid this error.

Neither the API nor anything else in MediaWiki that I can find will raise an HTTP 413 error. Some software handling the request (e.g. varnish, apache, hhvm, or possibly even the library that tool is using to make the request) is throwing that error before MediaWiki ever sees it, because the request is exceeding the maximum size it is configured to allow. I don't know which piece of the infrastructure is responsible or what the configured maximum for this might be.
The developer of that tool may find chunked uploading helpful to avoid this error.

@Tpt Wondering whether you can review @Anomie's feedback here. Thanks.

@Tpt: Could you reply to the last comments?

Aklapper changed the task status from Open to Stalled.Mar 19 2015, 2:46 PM

Setting task status to STALLED as @Tpt has not replied yet.

Restricted Application added a subscriber: Matanya. · View Herald TranscriptJun 16 2015, 4:52 PM
Steinsplitter moved this task from Incoming to Backlog on the Commons board.Jun 17 2015, 11:47 AM
saper changed the task status from Stalled to Open.Sep 1 2015, 1:08 AM
saper edited projects, added acl*sre-team; removed Wikimedia-General-or-Unknown.
saper added a subscriber: saper.EditedSep 1 2015, 1:15 AM

Here's a complete shell script using curl to reproduce the problem without any external tools:

https://github.com/saper/upload-bug86436

This script gives the following output to me in the http.log.02 file:

1== Info: SSL connection using TLSv1.2 / ECDHE-ECDSA-AES128-GCM-SHA256
2== Info: Server certificate:
3== Info: subject: C=US; ST=California; L=San Francisco; O=Wikimedia Foundation, Inc.; CN=*.wikipedia.org
4== Info: start date: 2015-06-23 18:37:07 GMT
5== Info: expire date: 2017-02-19 12:00:00 GMT
6== Info: subjectAltName: test2.wikipedia.org matched
7== Info: issuer: C=BE; O=GlobalSign nv-sa; CN=GlobalSign Organization Validation CA - SHA256 - G2
8== Info: SSL certificate verify ok.
9=> Send header, 436 bytes (0x1b4)
100000: 50 4f 53 54 20 2f 77 2f 61 70 69 2e 70 68 70 20 POST /w/api.php
110010: 48 54 54 50 2f 31 2e 31 0d 0a 55 73 65 72 2d 41 HTTP/1.1..User-A
120020: 67 65 6e 74 3a 20 55 73 65 72 3a 53 61 70 65 72 gent: User:Saper
130030: 20 74 65 73 74 69 6e 67 20 66 6f 72 20 68 74 74 testing for htt
140040: 70 73 3a 2f 2f 70 68 61 62 72 69 63 61 74 6f 72 ps://phabricator
150050: 2e 77 69 6b 69 6d 65 64 69 61 2e 6f 72 67 2f 6d .wikimedia.org/m
160060: 61 6e 69 70 68 65 73 74 2f 74 61 73 6b 2f 65 64 aniphest/task/ed
170070: 69 74 2f 38 36 34 33 36 2f 0d 0a 48 6f 73 74 3a it/86436/..Host:
180080: 20 74 65 73 74 32 2e 77 69 6b 69 70 65 64 69 61 test2.wikipedia
190090: 2e 6f 72 67 0d 0a 41 63 63 65 70 74 3a 20 2a 2f .org..Accept: */
2000a0: 2a 0d 0a 43 6f 6f 6b 69 65 3a 20 43 6f 6f 6b 69 *..Cookie: Cooki
2100b0: 65 3a 20 74 65 73 74 32 77 69 6b 69 55 73 65 72 e: test2wikiUser
2200c0: 4e 61 6d 65 3d 53 61 70 65 72 3b 74 65 73 74 32 Name=Saper;test2
2300d0: 77 69 6b 69 55 73 65 72 49 44 3d 35 30 35 36 3b wikiUserID=XXXX;
2400e0: 63 65 6e 74 72 61 6c 61 75 74 68 5f 55 73 65 72 centralauth_User
2500f0: 3d 53 61 70 65 72 3b 63 65 6e 74 72 61 6c 61 75 =Saper;centralau
260100: 74 68 5f 54 6f 6b 65 6e 3d XXXXXXXXXXXXXXXXXXXX th_Token=XXXXXXX
270110: XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX XXXXXXXXXXXXXXXX
280120: XXXXXXXXXXXXXXXXXXXXXXXXXX 0d 0a 43 6f 6e 74 65 XXXXXXXXX..Conte
290130: 6e 74 2d 4c 65 6e 67 74 68 3a 20 33 35 36 38 34 nt-Length: 35684
300140: 34 31 33 39 0d 0a 45 78 70 65 63 74 3a 20 31 30 4139..Expect: 10
310150: 30 2d 63 6f 6e 74 69 6e 75 65 0d 0a 43 6f 6e 74 0-continue..Cont
320160: 65 6e 74 2d 54 79 70 65 3a 20 6d 75 6c 74 69 70 ent-Type: multip
330170: 61 72 74 2f 66 6f 72 6d 2d 64 61 74 61 3b 20 62 art/form-data; b
340180: 6f 75 6e 64 61 72 79 3d 2d 2d 2d 2d 2d 2d 2d 2d oundary=--------
350190: 2d 2d 2d 2d 2d 2d 2d 2d 2d 2d 2d 2d 2d 2d 2d 2d ----------------
3601a0: 65 64 39 66 66 63 34 36 65 38 64 39 38 66 32 34 ed9ffc46e8d98f24
3701b0: 0d 0a 0d 0a ....
38<= Recv header, 39 bytes (0x27)
390000: 48 54 54 50 2f 31 2e 31 20 34 31 33 20 52 65 71 HTTP/1.1 413 Req
400010: 75 65 73 74 20 45 6e 74 69 74 79 20 54 6f 6f 20 uest Entity Too
410020: 4c 61 72 67 65 0d 0a Large..
42<= Recv header, 21 bytes (0x15)
430000: 53 65 72 76 65 72 3a 20 6e 67 69 6e 78 2f 31 2e Server: nginx/1.
440010: 39 2e 33 0d 0a 9.3..
45<= Recv header, 37 bytes (0x25)
460000: 44 61 74 65 3a 20 54 75 65 2c 20 30 31 20 53 65 Date: Tue, 01 Se
470010: 70 20 32 30 31 35 20 30 31 3a 30 32 3a 34 32 20 p 2015 01:02:42
480020: 47 4d 54 0d 0a GMT..
49<= Recv header, 25 bytes (0x19)
500000: 43 6f 6e 74 65 6e 74 2d 54 79 70 65 3a 20 74 65 Content-Type: te
510010: 78 74 2f 68 74 6d 6c 0d 0a xt/html..
52<= Recv header, 21 bytes (0x15)
530000: 43 6f 6e 74 65 6e 74 2d 4c 65 6e 67 74 68 3a 20 Content-Length:
540010: 31 39 38 0d 0a 198..
55<= Recv header, 19 bytes (0x13)
560000: 43 6f 6e 6e 65 63 74 69 6f 6e 3a 20 63 6c 6f 73 Connection: clos
570010: 65 0d 0a e..
58<= Recv header, 2 bytes (0x2)
590000: 0d 0a ..
60<= Recv data, 198 bytes (0xc6)
610000: 3c 68 74 6d 6c 3e 0d 0a 3c 68 65 61 64 3e 3c 74 <html>..<head><t
620010: 69 74 6c 65 3e 34 31 33 20 52 65 71 75 65 73 74 itle>413 Request
630020: 20 45 6e 74 69 74 79 20 54 6f 6f 20 4c 61 72 67 Entity Too Larg
640030: 65 3c 2f 74 69 74 6c 65 3e 3c 2f 68 65 61 64 3e e</title></head>
650040: 0d 0a 3c 62 6f 64 79 20 62 67 63 6f 6c 6f 72 3d ..<body bgcolor=
660050: 22 77 68 69 74 65 22 3e 0d 0a 3c 63 65 6e 74 65 "white">..<cente
670060: 72 3e 3c 68 31 3e 34 31 33 20 52 65 71 75 65 73 r><h1>413 Reques
680070: 74 20 45 6e 74 69 74 79 20 54 6f 6f 20 4c 61 72 t Entity Too Lar
690080: 67 65 3c 2f 68 31 3e 3c 2f 63 65 6e 74 65 72 3e ge</h1></center>
700090: 0d 0a 3c 68 72 3e 3c 63 65 6e 74 65 72 3e 6e 67 ..<hr><center>ng
7100a0: 69 6e 78 2f 31 2e 39 2e 33 3c 2f 63 65 6e 74 65 inx/1.9.3</cente
7200b0: 72 3e 0d 0a 3c 2f 62 6f 64 79 3e 0d 0a 3c 2f 68 r>..</body>..</h
7300c0: 74 6d 6c 3e 0d 0a tml>..
74== Info: Closing connection 0
75== Info: TLSv1.2, TLS alert, Client hello (1):
76=> Send SSL data, 2 bytes (0x2)
770000: 01 00 ..

saper added a comment.Sep 1 2015, 1:16 AM

I was connecting this way:

m> traceroute6 2620:0:862:ed1a::1
traceroute6 to 2620:0:862:ed1a::1 (2620:0:862:ed1a::1) from 2a01:4f8:a0:7383::, 64 hops max, 12 byte packets
 1  ex3k7.rz1.hetzner.de  0.871 ms  1.196 ms  1.169 ms
 2  hos-tr1.juniper1.rz1.hetzner.de  0.275 ms
    hos-tr3.juniper2.rz1.hetzner.de  0.280 ms
    hos-tr2.juniper1.rz1.hetzner.de  0.256 ms
 3  core11.hetzner.de  0.313 ms
    core11.hetzner.de  0.430 ms
    core11.hetzner.de  0.537 ms
 4  juniper4.rz2.hetzner.de  0.489 ms  8.861 ms  0.443 ms
 5  2001:7f8:30:0:2:1:0:6939  18.131 ms  16.295 ms  9.661 ms
 6  10ge1-4.core1.prg1.he.net  15.062 ms  15.035 ms  20.294 ms
 7  10ge10-12.core1.fra1.he.net  27.239 ms  15.267 ms  44.788 ms
 8  100ge6-1.core1.ams1.he.net  27.199 ms  20.889 ms  20.953 ms
 9  ae2.cr1-esams.wikimedia.org  21.262 ms  21.643 ms  21.037 ms
10  text-lb.esams.wikimedia.org  22.001 ms  22.034 ms  21.704 ms
BBlack claimed this task.Sep 1 2015, 1:30 AM
BBlack added a subscriber: BBlack.

The pasted headers look like the TLS terminators (nginx 1.9.3) are returning the error, which is strange because they're configured with client_max_body_size 100m; in the http section of the primary config file, and that directive isn't set anywhere else. Google searches turn up some rumblings in various nginx versions about it only actually working in server context in spite of the documentation. Will experiment later and see if we need to move it there.

Yann added a subscriber: Yann.Sep 1 2015, 11:15 AM
Jdforrester-WMF moved this task from Untriaged to Backlog on the Multimedia board.Sep 4 2015, 6:23 PM

Same problem while overwriting an existing file with a cropped version via Special:Upload on Commons:

413 Request Entity Too Large
nginx/1.9.4

555 added a subscriber: 555.Sep 24 2015, 7:29 PM

The file was 57MB in size.

If the original DjVu is 57 MB, then it's likely that the upload was indeed over 100 MB, due to https://github.com/Tpt/ia-upload/issues/5

this is a problem for uploading books
with IA moving away from dejavu the bloated jpegs will be larger
important copyright information is unworkable
https://commons.wikimedia.org/wiki/File:Upload_file_too_large.png

Restricted Application added a subscriber: Poyekhali. · View Herald TranscriptApr 18 2016, 2:07 PM
Slowking4 added a comment.EditedApr 18 2016, 3:32 PM

ok i have uploaded this 69.29 MB volume using metadata from IA uploader and uploading using chunked uploads.
https://commons.wikimedia.org/w/index.php?title=File:1977_Books_and_Pamphlets_July-Dec.djvu
could someone please incorporate chunked upload functionality into IA uploader?

Nemo_bis removed BBlack as the assignee of this task.Apr 21 2016, 6:26 PM
Nemo_bis edited projects, added Tools; removed Operations, Traffic.
matmarex renamed this task from Commons API fails (413 error) to upload file within 100MB threshold to Commons API and Special:Upload fail to upload file over or almost 100 MB (with cryptic "413 Request Entity Too Large" error).Jul 1 2016, 8:52 PM
matmarex edited projects, added MediaWiki-Uploading; removed Tools.

I merged T115984 since it's the same underlying issue of server configuration having a limit on POST size (or something) that is (presumably) precisely 100 MB, which means that files slightly under that limit fail to upload without ever reaching the MediaWiki code (depending on the encoding used by browser/tool, files around 60 MB may fail).

It sounds like this is not a IA Upload issue after all, but we are working on T175680: Switch to chunked uploading for large DjVus anyway.