Page MenuHomePhabricator

Upload a large file (>4 GiB) to Commons
Closed, DeclinedPublic

Description

Please upload the files from encoding01.eqiad.wmflabs:/srv/workspace/Informed_but_Unempowered.webm to Wikimedia Commons.

{{Information
        |Description    = {{en|1=Informed but Unempowered talk at wikimania 2014}}
        |Source         = Wikimania 2014 video team
        |Author         = Benjamin Mako Hill, Aaron Shaw
        |Date           = 29.8.2014
        |Permission     =
        |other_versions =
        }}
=={{int:license-header}}==
{{cc-by-sa-4.0}}

user name for upload: matanya

Event Timeline

Matanya assigned this task to Legoktm.
Matanya raised the priority of this task from to Needs Triage.
Matanya updated the task description. (Show Details)
Matanya subscribed.
Catrope set Security to None.

SHA1sum: 874f40c2ec10d33f619aaf11e2383943b6646aac

legoktm@terbium:~/upload$ sha1sum Informed_but_Unempowered.webm 
874f40c2ec10d33f619aaf11e2383943b6646aac  Informed_but_Unempowered.webm
legoktm@terbium:~/upload$ nano Informed_but_Unempowered.txt
legoktm@terbium:~/upload$ mwscript importImages.php --wiki=commonswiki --comment-ext=txt --user=Matanya /home/legoktm/upload/
Import Images

Importing Informed_but_Unempowered.webm...failed. (* Could not write file "mwstore://local-swift-eqiad/local-public/c/c3/Informed_but_Unempowered.webm" because it is larger than {{PLURAL:4294967296|one byte|4294967296 bytes}}.
* Could not store file "/home/legoktm/upload//Informed_but_Unempowered.webm" at "mwstore://local-swift-eqiad/local-public/c/c3/Informed_but_Unempowered.webm".
)

Found: 1
Failed: 1

so...it's too large? but:

legoktm@terbium:~/upload$ du Informed_but_Unempowered.webm 
4433696 	Informed_but_Unempowered.webm

4294967296-4433696=4290533600

curious, I wonder how long ago this broke? I'm not a huge expert in filebackend/filerepo, what seems to be breaking from a quick look is calculating the file size via filesize in FileBackendStore or FileOp.php ? perhaps @aaron can help too

du units are 1024 bytes each.

aaron@terbium:/home/legoktm/upload$ ls -lh
total 4.3G
-rw-rw-r-- 1 legoktm wikidev 350 Jul 22 03:18 Informed_but_Unempowered.txt
-rw-r--r-- 1 legoktm wikidev 4.3G Jul 21 22:41 Informed_but_Unempowered.webm

This is 4.00469 GiB, with is beyond the FileBackend limit.

In any case, the Swift limit is 5Gb (4.65661 GiB), though for compatibility, the subclass does not override FileBackendStore's 4GiB limit. It could be changed to do that. That would buy us a bit more size and would handle this file, I suppose.

Aklapper renamed this task from Upload a large file to Commons to Upload a large file (>4 GiB) to Commons.Jul 23 2015, 8:42 PM

Given this info, i'll close this bug and split the file. sigh.