Raise max upload limit above 1GB
Closed, ResolvedPublic

Description

The current limit for uploads is 1GB. Nowadays were full HD videos are available, this limit is very low, and practically limits full hd videos to several minutes. Please raise this limit all together or at least if the file type is ogv or webm.

Matanya created this task.Dec 3 2014, 2:18 PM
Matanya added a project: Multimedia.
Matanya changed Security from none to None.
Matanya added a subscriber: Matanya.

Are there technical reasons for the current limitation (described on https://commons.wikimedia.org/wiki/Commons:Maximum_file_size )?
If so, have they been sorted out?
Where would the "new" max limit be?

Gilles triaged this task as "Normal" priority.
Gilles added subscribers: acl*operations-team, Gilles.
Steinsplitter moved this task from Incoming to Uploading on the Commons board.Mar 12 2015, 3:02 PM
BBlack added a subscriber: BBlack.Mar 12 2015, 3:10 PM

I don't know, I haven't even begun to look at the technical implications. I'm inclined not to raise the limit blindly. The upload varnishes are already a special case now, I'm inclined to think they're not the right architecture for streaming or uploading enormous files like these.

Fae added a subscriber: Fae.Mar 12 2015, 3:54 PM

Talked this over a bit with mark, I think we could do this if we set varnish to hit_for_pass in vcl_fetch on files over a certain size (probably somewhere in the midst of the 100MB to 1GB range we're caching for large files now), so that the largest ones aren't cached at the varnish layer at all. Needs some VCL changes to support.

Fae added a comment.Mar 12 2015, 4:48 PM

I would like the idea of refining this by media type. For example 100MB SVG files are a mad idea and we should probably reject any, but we have perfectly valid 250MB JPEG files and keeping a 1GB limit on image-types is still practical.

At the same time, being able to upload, say, a 5GB video file could make sense, if we are going to be able to take higher quality videos than the current effective limits of sub-HD resolutions. With institutions becoming interested in donating professional quality video and current examples of 1GB+ files being made public domain by the White House, making it easier to upload larger video files could encourage more video related projects.

For all types there are exceptions, but handling these with "ask an admin (or certain other rights holders) to do it for you" is a lot better than having to raise a Phabricator request just to upload a file.

Remember, if you increase this limit, also need to increase url-downloader.wikimedia.org proxy limit.


I'm doubtful the video scalars would be able to convert multi-gb HD video files with in the (1 hour?) timeout they currently have set. Of course, its better to have the videos and not have derrivitive transcodes, then to not have the video at all.

The implications do depend on file type somewhat. For image files, most multi-gb image files will probably fail to render. Just the time to get the file from swift -> image scalar is problematic [Videos don't have that problem because they aren't rendered on-demand on view].

In T76614#1113493, @Fae wrote:

At the same time, being able to upload, say, a 5GB video file could make sense, if we are going to be able to take higher quality videos than the current effective limits of sub-HD resolutions. With institutions becoming interested in donating professional quality video and current examples of 1GB+ files being made public domain by the White House, making it easier to upload larger video files could encourage more video related projects.

I'm sure opinions on the role of commons vs e.g. archive.org and what we should be hosting to what ends differ within the community and the organization, but at a technical level there's a bottom line here: we don't currently have the infrastructure or manpower to scale up to, say, accepting and serving 10x or 100x the video we do today. Doing so would require an organizational initiative around it with staffing and budget. Massive video traffic is not a cheap or easy thing to handle well.

We're doing ok with what we have today (although it does strain us at some layers), but we're not ready to flip a switch where the up+down video traffic undergoes a massive increase (due not just to the length of the files, but the increased perceived utility of our video storage/playback for millions of users). Allowing the automated inbound flow of much larger video content could end up being that switch in practice.

I'm not rejecting the idea outright, I just think these kinds of thoughts need to be part of the discussion.

We have a lot of things making video support be bad. I suspect that there are manybothings blocking commons from being a video paradise.

For reference, here is the current distribution of video sizes:

MariaDB [commonswiki_p]> select count(*), power( 2, floor( log2( img_size ) ) ) from image where img_media_type = 'VIDEO' group by floor(log2(img_size)) order by floor(log2(img_size)) desc limit 50;

count(*)power( 2, floor( log2( img_size ) ) )
202147483648
561073741824
206536870912
370268435456
830134217728
444967108864
596933554432
779116777216
84608388608
75354194304
63262097152
50241048576
3958524288
2910262144
2091131072
105465536
58332768
27516384
1098192
494096
32048

So out of 58,068 videos, only 1482 (2%) are larger than 128mb. Only 10% are larger than 64 MB. So it seems as it stands only a smaller percentage of videos are even nearing the upper reaches of what we allow, so I don't think further increasing the limit will open a sudden floodgate.

Restricted Application added a subscriber: Steinsplitter. · View Herald TranscriptJul 23 2015, 4:53 PM
Jdforrester-WMF moved this task from Untriaged to Backlog on the Multimedia board.Sep 4 2015, 6:30 PM

Change 266544 had a related patch set uploaded (by Krinkle):
Raise file upload limit to 2047MB

https://gerrit.wikimedia.org/r/266544

Change 266544 merged by jenkins-bot:
Raise file upload limit to 2047MB

https://gerrit.wikimedia.org/r/266544

https://commons.wikimedia.org/wiki/Help:Server-side_upload mentions UW does up to 1GiB - is that now also raised by this change?

Apparently yes.

Yes, the limit should be 2047MB now, please confirm.

I'll post a notice to commons Village Pump shortly.

Yann added a subscriber: Yann.Feb 26 2016, 5:10 PM
Sporti added a subscriber: Sporti.Feb 28 2016, 10:48 AM
matmarex closed this task as "Resolved".Feb 28 2016, 9:36 PM
matmarex removed a project: Patch-For-Review.