Page MenuHomePhabricator

1gb file upload limit is too restrictive for conference presentation videos
Closed, DuplicatePublic

Description

The 1gb file upload limit is a bit too restrictive for conference presentation video recordings; if recorded in HD, it's very easy for a 30-90 minute video to run over a gigabyte.

Current workarounds include recompressing with a lower bitrate (and thus quality) or chopping a large file into smaller pieces, neither of which is ideal.

The limit is configured as:

'wgMaxUploadSize' => array(
	// Affects URL uploads and chunked uploads (experimental).
	// Limit on other web uploads is enforced by PHP.
	'default' => 1024 * 1024 * 1000, // 1000 MB
	'ptwiki' => 1024 * 500, // 500 KB - T25186
),

Additionally, there is a maxFileSizeInternal, which currently is 4294967296 Bytes and hard coded in FileBackendStore.php, after which according to @aaron several very hard limits might potentially start becoming a problem. (See https://gerrit.wikimedia.org/r/#/c/22038 ) T40719: FileBackend.php has 1 GB limit, despite $wgMaxUploadSize.

The problem with further raising the limit, is mostly that we aren't too sure what a sane limit is, should be etc, and that it will require an experiment to verify that it can be done safely, without causing problems (I guess). [DJ]

Details

Related Gerrit Patches:
operations/mediawiki-config : masterRaise file upload limit to 2047MB

Event Timeline

brion created this task.Oct 25 2015, 6:02 AM
brion raised the priority of this task from to Needs Triage.
brion updated the task description. (Show Details)
brion added a project: Wikimedia-Video.
brion added a subscriber: brion.
Restricted Application added a subscriber: Aklapper. · View Herald TranscriptOct 25 2015, 6:02 AM
Restricted Application added a subscriber: Steinsplitter. · View Herald TranscriptOct 25 2015, 9:22 AM

Yes, yes a thousand times (or at least 10x) yes. It's fairly typical for a 90 minute session at a conference (ie. Wikimania) to take up about 2 gbytes in a video. So just lifting this to 2 Gbytes would be a tremendous help. Right now we are suffering in trying to get conferences videos up in a timely manner.

matmarex set Security to None.
Restricted Application added subscribers: JEumerus, StudiesWorld, Matanya. · View Herald TranscriptJan 26 2016, 3:56 PM
TheDJ added a subscriber: TheDJ.Jan 26 2016, 4:05 PM

Removing Wikimedia-Site-requests - this is not a specific upload request

TheDJ updated the task description. (Show Details)Jan 26 2016, 4:32 PM
TheDJ added a subscriber: aaron.

It seems to me although we are definitely 'limited' in file size, it might well be possible to raise the upload limit to 2GB or 2,5GB without too much risk ?

TheDJ updated the task description. (Show Details)Jan 26 2016, 4:44 PM

Change 266544 had a related patch set uploaded (by TheDJ):
Raise file upload limit to 2,5 GB

https://gerrit.wikimedia.org/r/266544

Reedy added a subscriber: fgiunchedi.
Reedy added a subscriber: Reedy.

@fgiunchedi Want to give a yes/no to this or the changeset? :)

for sure @Reedy, LGTM, on the swift side the default maximum upload size for a single object is 5GB FYI

BBlack added a subscriber: BBlack.Feb 1 2016, 6:20 PM

Have we actually tested a ~2.4 GB upload and seen it work? I worry that somewhere in the stack, something is using a signed 32-bit integer and will break when we cross 2^31-1.

re: testing, as of a week ago there's a swift cluster in beta as per T64835: Setup a Swift cluster on beta-cluster to match production, mediawiki config change hasn't happened yet tho

If we have a strong reason to stick with 2.5GB, we should really test this through the whole stack somehow (I guess in beta cluster!) before we try enabling it in prod. If we can't test it or the test doesn't fly, a safer alternative would be to raise it to 1024 * 1024 * 2047 instead of 1024 * 1024 * 2500. This would keep us comfortably below the 2^31-1 danger point and (IMHO) not require rigorous testing.

Dzahn added a subscriber: Dzahn.Feb 2 2016, 6:51 PM

I think it's a good idea to raise it to the 1024 * 1024 * 2047 for now. As @Fuzheado said 2GB would already make a huge difference but we would avoid the risk pointed out by @BBlack by staying just a bit below the 32-bit limit. Once we can test and if it's not an actual issue we could go further up (i guess up to the 5GB limit in swift).

Change 266544 merged by jenkins-bot:
Raise file upload limit to 2047MB

https://gerrit.wikimedia.org/r/266544

I added that change to SWAT today and it was merged by Roan.

This should resolve this ticket. Please confirm.