Page MenuHomePhabricator

1gb file upload limit is too restrictive for conference presentation videos
Closed, DuplicatePublic


The 1gb file upload limit is a bit too restrictive for conference presentation video recordings; if recorded in HD, it's very easy for a 30-90 minute video to run over a gigabyte.

Current workarounds include recompressing with a lower bitrate (and thus quality) or chopping a large file into smaller pieces, neither of which is ideal.

The limit is configured as:

'wgMaxUploadSize' => array(
	// Affects URL uploads and chunked uploads (experimental).
	// Limit on other web uploads is enforced by PHP.
	'default' => 1024 * 1024 * 1000, // 1000 MB
	'ptwiki' => 1024 * 500, // 500 KB - T25186

Additionally, there is a maxFileSizeInternal, which currently is 4294967296 Bytes and hard coded in FileBackendStore.php, after which according to @aaron several very hard limits might potentially start becoming a problem. (See ) T40719: FileBackend.php has 1 GB limit, despite $wgMaxUploadSize.

The problem with further raising the limit, is mostly that we aren't too sure what a sane limit is, should be etc, and that it will require an experiment to verify that it can be done safely, without causing problems (I guess). [DJ]

Event Timeline

brion raised the priority of this task from to Needs Triage.
brion updated the task description. (Show Details)
brion added a project: Wikimedia-Video.
brion added a subscriber: brion.

Yes, yes a thousand times (or at least 10x) yes. It's fairly typical for a 90 minute session at a conference (ie. Wikimania) to take up about 2 gbytes in a video. So just lifting this to 2 Gbytes would be a tremendous help. Right now we are suffering in trying to get conferences videos up in a timely manner.

Removing Wikimedia-Site-requests - this is not a specific upload request

TheDJ added a subscriber: aaron.

It seems to me although we are definitely 'limited' in file size, it might well be possible to raise the upload limit to 2GB or 2,5GB without too much risk ?

Change 266544 had a related patch set uploaded (by TheDJ):
Raise file upload limit to 2,5 GB

Reedy added a subscriber: fgiunchedi.
Reedy added a subscriber: Reedy.

@fgiunchedi Want to give a yes/no to this or the changeset? :)

for sure @Reedy, LGTM, on the swift side the default maximum upload size for a single object is 5GB FYI

Have we actually tested a ~2.4 GB upload and seen it work? I worry that somewhere in the stack, something is using a signed 32-bit integer and will break when we cross 2^31-1.

re: testing, as of a week ago there's a swift cluster in beta as per T64835: Setup a Swift cluster on beta-cluster to match production, mediawiki config change hasn't happened yet tho

If we have a strong reason to stick with 2.5GB, we should really test this through the whole stack somehow (I guess in beta cluster!) before we try enabling it in prod. If we can't test it or the test doesn't fly, a safer alternative would be to raise it to 1024 * 1024 * 2047 instead of 1024 * 1024 * 2500. This would keep us comfortably below the 2^31-1 danger point and (IMHO) not require rigorous testing.

I think it's a good idea to raise it to the 1024 * 1024 * 2047 for now. As @Fuzheado said 2GB would already make a huge difference but we would avoid the risk pointed out by @BBlack by staying just a bit below the 32-bit limit. Once we can test and if it's not an actual issue we could go further up (i guess up to the 5GB limit in swift).

Change 266544 merged by jenkins-bot:
Raise file upload limit to 2047MB

I added that change to SWAT today and it was merged by Roan.

This should resolve this ticket. Please confirm.