1gb file upload limit is too restrictive for conference presentation videos
Closed, DuplicatePublic
Actions

Assigned To

None

Authored By

	• brion
	Oct 25 2015, 6:02 AM

Description

The 1gb file upload limit is a bit too restrictive for conference presentation video recordings; if recorded in HD, it's very easy for a 30-90 minute video to run over a gigabyte.

Current workarounds include recompressing with a lower bitrate (and thus quality) or chopping a large file into smaller pieces, neither of which is ideal.

The limit is configured as:

'wgMaxUploadSize' => array(
	// Affects URL uploads and chunked uploads (experimental).
	// Limit on other web uploads is enforced by PHP.
	'default' => 1024 * 1024 * 1000, // 1000 MB
	'ptwiki' => 1024 * 500, // 500 KB - T25186
),

Additionally, there is a maxFileSizeInternal, which currently is 4294967296 Bytes and hard coded in FileBackendStore.php, after which according to @aaron several very hard limits might potentially start becoming a problem. (See https://gerrit.wikimedia.org/r/#/c/22038 ) T40719: FileBackend.php has 1 GB limit, despite $wgMaxUploadSize.

The problem with further raising the limit, is mostly that we aren't too sure what a sane limit is, should be etc, and that it will require an experiment to verify that it can be done safely, without causing problems (I guess). [DJ]

Details

	Subject	Repo	Branch	Lines +/-
	Raise file upload limit to 2047MB	operations/mediawiki-config	master	+1 -1

Customize query in gerrit

Related Objects
Search...

		Status	Subtype	Assigned	Task
		Open		None	T116510 Batch video upload tracking bug
		Duplicate		None	T116514 1gb file upload limit is too restrictive for conference presentation videos

Event Timeline

• brion created this task.Oct 25 2015, 6:02 AM

• brion raised the priority of this task from to Needs Triage.

• brion updated the task description. (Show Details)

• brion added a project: Video.

• brion subscribed.

Restricted Application added a subscriber: Aklapper. · View Herald TranscriptOct 25 2015, 6:02 AM

• brion added a parent task: T116510: Batch video upload tracking bug.Oct 25 2015, 6:02 AM

Aklapper added a project: Wikimedia-Media-storage.Oct 25 2015, 9:22 AM

Restricted Application added a subscriber: Steinsplitter. · View Herald TranscriptOct 25 2015, 9:22 AM

Fuzheado subscribed.Oct 27 2015, 3:07 PM

Yes, yes a thousand times (or at least 10x) yes. It's fairly typical for a 90 minute session at a conference (ie. Wikimania) to take up about 2 gbytes in a video. So just lifting this to 2 Gbytes would be a tremendous help. Right now we are suffering in trying to get conferences videos up in a timely manner.

Fuzheado awarded a token.Oct 27 2015, 3:27 PM

matmarex edited projects, added Wikimedia-Site-requests; removed Wikimedia-Media-storage.Jan 26 2016, 3:56 PM

matmarex set Security to None.

Restricted Application added subscribers: JEumerus, StudiesWorld, Matanya. · View Herald TranscriptJan 26 2016, 3:56 PM

TheDJ subscribed.Jan 26 2016, 4:05 PM

Removing Wikimedia-Site-requests - this is not a specific upload request

Krenair removed a project: Wikimedia-Site-requests.Jan 26 2016, 4:09 PM

It seems to me although we are definitely 'limited' in file size, it might well be possible to raise the upload limit to 2GB or 2,5GB without too much risk ?

TheDJ updated the task description. (Show Details)Jan 26 2016, 4:44 PM

Change 266544 had a related patch set uploaded (by TheDJ):
Raise file upload limit to 2,5 GB

https://gerrit.wikimedia.org/r/266544

gerritbot added a project: Patch-For-Review.Jan 26 2016, 6:20 PM

Florian subscribed.Jan 27 2016, 6:34 AM

@fgiunchedi Want to give a yes/no to this or the changeset? :)

for sure @Reedy, LGTM, on the swift side the default maximum upload size for a single object is 5GB FYI

Have we actually tested a ~2.4 GB upload and seen it work? I worry that somewhere in the stack, something is using a signed 32-bit integer and will break when we cross 2^31-1.

re: testing, as of a week ago there's a swift cluster in beta as per T64835: Setup a Swift cluster on beta-cluster to match production, mediawiki config change hasn't happened yet tho

zhuyifei1999 subscribed.Feb 1 2016, 8:53 PM

If we have a strong reason to stick with 2.5GB, we should really test this through the whole stack somehow (I guess in beta cluster!) before we try enabling it in prod. If we can't test it or the test doesn't fly, a safer alternative would be to raise it to 1024 * 1024 * 2047 instead of 1024 * 1024 * 2500. This would keep us comfortably below the 2^31-1 danger point and (IMHO) not require rigorous testing.

I think it's a good idea to raise it to the 1024 * 1024 * 2047 for now. As @Fuzheado said 2GB would already make a huge difference but we would avoid the risk pointed out by @BBlack by staying just a bit below the 32-bit limit. Once we can test and if it's not an actual issue we could go further up (i guess up to the 5GB limit in swift).