Page MenuHomePhabricator

Consider increasing $wgTranscodeBackgroundSizeLimit to 5GB
Open, Needs TriagePublic

Description

Right now, $wgTranscodeBackgroundSizeLimit is 3GB. This is the hard limit for max size of a video transcode. (There is also a soft limit at 2GB which requires an admin to ask for the transcode).

Now that we allow 4GB files, and soon 5GB files, it seems like the transcode limit should be 5GB to match if we can support that. It seems

I think swift would be fine storing these (There probably shouldn't be very many, so its probably not a disk space issue). I guess the question is, will this pose any issues to shellbox or the job runners?

Event Timeline

Here's a 4K video that fits in the previous upload limit but has an estimated bitrate resulting in exceeding the 2GB and 3GB limits:

2160p estimated at about 4.5 gb
1440p is running now under a manual override of the 2gb soft limit

I estimate the 1440p should run in about 4-5 hours and the 2160 in about 8-9 hours for this 1-hour 4k source video, but we'll see. :D We can run the 2160p if we raise the harm limit, without affecting what's run by batch operations and regular automatic transcodes until we raise the soft limit.

This is the hard limit for max size of a video transcode.

It is?

$ wget "https://upload.wikimedia.org/wikipedia/commons/transcoded/d/d5/Politparade.webm/Politparade.webm.1440p.vp9.webm?download"
HTTP request sent, awaiting response... 200 OK
Length: 4168557856 (3.9G) [video/webm]
Saving to: ‘Politparade.webm.1440p.vp9.webm?download’

@Bawolff what will be affected by this limit?

If MW estimates the transcode will be bigger than that size, then it will refuse to do it. In essence, it will change the error on the VP9 2160P transcode of File:Politparade.webm.

Historically, $wgTranscodeBackgroundSizeLimit was passed to ulimit to prevent files larger than that size from being created, but it seems this was removed in 7d9fb4ce1.

If MW estimates the transcode will be bigger than that size, then it will refuse to do it. In essence, it will change the error on the VP9 2160P transcode of File:Politparade.webm.

Historically, $wgTranscodeBackgroundSizeLimit was passed to ulimit to prevent files larger than that size from being created, but it seems this was removed in 7d9fb4ce1.

But the VP9 1440p 3.9G transcode was Completed 13:28, 28 March 2022. @Joe 's commit is from Jan 25. So how can the 3.9G transcode exist if the limit is 3GB?

TranscodeBackgroundSizeLimit was compared to the predictive estimated result size. Estimated size is however very format and input dependent due to variable bit rate compression. It's mostly to prevent starting to write a 10GB file and figuring out at 3GB that it was too much.

It's weird that the process limit was not in effect though.. maybe it got processed via the maintenance script or something ?

Right now this limit is not even 4GB.