Page MenuHomePhabricator

Implement replacement for Firefogg (on-the-fly video transcoding)
Open, Needs TriagePublic


For policy reasons, Wikimedia Commons doesn't want to host videos using proprietary file formats, like MP4 ( UploadWizard used to use the Firefogg extension ( for on-the-fly transcoding of MP4 files to OGG in the user's browser. But due to changes to how Firefox handles extensions, this is soon to stop working ( and it seems already impossible to install the extension.

With modern browsers' fast JavaScript engines and HTML5 file APIs, it should totally be possible to implement transcoding in JavaScript, without shelling out to external executables, like Firefogg did. As a bonus, that would work on more browsers than Firefox.

Event Timeline

Restricted Application added a subscriber: Aklapper. · View Herald Transcript

My question: Suppose a user wants to upload long video (eg. a copyright expired movie), and transcode will likely take hours, if we are implementing transcoding in JavaScript, is it necessary to keep the browser window open? Note that transcoding will also have a high CPU usage (load is slightly larger than 1 on v2c).

I'm unfamiliar with the current Firefogg integration.

Yes, but that is also how Firefogg works.

[18:46]	<MatmaRex> also, is anyone looking into implementing a replacement [for Firefogg]? we could integrate something like (if you google 'ffmpeg js', this is the top result, but there's like ten other libraries that do the same thing in slightly different ways. surely one of them will fit our needs)
[18:49]	<marktraceur> I wonder if brion has any thoughts on that.
[18:50]	<brion> MatmaRex: ffmpeg.js is pretty slow but does work in a pinch :)
[18:50]	<brion> i think we might get a better experience from a native app converter+uploader but that's work too
[18:50]	<brion> and/or i can keep working on intermediary conversion :)
[18:51]	<brion> oh yeah one other problem with those: legal (ffmpeg JS conversion)
[18:51]	<brion> since it compiles and ships an h.264 decoder
[18:51]	<marktraceur> brion: I figured that might be a thing
[18:51]	<MatmaRex> brion: right. i'm just hoping to retain some no-fuss option in uploadwizard for non-savvy users, to replace firefogg.
[18:52]	<brion> yeah i think i'm just going to have to finish the intermediary stuff, shipping out to toollabs or for conversion
[18:52]	<marktraceur> That seem slike a good option, yeah
[18:52]	<brion> i suspect i can make that integrate with uploadwizard but am prototyping separate

video2commons already has the video transcode (on the backend instead of frontend) + upload, it wouldn't be impossible to port that to production and integrate with UW.

video2commons already has the video transcode (on the backend instead of frontend) + upload, it wouldn't be impossible to port that to production and integrate with UW.

We'd need to run this through legal to use anything in production; it's already potentially dubious to let an unlicensed h.264 decoder run on labs.

videoconverter.js hasn't seen updates in over 2.5 years. does the same thing (Emscripten port of ffmpeg) but seems to be maintained.

I've tried it and it took 4:43 hours for ffmpeg.js to convert my test file (compared to 2:41 for ffmpeg binary).
While this was just 1 simple test, I was surprised with the encoding speed. It's not even 2 times slower in the browser.

However, we'd have to ship with an additional 3.5MB script (and maintain it).
And I assume the licensing could be even more of an issue than if we'd host a transcoder somewhere.

The chief problem with ffmpeg.js is that it has the h.264 and AAC decoders built in and shipped to the client -- like in firefogg's embedded copy of ffmpeg -- making it a legal minefield.

I have some theories on how to use the browser's native mp4 playback to capture audio and video frames to feed into the encoder, which might be easier to do by extending ogv.js with encoding than figuring out how to hack a low level API into the ffmpeg.js wrapper. (I've gotten a work branch of WebM encoding working in the iOS native version OGVKit, should be able to adapt it to my JS interfaces. And we're already shipping ogv.js for playback, so maintenance burden may increase less for us.)

I'm also not certain of the performance impact of the JS encoder; ffmpeg.js uses multiple worker threads by default so might have been a bigger per-thread ratio difference versus a local ffmpeg run that was not divided up that way. I'll run some more benchmarks ...

dr0ptp4kt added a subscriber: dr0ptp4kt.

Moving this to the "Product owner backlog" as a potential opportunity more generally across different browsers.

While Firefogg seems to be entirely dead now (I couldn’t find anything on about the extension), it’s actually possible to shell out from browser extensions in 2024 – using native messaging, both in Firefox and in Chrome. This requires the user to install two things (the browser extension and the native application), but the latter could bundle ffmpeg, thus avoiding to have to install more things than in the pre-WebExtension times. Having to install an extension still excludes some users (e.g. those who use smartphones), but if it’s more efficient and has less legal risks, it may be worth a try. (And to address the concerns in T157319#3001962, the native app could be written in a way that it continues converting even if the browser tab/window is closed, and the extension part can get the result when the tab is re-opened.)

I suspect in 2024 we're actually better off not using an extension; Web Codecs is deployed in Chrome and Safari and can do hardware-accelerated decoding and encoding of video frames and audio data, leaving only demuxing/muxing to the JavaScript side to do. Firefox is also implementing Web Codecs, and should deploy it at some point.

The main limitation is that only some codecs are supported and may vary between implementations, but the common case is H.264 or HEVC video with AAC-LC audio in some flavor of .mp4 or .mov (same container format, different constraints) and both Chrome and Safari's implementations support VP9 video.

Oh, nice, didn’t know about WebCodecs. If it can be done without an extension (and someone actually does it), that’s definitely better.