Page MenuHomePhabricator

Support MJPEG & PCM audio as last-ditch fallback output for iOS
Closed, DeclinedPublic

Description

ogv.js works pretty well on faster iOS devices in Safari and apps that use the JIT-enabled WKWebView, but in Chrome and other apps using the old UIWebView control it's too slow to play back even low-res video.

However, iOS actually does support a free-software-compatible format: AVI container with MJPEG video and PCM audio.

This is a horrible format for many reasons:

  • MJPEG has no colorspace standards
  • MJPEG is inefficient, so to get the bitrate down must decimate the framerate and it still looks bad
  • PCM is inefficient, so to get the bitrate down must decimate the sampling rate and bit depth, so it sounds bad
  • AVI sux lol!

But it can be easily created from avconv, and it works in a <video> element. YAY

Something like this seems to work:

avconv -i Sneak_Preview_-_Wikipedia_VisualEditor.webm -filter:v scale=640:360: -c:v mjpeg -b 512k -r 5 -ac 1 -ar 22050 -c:a pcm_u8 preview-u8-b512.avi

sample at https://brionv.com/misc/mjpeg/

Note this AVI won't load in Safari on desktop! Only works on iOS. :P

Event Timeline

brion claimed this task.
brion raised the priority of this task from to Low.
brion updated the task description. (Show Details)
brion added a project: TimedMediaHandler.
brion subscribed.
Restricted Application added a subscriber: Aklapper. · View Herald Transcript

mu-law PCM sounds much better than 8-bit unsigned at the same size.

... -c:a pcm_mulaw ...

Hrm, seems like the browser tries to download the whole AVI file before video playback begins. Sighhhhhhh. Audio in WAV seems to play during progressive download, so that's something at least. :)

Having better luck with QuickTime (.mov) container. The version of avconv in Trusty doesn't have the '-movflags faststart' option, but it still seems to stream without that.

Settled on 160p15 for now; higher resolutions look slightly better but bump up the bandwidth or need crappier frame rates which look worse to me.

Note the QuickTime container MJPEGs work in desktop safari too, though currently some quirk of MwEmbedPlayer keeps it from working in the desktop view. :)

ogv.js is deployed for desktop and getting closer for mobile, so there's less need to go with the last-ditch fallback. Can reopen if there's stronger interest, but the quality of the MJPEGs is pretty bad compared to Ogg at the same resolution.

Reopening this as the frontend rewrite for TMH has dragged for a long time, so we're still stuck with no iOS video solution for the moment.

Also, MP3 should work now for the audio track, giving better audio quality. Will test.

Change 380886 had a related patch set uploaded (by Brion VIBBER; owner: Brion VIBBER):
[mediawiki/extensions/TimedMediaHandler@master] [WIP] QuickTime MJPEG/MP3 240p video fallback for iOS

https://gerrit.wikimedia.org/r/380886

Darn, this seems to work well when there's a <video> directly embedded, but doesn't help on popup video transforms, which are just a thumbnail and a link to the original source file... this means nearly all videos on Wikipedia in current configuration (with minimum player size set large).

Will need to change something in the output to make those turn into actual <video>s, or else a small JS shim, at which point it may be better to load ogv.js and get better video quality on the WebMs.

Tested some more on my iPad. So the good thing is that you get full native playback capabilities:

  • it won't dim the screen during playback
  • fullscreen works
  • picture-in-picture works

whereas none of those work with ogv.js at this time.

Will consider options.

I've tried this and it works but damn the quality is just so godawful. Declining again.

(Note that the alternative, which I'm moving forward on, is finishing the migration to Video.JS frontend and using ogv.js on mobile as well as desktop Safari.)

Change 380886 abandoned by Brion VIBBER:
[WIP] QuickTime MJPEG/MP3 240p video fallback for iOS

Reason:
Re-abandoning this as quality/bandwidth is just too awful. Good enough quality is too slow to download on mobile, and small enough looks really bad.

https://gerrit.wikimedia.org/r/380886

@brion Why don't you just stick MPEG-2 with MP3 in an MP4 container? Or doesn't iOS accept that either?

@brion @TheDJ MP4 supports it, it would be daft if Apple skipped that support.. Other options: h.263 (compatible with MPEG-4 Part 2) or dare I say, Sorenson in a .mov?

Simply MPEG-4 Part 2 may also be possible. Most of the patents for that have expired, it's possible (possilikely) those that are left only describe optional features that can be disabled.

Hadn't thought of trying h.263; testing... I've put up some samples at https://brionv.com/misc/pumpjack/

MPEG-2 in mov container won't play on an iPad or iPhone, though it does seem to work in Safari for Mac. MPEG-2 in mp4 container doesn't work in either, though ffmpeg lets me create the files.

h.263-and-MP3-in-mov-container *does* seem to work on my iPad (iOS 11) *but* doesn't play correctly on my Mac (macOS 10.13) where I see only a green buffer...

Note that ffmpeg won't create an MP4 container with h.263 video in it (MP4 and .mov are the same except for constraints). I don't know whether it's possible to "fake" it by claiming the h.263 is h.264 or if the bitstream formats are not backwards-compatible in the same encoder and that won't work.

Note that h.263 has a fixed number of supported resolutions as well, meaning we'd have to scale things and can't go beyond 1408x1152 (might be ok if we use non-square pixels to represent the actual aspect ratio).

h.263 also doesn't compress nearly as well as VP9, so if we can make this work we'd still be eating extra disk space. Not a serious problem but should be managed if we do such a thing.

@brion https://meta.wikimedia.org/wiki/Have_the_patents_for_MPEG-4_Visual_expired_yet%3F

Time for devs and legal to take a look at that. I've already done much of the work. Only 13 patents are left (in a month it'll be only 12). I suspect most if not all are irrelevant to Wikimedia for offering MPEG-4 Visual videos.

Devs should examine the list to determine if any of these may be relevant and if they are, see if they can be avoided. Legal should look at it because I don't offer legal advice.

https://meta.wikimedia.org/wiki/Have_the_patents_for_MPEG-4_Visual_expired_yet%3F#Options

Long story short: I believe the encoder part of XviD, revision 599 (XviD from 2002-10-16) will no longer be patent-encumbered on 2020-12-15.

Possibly sooner, but 2020-12-15 will be relatively easy to confirm as we'd only have to confirm that 6,959,046 is just decoding.