Summary update:

• iOS doesn't include support for Ogg or WebM media, so native display doesn't work
• I have a library available which provides a suitable player widget, using the standard open-source decoder libraries for both Ogg and WebM
  • cleaning that up so it's easy to pull into the Wikipedia app is task T91689

So roughly what needs to happen:

• clean up the OGVKit framework packaging (T91689)
• add code to the media view mode in Wikipedia app to:
  • fetch transcode/deriv file data
  • launch an OGVKit player with suitably-selected video or audio transcode for audio/* or video/* files in place of the image
• make sure the in-article view makes sense
  • make sure videos have a play icon?
  • make sure audio makes sense?
  • might have to handle direct links to .ogg files for pronunication etc?

Mentioned on Commons VP https://commons.wikimedia.org/wiki/Commons:Village_pump#WebM_files_on_iOS_devices_will_not_play.

PoC of using brion's OGVKit in wikipedia iOS

https://www.youtube.com/watch?v=rTb9peS9X6c

I didn't try to run it on real device. I tried to use brion's ogv.js but cannot run it in simulator due to misunderstanding how webkit webvideo works.
PoC reads meta from api.php for 'File:<name>.ogv' files and pass url for the file with lowest bandwith to OGVKit.

Thanks @yangand !! Is there a pull request or branch we could peek at?

I'll put this on our roadmap for revisiting after our next version, since this seems like good progress to consider.

@JMinor https://github.com/wikimedia/wikipedia-ios/pull/1595 added but please don't merge it to develop, it should be tested on iPad and need additional actions to develop like 'close' buttons, different video quality selection, errors processing, it supports .ogv files only (I just parse html images for .ogv files only).

Thanks for the PR. Don't worry, it definitely won't get merged as is, have a PR just helps us to potentially give you high level feedback and understand the POC for planning.

Note if y'all are interested in moving forward on this, I can finish up the CocoaPods packaging on OGVKit. :)

Audio should work now (using MP3 transcodes). Video still a work in progress.

Yes, we tested yesterday and everything "just works". Huge!

We noticed that pronunciation examples haven't been transcoded? Are they handled differently than other audio?

I filed a task to look into it, but not sure where to take that.

I think a lot of pronunciation samples are linked directly to the source file -- eg [[Media:Ja-Godzilla.oga]] instead of inline players [[File:Ja-Godzilla.oga]]. That bypasses the multi-format support in the <audio> element, and if the source format isn't natively supported, no dice. We can either do a 'smart link handler' in the app -- eg see a ".ogg" or ".oga" link and transform it into the derivative .mp3 link -- or we can devise a new way for users to embed tiny players that are more reliable.

I'd wager the former is easier. :)

for reference that's T183471 -- added note.

Adding OTRS ticket

https://ticket.wikimedia.org/otrs/index.pl?Action=AgentTicketZoom;TicketID=11291821

When will we be able to play video media in the Wikipedia app? Thanks! Steve

Adding another OTRS ticket.

https://ticket.wikimedia.org/otrs/index.pl?Action=AgentTicketZoom;TicketID=11341900

Sorry but i don‘t can Open Videos

Adding OTRS ticket

https://ticket.wikimedia.org/otrs/index.pl?Action=AgentTicketZoom;TicketID=11473873

The video does not play in the "Jupiter" item

Adding OTRS ticket

https://ticket.wikimedia.org/otrs/index.pl?Action=AgentTicketZoom;TicketID=11488729

Galactic Center article - videos don't play

Hi all.

For now, one quick workaround might be to intercept the click and spawn a WKWebView to the desktop pinned version of the Commons URL like https://commons.wikimedia.org/w/index.php?title=File:Jupiter-HubbleSpaceTelescope-Animation-20190823.webm&mobileaction=toggle_view_desktop where playback works. For a more mobile-friendly skin but still desktop try https://commons.wikimedia.org/w/index.php?title=File:Jupiter-HubbleSpaceTelescope-Animation-20190823.webm&mobileaction=toggle_view_desktop&useskin=Minerva . The URL formats have been relatively stable, but you probably should check with Web in case any changes are anticipated in the future. TimedMediaHandler (TMH) will be getting updates that allow more seamless mdot WebKit compatible playback from a ResourceLoader-based JS context - point being eventually you may not need extra URL parameters in the future.

Recent 1 star rating on the Appstore related to this issue (Review: Any plans in the next decade to fix the video player?)

It'll be a bit before TMH has the updates for things to Just Work™ on the mobile web. And that's still not the app. We're hoping some day a universal file format acceptable on Commons will be supported natively on iOS, but it may be a while yet and it's possible it will require higher end hardware.

Resourcing for a full multimedia capability is of perennial interest, but even if resourcing made itself available it may be some time before it's poised to tackle this more fully.

So, shorter run if a research spike might be possible, I would probably suggest trying to see if it's possible to zoom in on the <div> with class mediaContainer in a webview using a URL that will pull in the right transcoding JavaScript (which is needed because there's no MP4 support on Commons and iOS basically needs MP4). I'm not sure how hard that would be. The player controls may need to be nudged to display somehow as those don't surface except with a mouse over event.

The other alternative is to bundle in transcoding libraries into the app and handle it natively, but that would require weighing the cost of how much that inflates the download size of the app. It does seem like there's been some progress on VP9 support in WebKit (note that WebRTC VP9 is a default off experimental feature in Safari mobile) which might be helpful for potential exploration in this space (cf. https://bugs.webkit.org/show_bug.cgi?id=213778).

@brion thoughts and pointers on VLC libs?

While VP9 support is slowly creeping into WebKit via WebRTC there's still no firm sign yet they'll support it in <video> or through iOS's native controls. Based on my past research, I recommend using MobileVLCKit if we want to go the route of WebM VP9+opus playback on iOS:

https://wiki.videolan.org/VLCKit

This has some tradeoffs:

• good - provides a native iOS control for playback of any supported & enabled file type
• bad - but that means you have to pop up a native control either overlaying WebKit or on top of it
• good - decoder is optimized for arm64, so uses less battery than would using the WebAssembly shim inside WebKit
• bad - but you have to ship the code for the decoder, so it adds to the size of the app
• good - VLC can be compiled with just the free codecs with a single switch, or with just the codecs we need with a few more switches, so doesn't pull as many dependencies
• bad - no prebuilt packages, so we'd have to test updated builds
• good - it's open source and everyone loves them

I did some experiments using the codec libraries myself with a lightweight player wrapper, but maintaining it was difficult. Better to let an existing open source project do that work, and share with it!

The star-spangled banner is not playing on the united states article. I tried a bunch of other national anthems but this is the only one that is not playing so far. Tested on iOS App 6.8.0 (1798)

Note - the stars and stripes forever seems to play fine on the same article

https://en.wikipedia.org/wiki/United_States

An alternative might be to transform the dom and replace the video/audio element and use the embed player mode to iframe the content ?

We received this from one of our users through our iOS support email:

You can find a screen recording of the issue I reported in my review, concerning videos in the app. As you can see, the video in the article does not play and the controls for fullscreen do not work for me. Though this is an example, this issue shows everytime I try to play any given video in any given article.

Check the video here:
https://drive.google.com/file/d/1dVXLW2PjBYGdn3QMCm4l1WaJlq7bc4qq/view?usp=share_link

+1 user through the iOS support email:

When I try to watch a video in an article, it never plays

Version: WikipediaApp/7.0.0.2013 (iOS 16.1.1; Phone)

+1 user with the following details:

We received this claim from one of our users; pls check this video for more details.

{F36905145}

State of the world update!

My current research on this is very promising -- thanks to Apple adding a hardware VP9 codec to the newer GPUs, recent model year devices can play back VP9-encoded video. We've been held back specifically by the WebM format support being missing on iOS (it works on macOS now) but can sidestep it by formatting the streams as for HTTP Live Streaming (HLS). These streams can be played back directly in Safari and in the web view in the app or a custom video player using standard iOS widgets, and can also be used in other browsers via video.js.

This means we shouldn't have to ship a custom codec or low-level media player widget like MobileVLCKit, which I think greatly reduces the amount of integration work on the apps end. I'm still finalizing the MediaWiki-side work of generating the tracks & playlists and loading them properly, and will update some tasks w/ more info shortly.

+1 from our support email:

Video materials in the application are not played.
Version: WikipediaApp/7.3.0.2242 (iOS 15.7.7; Phone)

+1:
"videos within articles never play.The problem seems to be across all different video file formats.
Version: WikipediaApp/7.3.0.2242 (iOS 16.1.2; Phone)"

Change 967531 had a related patch set uploaded (by Brion VIBBER; author: Brion VIBBER):

[operations/mediawiki-config@master] "Soft-launch" iOS-compatible HLS video transcodes

https://gerrit.wikimedia.org/r/967531

Change 967531 merged by jenkins-bot:

[operations/mediawiki-config@master] "Soft-launch" iOS-compatible HLS video transcodes

https://gerrit.wikimedia.org/r/967531

Mentioned in SAL (#wikimedia-operations) [2023-10-26T20:12:06Z] <brennen@deploy2002> Started scap: Backport for [[gerrit:967531|"Soft-launch" iOS-compatible HLS video transcodes (T68722)]]

Mentioned in SAL (#wikimedia-operations) [2023-10-26T20:13:23Z] <brennen@deploy2002> brennen and brion: Backport for [[gerrit:967531|"Soft-launch" iOS-compatible HLS video transcodes (T68722)]] synced to the testservers (https://wikitech.wikimedia.org/wiki/Mwdebug)

Mentioned in SAL (#wikimedia-operations) [2023-10-26T20:20:35Z] <brennen@deploy2002> Finished scap: Backport for [[gerrit:967531|"Soft-launch" iOS-compatible HLS video transcodes (T68722)]] (duration: 08m 29s)

Background run for backfill is going; this may take quite some time to complete, but I think should have an initial low-res data set within a few days for everything. I'll continue to monitor it while tweaking the app for a cleaner display & interaction, but basic playback should "just start working" in the iOS app over time :D

Change 973397 had a related patch set uploaded (by Brion VIBBER; author: Brion VIBBER):

[mediawiki/extensions/TimedMediaHandler@master] Fixes to requeueTranscodes to make it easier to batch-fill

https://gerrit.wikimedia.org/r/973397

Change 973397 merged by jenkins-bot:

[mediawiki/extensions/TimedMediaHandler@master] Fixes to requeueTranscodes to make it easier to batch-fill

https://gerrit.wikimedia.org/r/973397

Change 973791 had a related patch set uploaded (by Brion VIBBER; author: Brion VIBBER):

[mediawiki/extensions/TimedMediaHandler@wmf/1.42.0-wmf.4] Fixes to requeueTranscodes to make it easier to batch-fill

https://gerrit.wikimedia.org/r/973791

Change 973791 merged by jenkins-bot:

[mediawiki/extensions/TimedMediaHandler@wmf/1.42.0-wmf.4] Fixes to requeueTranscodes to make it easier to batch-fill

https://gerrit.wikimedia.org/r/973791

Mentioned in SAL (#wikimedia-operations) [2023-11-13T21:21:25Z] <urbanecm@deploy2002> Started scap: Backport for [[gerrit:973808|Undeploy pilot survey on metawiki (T349854)]], [[gerrit:973790|Don't change transcode rows during read operations (T152851)]], [[gerrit:973791|Fixes to requeueTranscodes to make it easier to batch-fill (T68722)]], [[gerrit:973793|Only include completed transcodes in .m3u8 playlist (T350996)]]

Mentioned in SAL (#wikimedia-operations) [2023-11-13T21:22:45Z] <urbanecm@deploy2002> urbanecm and brion and dani: Backport for [[gerrit:973808|Undeploy pilot survey on metawiki (T349854)]], [[gerrit:973790|Don't change transcode rows during read operations (T152851)]], [[gerrit:973791|Fixes to requeueTranscodes to make it easier to batch-fill (T68722)]], [[gerrit:973793|Only include completed transcodes in .m3u8 playlist (T350996)]] synced to the testservers (https://wikitech.wiki

Mentioned in SAL (#wikimedia-operations) [2023-11-13T21:32:02Z] <urbanecm@deploy2002> Finished scap: Backport for [[gerrit:973808|Undeploy pilot survey on metawiki (T349854)]], [[gerrit:973790|Don't change transcode rows during read operations (T152851)]], [[gerrit:973791|Fixes to requeueTranscodes to make it easier to batch-fill (T68722)]], [[gerrit:973793|Only include completed transcodes in .m3u8 playlist (T350996)]] (duration: 10m 37s)

Change 976865 had a related patch set uploaded (by Brion VIBBER; author: Brion VIBBER):

[mediawiki/extensions/TimedMediaHandler@master] Encoding cleanup with remuxing support

https://gerrit.wikimedia.org/r/976865

BTW. I realised that HLS doesn't support media temporal fragments, so likely start and end params won't work on iOS. There is probably no easy way around that, we'll likely have to come up with something custom.

m3u8 has #EXT-X-START:TIME-OFFSET=-15, but as start/end are transclusion params, we can't easily embed them into the manifest without using dynamic manifests (currently we have static manifests)

Change 976865 merged by jenkins-bot:

[mediawiki/extensions/TimedMediaHandler@master] Encoding cleanup with remuxing support

https://gerrit.wikimedia.org/r/976865

Change 979694 had a related patch set uploaded (by Brion VIBBER; author: Brion VIBBER):

[mediawiki/extensions/TimedMediaHandler@wmf/1.42.0-wmf.7] Encoding cleanup with remuxing support

https://gerrit.wikimedia.org/r/979694

Change 979694 merged by jenkins-bot:

[mediawiki/extensions/TimedMediaHandler@wmf/1.42.0-wmf.7] Encoding cleanup with remuxing support

https://gerrit.wikimedia.org/r/979694

Started bulk re-encoding with the remuxing being used transparently whenever possible. I expect full coverage at at least base resolution by end of calendar year. :D

In T68722#9356430, @TheDJ wrote:

BTW. I realised that HLS doesn't support media temporal fragments, so likely start and end params won't work on iOS. There is probably no easy way around that, we'll likely have to come up with something custom.

m3u8 has #EXT-X-START:TIME-OFFSET=-15, but as start/end are transclusion params, we can't easily embed them into the manifest without using dynamic manifests (currently we have static manifests)

I think temporal fragments from [[File:foo.webm|start=N|end=N]] would be applied at the <video> element player logic level (so either part of WebKit for iOS, or video.js) on whatever video source is selected via the selected <source>'s URL and should not have anything to do with the content of the HLS manifests.

The manifest will list the entire media object, which expects to have a starting time at 0 and some positive time duration. (not always true of some of our weirdly muxed input files that seem to be clipped out of tv stream captures with weird timestamps, but that may be something we need to fix externally by remuxing them because most players don't handle this well including the browsers themselves).

In T68722#9381371, @brion wrote:

In T68722#9356430, @TheDJ wrote:

BTW. I realised that HLS doesn't support media temporal fragments, so likely start and end params won't work on iOS. There is probably no easy way around that, we'll likely have to come up with something custom.

m3u8 has #EXT-X-START:TIME-OFFSET=-15, but as start/end are transclusion params, we can't easily embed them into the manifest without using dynamic manifests (currently we have static manifests)

I think temporal fragments from [[File:foo.webm|start=N|end=N]] would be applied at the <video> element player logic level (so either part of WebKit for iOS, or video.js) on whatever video source is selected via the selected <source>'s URL and should not have anything to do with the content of the HLS manifests.

Tested this at https://commons.wikimedia.org/wiki/Commons:Video#Temporal_media_fragments and it does seem that the iOS player takes care of this. If we are to ever run this on other platforms, we'd have to check if the videojs player's HLS takes care of this as well.

This task was created years before VideoJS was implemented and Kaltura player was phased out. I tested out the iOS app's latest version, and the "blowing a raspberry" video works well. So do audio recordings of "When Johnny Comes Marching Home".

Unfortunately, the same cannot be said for unsupported prior app versions via older devices using iOS 12 and below. The (probably?) old player no longer works there.

Is the task by any chance resolved yet? (see my reply below)

I can't help wonder the issue with VideoJS player has resolved. I created tasks T354501 and T354502 about VideoJS player on older iOS versions, but are they related to this task?

In T68722#9440104, @gh87 wrote:

I can't help wonder the issue with VideoJS player has resolved. I created tasks T354501 and T354502 about VideoJS player on older iOS versions, but are they related to this task?

Not w that most of the base res conversions in vp9 are done too I'll be doing another bug fix pass on this stuff. :) There are minimum requirements due to the HLS fragmented MP4 packaging for iOS but it *ought* to work on iOS 15 even in older app versions except where no suitable transcode is available. Be sure to cite specific links I can test.

links to sample:

• blowing a raspberry: https://en.wikipedia.org/wiki/Blowing_a_raspberry
• wall plug: https://en.wikipedia.org/wiki/Wall_plug
• study of animal locomotion: https://en.wikipedia.org/wiki/Study_of_animal_locomotion

A couple more to sample:

• binary black hole: https://en.wikipedia.org/wiki/Binary_black_hole
• Elephant Dream: https://en.wikipedia.org/wiki/Elephants_Dream