Support video and audio content
Closed, ResolvedPublic

Tokens
"Like" token, awarded by Pine."Like" token, awarded by Liuxinyu970226."Doubloon" token, awarded by Dbrant."Manufacturing Defect?" token, awarded by bearND."Like" token, awarded by Kelson.
Assigned To
Authored By
GWicke, Mar 5 2014

Description

We currently treat video and audio content like images, which is not ideal for Flow, mobile and offline dumps.

We need to

To make this directly useful for page views we might also need to integrate with https://www.mediawiki.org/wiki/Extension:TimedMediaHandler, ideally using client-side JS code.


Version: unspecified
Severity: major
See Also:
T74912: Flow: Score vorbis content doesn't roundtrip (html/wikitext) properly

Details

Reference
bz62270

Related Objects

There are a very large number of changes, so older changes are hidden. Show Older Changes
Liuxinyu970226 added a subscriber: Liuxinyu970226.
Alsee added a comment.EditedJan 4 2017, 2:04 PM

T154387 was just closed as a dupe of this because the general issue is being dealt with here, however T154387 is a much more specific bug with much more specific requirements.

The issue being resolve here appears to be making sure Parsoid does something useful with [[File:Foo]] when Foo is an audio or video file instead of an image.

Solving bug T154387 imposes a constraint making sure Parsoid exactly matches the genuine article view with [[File:Foo]] when Foo is an audio or video file instead of an image.


The above image was taken from the EnWiki article for United States. The left half of the image shows the normal wikitext editor previewing the OGG files in the infobox. That preview correctly matches the genuine article view. The right half of the image shows New Wikitext Editor's broken preview of the same area.

Incidentally, I assume fixing this bug would make Flow and Visual Editor views slightly less broken as well.

I am wondering.. What's the plan here ? It's not clear to me. I haven't seen an update on this from the parsoid team in over a year.

I am wondering.. What's the plan here ? It's not clear to me. I haven't seen an update on this from the parsoid team in over a year.

I had taken this up for implementation in August/September, but I couldn't devote dedicated time to getting this done. So, I finally handed this over to @Arlolra before the devsummit and he is coming up to speed on this and he will be working on this.

And to be fair, it was on my plate before that, but I couldn't get to it. I was actually hoping to touch base on this at the dev summit to make sure we were all still sync'ed up -- but as it turned out, one day of "getting things done" wasn't actually enough. :( Anyway, maybe we should have a brief hangout at some point once @Arlolra has paged in the current state of work to handle any remaining questions/discrepancies.

TheDJ added a comment.Jan 25 2017, 4:15 PM

The markup and how parsoid and VE interacts with this markup still has some uncertainties to me personally with regard to T133673: Add width/height attributes to the <audio><video> tag. I'm seriously considering using the <video> tag for audio only players and adding a --data-file- attribute to indicate audio vs video.

I'd love to talk over that specific part with the parsoid team at some point, but it's hard to discuss something like this when we are not focusing on this ticket first, so yes, a call at some point when people are up to speed would be wonderful.

The Kiwix team needs to have this ticket implemented to be able to fully implement this project:
https://meta.wikimedia.org/wiki/Grants_talk:Project/Stephane/Kiwix/Offline_medical

It would be really helpful for us to get a timeline for the implementation.

The Kiwix team needs to have this ticket implemented to be able to fully implement this project:
https://meta.wikimedia.org/wiki/Grants_talk:Project/Stephane/Kiwix/Offline_medical

It would be really helpful for us to get a timeline for the implementation.

Thanks for that feedback. @Arlolra has this as the next project on his list (besides regular tasks like quick/high-priority bug fixes, code review, code deploys). Based on previous discussions, it looks like this should be doable by end of March, but I don't want to promise that till @Arlolra has had a chance to look at this a bit more.

@Kelson, and that is just parsoid, which doesn't necessarily mean that kiwix will be able to play back the video.. I read that kiwix used to use xulrunner, and i'm not sure that supports video playback in the formats that we support.

You should start exploring that in a parallel track. Your target device and environment will all need to be vetted separately !

Liuxinyu970226 awarded a token.
Arlolra claimed this task.Jan 30 2017, 10:58 PM
Arlolra moved this task from Next Up to In Progress on the Parsoid board.
Arlolra updated the task description. (Show Details)Jan 31 2017, 12:53 AM
Arlolra updated the task description. (Show Details)

Change 335956 had a related patch set uploaded (by Arlolra):
[WIP] T64270: Support video and audio content

https://gerrit.wikimedia.org/r/335956

Ok, I've had a chance to look into this a bit, did some refactoring, and made a first stab at implementing what's spec'd for <video> just to get a feel for it. See the set of patches leading to https://gerrit.wikimedia.org/r/335956

Scheduling a call for next week (or, at everyone's convenience) would be fine.

My first thought is that, as with T114256 for the cite extension, we'll want to find a way to share the tests in,
https://github.com/wikimedia/mediawiki-extensions-TimedMediaHandler/blob/master/tests/parserTests.txt
which probably means generalizing Parsoid's sync parserTests tools. I'll try and bite that off early next week.

My first thought is that, as with T114256 for the cite extension, we'll want to find a way to share the tests in,
https://github.com/wikimedia/mediawiki-extensions-TimedMediaHandler/blob/master/tests/parserTests.txt
which probably means generalizing Parsoid's sync parserTests tools. I'll try and bite that off early next week.

Ok, that's done,.

And, now that @cscott is back from vacation, let's see about scheduling that call.

I'm mostly free next week (other than the archcom meetings Wednesday), happy to start getting this issue back in gear. :)

Pine awarded a token.Feb 16 2017, 5:23 AM
Pine added a subscriber: Pine.

Thanks @brion, still waiting to hear from others.

This comment was removed by Kelson.
Quiddity removed a subscriber: Quiddity.Feb 17 2017, 8:36 PM
TheDJ added a comment.Mar 7 2017, 12:43 PM

Sorry, id been busy.

During the week, i'll be only available in CEST (late) evening time.
This week, I have time either on the 8th, 9th. Otherwise it's gonna be the 13th, 15th or 16th.

Not a problem.

It sounds like @brion is busy on Wednesdays, so how about Thursday the 9th at 10am PST (8pm CEST).

Unfortunately, I'm going to be away next week.

brion added a comment.Mar 7 2017, 6:23 PM

9th @ 10am PST/8pm CEST works for me.

TheDJ added a comment.Mar 8 2017, 12:14 PM

Good for me !

Ok, great, so be it.

I put a pad together w/ some questions I had here, https://etherpad.wikimedia.org/p/T64270

Change 341936 had a related patch set uploaded (by Arlolra):
[mediawiki/services/parsoid] T64270: Support audio content

https://gerrit.wikimedia.org/r/341936

I scheduled a hangout but if anyone prefers IRC, we could do that instead.

Thanks for your time today @brion, @TheDJ, @cscott!

Change 341936 abandoned by Arlolra:
T64270: Support audio content

Reason:
Not using <audio> tags

https://gerrit.wikimedia.org/r/341936

I implemented what we discussed yesterday. The spec and patch are ready for people to start poking at,

https://www.mediawiki.org/wiki/Specs/HTML/1.3.0#Audio.2FVideo_.28Proposal.29
https://gerrit.wikimedia.org/r/#/c/335956/

Krinkle removed a subscriber: Krinkle.Mar 11 2017, 12:42 AM

Change 335956 merged by jenkins-bot:
[mediawiki/services/parsoid@master] T64270: Support video and audio content

https://gerrit.wikimedia.org/r/335956

This probably deserves bumping the spec to 1.4.0

Thanks to all who made this happen. Looking forwards to seeing video in our offline apps :-)

jhsoby removed a subscriber: jhsoby.Apr 27 2017, 6:26 PM

Just want to confirm that thx to this work we have been able to support video/audio in ZIM files for Kiwix.