Page MenuHomePhabricator

Score audio uses different players depending on whether the page has other embedded media
Open, LowPublicBUG REPORT

Description

List of steps to reproduce (step by step, including full links if applicable):

  • Open a page like this, with <score sound="1"> but no AV media embedded via [[File:...]]
  • Open a page like this, with both <score sound="1"> and media embedded via [[File:...]]
  • Compare the two

What happens?:

  • In the first, the browser's native audio player is used.
  • In the second, the TMH player shows up for the score.

What should have happened instead?:
The behavior should be consistent.

Event Timeline

This task is about how TimedMediaHandler's player is sometimes used for Score and at other times not (depending on the content of the rest of the page), whether VideoJS is enabled or not. That VideoJS doesn't work with Score is a completely separate issue.

Ack, sorry about the mis-closure then.

I think the problematic code is in ext.tmh.player.js:

	mw.hook( 'wikipage.content' ).add( function ( $content ) {
		$content.find( 'video, audio' ).loadVideoPlayer();
	} );

The main questions I have are:

  1. Should Score use the native <audio> controls and not whatever player TMH has? T245377: Video.js player doesn’t work with Score is relevant too. What does Score lose by not using TMH?
  2. If so, how should we communicate that, add some no-tmh-please CSS class to the <audio> element?

cc @TheDJ and @brion as TMH experts.

Change 740942 had a related patch set uploaded (by TheDJ; author: TheDJ):

[mediawiki/extensions/TimedMediaHandler@master] Make dialog not crash on missing title

https://gerrit.wikimedia.org/r/740942

Either VideoJS/Kaltura always works for score (preferably without delay)

The JS players always introduce delay, because they have to be loaded asynchronously.

Given rESCRcf8a4f2133cb: Drop TimedMediaHandler support, the latter seems to be the intended behavior, but VideoJS/Kaltura replaces the HTML5 player whenever the page has other media embedded using [[File:...]].

Just because you don't insert the player modules, doesn't mean of course that other elements don't insert the players. There is also no specific opt out for TMH right now. So that commit was bound to break things.

The main questions I have are:

  1. Should Score use the native <audio> controls and not whatever player TMH has? T245377: Video.js player doesn’t work with Score is relevant too. What does Score lose by not using TMH?

TMH just enriches HTML5 audio, video elements (all of them). Score loses consistency, because there isn't an explicit op-out from this behavior.

Score also loses visual consistency, as HTML5 audio players look different on all platforms. HTML 5 audio players were never rly intended to be for consumer use, so they are not standardized in terms of looks/functionality/dimensions.

One of the big problems of TMH is that it is lazy loaded by click, instead of lazy loaded upon load. This was done as a performance enhancement for pages with lots of video players on them. This new lazy loading has the unfortunate disadvantage, that you begin with a working player than can start the playback (if you are quick), which you then (visually) hide and replace with a button, which you then have to click and wait to load a player again in which you can play. (Maybe that should become a config option or something, because i think it's not really an option anyone but WMF actually desires).

  1. If so, how should we communicate that, add some no-tmh-please CSS class to the <audio> element?

That's an option mw-tmh-optout. Or have TMH opt-in and only process it's own videos. My personal original intent was that that should not be needed, and that actually the transcode and transform pipeline and the player should be completely independent of each other and possibly even be separate extensions. Then you could just switch between playback extensions when you wanted).

But this is something that someone should think through, and decide upon and also consider the new html layout that parsoid introduced etc...

The JS players always introduce delay, because they have to be loaded asynchronously.

But not like five seconds, which it takes for the Kaltura player to show up for the score audio.

The JS players always introduce delay, because they have to be loaded asynchronously.

But not like five seconds, which it takes for the Kaltura player to show up for the score audio.

Yes the kaltura player is extremely slow, part of why I've been trying to replace it for over 6 years now.

But it does show up pretty much immediately for uploaded media ([[File:...]]). (Not that it matters if you're going to disable it for Score, but.)

But it does show up pretty much immediately for uploaded media ([[File:...]]). (Not that it matters if you're going to disable it for Score, but.)

A quick check highlights that because the score tag doesn't feature the 'duration' attribute, Kaltura sets a 'waitformetadata' flag. This causes it to delay initialisation till after the 'loadedmetadata' event. This event either never triggers, or was already triggered before the event was installed. A fallback is in place at 5 secs to force the player to load.

Further checks. The audio metadata is already loaded before the event handler is installed. The waitformetacheck function should probably check the element.readyState to be higher than 0 instead (all this was written before HTML5 was actually available, so it's all kludges).

Change 740942 merged by jenkins-bot:

[mediawiki/extensions/TimedMediaHandler@master] Make dialog not crash on missing title

https://gerrit.wikimedia.org/r/740942

Change 741702 had a related patch set uploaded (by TheDJ; author: TheDJ):

[mediawiki/extensions/TimedMediaHandler@master] Check if the native player already has the metadata loaded

https://gerrit.wikimedia.org/r/741702

with these two patches at least score will work again. It doesn't solve the inconsistency of the players that are in use.

An opt-out class just like MediaViewer's noviewer and CirrusSearch's navigation-not-searchable would be great.

It can be done just by adding something like .not( '.no-embed-player *' ) to this line.

Change 741702 merged by jenkins-bot:

[mediawiki/extensions/TimedMediaHandler@master] Check if the native player already has the metadata loaded

https://gerrit.wikimedia.org/r/741702

The 5sec. delay and the videojs crash are both fixed now. The inconsistency however is still there.

Well, whether it uses Kultura or VideoJS simply reflects site/user settings, so that need not be consistent.

brion triaged this task as Low priority.Feb 18 2022, 6:51 PM
Tacsipacsi added a subscriber: Tacsipacsi.

I thought this has long been fixed. In the score+media case Score’s audio output simply disappears, making it impossible to play it. Since Video.js is already default on some wikis (which shouldn’t have happened without this bug being fixed, by the way), this should be fixed ASAP—either fix it really, or roll back everything to Kaltura, which is also inconsistent, but at least works (Kaltura provides a working UI for Score’s output). As a production example, https://fr.wikipedia.org/wiki/Chaconne_en_r%C3%A9_mineur_(Pachelbel) contains a score that should have MP3 output, but it disappears.

I thought this has long been fixed. In the score+media case Score’s audio output simply disappears, making it impossible to play it. Since Video.js is already default on some wikis (which shouldn’t have happened without this bug being fixed, by the way), this should be fixed ASAP—either fix it really, or roll back everything to Kaltura, which is also inconsistent, but at least works (Kaltura provides a working UI for Score’s output). As a production example, https://fr.wikipedia.org/wiki/Chaconne_en_r%C3%A9_mineur_(Pachelbel) contains a score that should have MP3 output, but it disappears.

This is a separate issue (and should be filed separately). The player loading broke when we switched the audio mode of videojs to become an inline loaded element.

This is a separate issue (and should be filed separately). The player loading broke when we switched the audio mode of videojs to become an inline loaded element.

It’s technically covered by this task’s title—the player is different depending on whether the page has other embedded media, it’s a native one when there’s no other player, and an invisible one when there’s one. 🙂 However, I see that the underlying issue is different. Fortunately T306836 was fixed since then, so we’re back at the original issue of players appearing in both situations, but different ones. This means that now it’s functional, so it’s not ASAP anymore, but it should still be fixed, and my opinion on where to fix it didn’t change since last week:

[…] it’s not Score’s fault that TMH’s JavaScript tries to catch up a valid and working HTML element and breaks on it; if it can’t handle HTML output by third parties (which is not only Score, I’m pretty sure there are other extensions outputting <audio>), it shouldn’t try to do so (or at least it should fail gracefully, leaving the page intact in case it breaks).