Page MenuHomePhabricator

TimedMediaHandler audio captions are out of sync in Firefox (and possibly other browsers)
Open, Needs TriagePublic

Description

Summary: In Firefox, captions for both the original TimedMediaHandler audio player and the beta audio player are out-of-sync by approximately 100 to 400 milliseconds (this seems to vary). This primarily affects UX, since the delay can be noticeable enough to be irritating.


While testing these captions, I noticed that Firefox seemed to be persistently out-of-sync when playing back captions (this happened with both the old media player and the beta media player). Audacity tells me that the recording starts almost exactly on-beat, but even knowing that I wasn't able to synchronize the captions. Safari did noticeably better.

So I uploaded a test file¹ with the same tempo and added these captions. What seems to happen is that Firefox will play the captions with some delay (it doesn't seem to be always the same, and it's worse in the old player), whereas Safari doesn't seem to have much of a delay at all – if there is any – but buffers before playing the file. All of the caption timings should be exact to within 0.01 ms, since I derived them directly from the tempo. I haven't tested any other browsers yet.

¹ The link goes to the English Wikipedia to avoid T230650, since on Commons the issue causes the captions to be displayed in the wrong place.

Event Timeline

Jc86035 created this task.Aug 17 2019, 5:35 PM
Restricted Application added a subscriber: Aklapper. · View Herald TranscriptAug 17 2019, 5:35 PM
brion added a subscriber: brion.Tue, Aug 20, 11:27 PM

Could this be latency from the timeupdate event? It can trigger with a wait as long as 250ms between events per spec.

I'm not sure if I can reproduce the playback error on macOS though; Firefox 68, Chrome 76, and Safari 12 are all giving me similar behavior from what I can tell. They _feel_ in sync, or else they're out of sync in a way that perfectly syncs with the wrong beats...

Jc86035 renamed this task from TimedMediaHandler captions are out of sync in Firefox (and possibly other browsers) to TimedMediaHandler audio captions are out of sync in Firefox (and possibly other browsers).Wed, Aug 21, 8:30 AM
Jc86035 added a comment.EditedWed, Aug 21, 8:36 AM

There's four-sevenths of a second between each beat, and I don't think the delay exceeds that.

The difference is definitely visible to me without having to resort to counting frames. I don't think it's something wrong with my install of Firefox; the same issue occurs when testing on a new profile.

Jc86035 added a comment.EditedWed, Aug 21, 9:56 AM

Firefox 68.0.2 on macOS 10.14.6; video captured and edited using QuickTime Player; audio redirected using Soundflower; exported using ffmpeg.

As an example, at 0:07 the "and" is clearly slightly late. The delays seem to be semi-random and are different each time; the captions are out of sync with each other as well as out of sync with the audio.

(Though the captions appear to be visually misaligned, they actually aren't; in Timeless the left sidebar is narrower than the right sidebar, and I cropped the video based on the Timeless layout.)

Safari 12.1.2 on macOS 10.14.6; video captured and edited using QuickTime Player; audio redirected using Soundflower; exported using ffmpeg.

The captions match up a lot better and are generally synchronized with both the audio and each other.

brion added a comment.Wed, Aug 21, 8:01 PM

I'm still not sure I can reliably tell the difference but I trust you. :) Titles being sometimes late by random amounts up to 250ms seems like it would make sense given the poor latency guarantees on the timeupdate event, so I'm not sure there's much we can do directly here...

It might be possible to change video.js's caption rendering component to fake a more precise event timer, but would probably not be a simple change. (I think it would be necessary to check when the next expected caption change will occur, set a local timer for the expected delay, and then check the current playback time after the timer to ensure that playback reached that spot, and then fire the caption changed event.)

TheDJ added a subscriber: TheDJ.Thu, Aug 22, 7:47 AM

For the old player this is T96504: Video subtitles delay and expected.

For the new player, I doubt it is timeupdate, as I hook directly into the browser event for the to be presented subtitle (though maybe that is internally driven by time update ?).