Page MenuHomePhabricator
Create Task
Maniphest T350996

HLS meta playlist .m3u8 includes not-yet-made transcodes
Closed, ResolvedPublic
Actions

Description

Something's awry in the .m3u8 meta playlist generation, it's in production including files that aren't generated yet, leading to failures when there's no VP9 transcodes yet.

Need to fix this before full launch of iOS, but it appears to be regressing because it prevents the use of the ogv.js shim for back-compat as well. :(

Will poke this over next few days.

Example:

The m3u8 file includes vp9s that don't appear to have been made yet, so the mjpeg doesn't get loaded as back-compat and it breaks everything preventing use of the webms via shim:

#EXTM3U
#EXT-X-MEDIA:TYPE=AUDIO,GROUP-ID="stereo.audio.mp3",NAME="Stereo (MP3)",AUTOSELECT=YES,DEFAULT=YES,CHANNELS="2",URI="Bombay_Cat_kitten_video_of_Blue_Sinfonie.ogv.stereo.audio.mp3.m3u8"
#EXT-X-MEDIA:TYPE=AUDIO,GROUP-ID="stereo.audio.opus.mp4",NAME="Stereo (Opus)",AUTOSELECT=YES,DEFAULT=YES,CHANNELS="2",URI="Bombay_Cat_kitten_video_of_Blue_Sinfonie.ogv.stereo.audio.opus.mp4.m3u8"
#EXT-X-STREAM-INF:BANDWIDTH=1000000,RESOLUTION=176x144,CODECS="jpeg,mp4a.6b",AUDIO="stereo.audio.mp3"
Bombay_Cat_kitten_video_of_Blue_Sinfonie.ogv.144p.video.mjpeg.mov.m3u8
#EXT-X-STREAM-INF:BANDWIDTH=1000000,RESOLUTION=176x144,CODECS="jpeg,Opus",AUDIO="stereo.audio.opus.mp4"
Bombay_Cat_kitten_video_of_Blue_Sinfonie.ogv.144p.video.mjpeg.mov.m3u8
#EXT-X-STREAM-INF:BANDWIDTH=308000,RESOLUTION=426x240,CODECS="vp09.00.51.08,mp4a.6b",AUDIO="stereo.audio.mp3"
Bombay_Cat_kitten_video_of_Blue_Sinfonie.ogv.240p.video.vp9.mp4.m3u8
#EXT-X-STREAM-INF:BANDWIDTH=308000,RESOLUTION=426x240,CODECS="vp09.00.51.08,Opus",AUDIO="stereo.audio.opus.mp4"
Bombay_Cat_kitten_video_of_Blue_Sinfonie.ogv.240p.video.vp9.mp4.m3u8
#EXT-X-STREAM-INF:BANDWIDTH=613000,RESOLUTION=640x360,CODECS="vp09.00.51.08,mp4a.6b",AUDIO="stereo.audio.mp3"
Bombay_Cat_kitten_video_of_Blue_Sinfonie.ogv.360p.video.vp9.mp4.m3u8
#EXT-X-STREAM-INF:BANDWIDTH=613000,RESOLUTION=640x360,CODECS="vp09.00.51.08,Opus",AUDIO="stereo.audio.opus.mp4"
Bombay_Cat_kitten_video_of_Blue_Sinfonie.ogv.360p.video.vp9.mp4.m3u8
#EXT-X-STREAM-INF:BANDWIDTH=1000000,RESOLUTION=854x480,CODECS="vp09.00.51.08,mp4a.6b",AUDIO="stereo.audio.mp3"
Bombay_Cat_kitten_video_of_Blue_Sinfonie.ogv.480p.video.vp9.mp4.m3u8
#EXT-X-STREAM-INF:BANDWIDTH=1000000,RESOLUTION=854x480,CODECS="vp09.00.51.08,Opus",AUDIO="stereo.audio.opus.mp4"
Bombay_Cat_kitten_video_of_Blue_Sinfonie.ogv.480p.video.vp9.mp4.m3u8
#EXT-X-STREAM-INF:BANDWIDTH=1993000,RESOLUTION=1280x720,CODECS="vp09.00.51.08,mp4a.6b",AUDIO="stereo.audio.mp3"
Bombay_Cat_kitten_video_of_Blue_Sinfonie.ogv.720p.video.vp9.mp4.m3u8
#EXT-X-STREAM-INF:BANDWIDTH=1993000,RESOLUTION=1280x720,CODECS="vp09.00.51.08,Opus",AUDIO="stereo.audio.opus.mp4"
Bombay_Cat_kitten_video_of_Blue_Sinfonie.ogv.720p.video.vp9.mp4.m3u8

Details

SubjectRepoBranchLines +/-
Only include completed transcodes in .m3u8 playlistmediawiki/extensions/TimedMediaHandlerwmf/1.42.0-wmf.4+1 -1
Only include completed transcodes in .m3u8 playlistmediawiki/extensions/TimedMediaHandlermaster+1 -1
Customize query in gerrit

Related Objects

Event Timeline

brion created this task.Fri, Nov 10, 4:58 PM
Restricted Application added a subscriber: Aklapper. · View Herald TranscriptFri, Nov 10, 4:58 PM
gerritbot added a comment.Mon, Nov 13, 5:47 PM

Change 973410 had a related patch set uploaded (by Brion VIBBER; author: Brion VIBBER):

[mediawiki/extensions/TimedMediaHandler@master] Only include completed transcodes in .m3u8 playlist

https://gerrit.wikimedia.org/r/973410

gerritbot added a project: Patch-For-Review.Mon, Nov 13, 5:47 PM
gerritbot added a comment.Mon, Nov 13, 6:47 PM

Change 973410 merged by jenkins-bot:

[mediawiki/extensions/TimedMediaHandler@master] Only include completed transcodes in .m3u8 playlist

https://gerrit.wikimedia.org/r/973410

gerritbot added a comment.Mon, Nov 13, 7:00 PM

Change 973793 had a related patch set uploaded (by Brion VIBBER; author: Brion VIBBER):

[mediawiki/extensions/TimedMediaHandler@wmf/1.42.0-wmf.4] Only include completed transcodes in .m3u8 playlist

https://gerrit.wikimedia.org/r/973793

ReleaseTaggerBot added a project: MW-1.42-notes (1.42.0-wmf.5; 2023-11-14).Mon, Nov 13, 7:00 PM
gerritbot added a comment.Mon, Nov 13, 9:19 PM

Change 973793 merged by jenkins-bot:

[mediawiki/extensions/TimedMediaHandler@wmf/1.42.0-wmf.4] Only include completed transcodes in .m3u8 playlist

https://gerrit.wikimedia.org/r/973793

Stashbot mentioned this in T68722: [iOS app] Some media (esp. video) files do not work.Mon, Nov 13, 9:21 PM
Stashbot mentioned this in T152851: TMH should not make DB writes on HTTP GET for its on-the-fly transcode corrections.
Stashbot mentioned this in T349854: Deploy pilot survey on metawiki.

Mentioned in SAL (#wikimedia-operations) [2023-11-13T21:21:25Z] <urbanecm@deploy2002> Started scap: Backport for [[gerrit:973808|Undeploy pilot survey on metawiki (T349854)]], [[gerrit:973790|Don't change transcode rows during read operations (T152851)]], [[gerrit:973791|Fixes to requeueTranscodes to make it easier to batch-fill (T68722)]], [[gerrit:973793|Only include completed transcodes in .m3u8 playlist (T350996)]]

Stashbot added a comment.Mon, Nov 13, 9:22 PM

Mentioned in SAL (#wikimedia-operations) [2023-11-13T21:22:45Z] <urbanecm@deploy2002> urbanecm and brion and dani: Backport for [[gerrit:973808|Undeploy pilot survey on metawiki (T349854)]], [[gerrit:973790|Don't change transcode rows during read operations (T152851)]], [[gerrit:973791|Fixes to requeueTranscodes to make it easier to batch-fill (T68722)]], [[gerrit:973793|Only include completed transcodes in .m3u8 playlist (T350996)]] synced to the testservers (https://wikitech.wiki

Maintenance_bot removed a project: Patch-For-Review.Mon, Nov 13, 9:30 PM
Stashbot added a comment.Mon, Nov 13, 9:32 PM

Mentioned in SAL (#wikimedia-operations) [2023-11-13T21:32:02Z] <urbanecm@deploy2002> Finished scap: Backport for [[gerrit:973808|Undeploy pilot survey on metawiki (T349854)]], [[gerrit:973790|Don't change transcode rows during read operations (T152851)]], [[gerrit:973791|Fixes to requeueTranscodes to make it easier to batch-fill (T68722)]], [[gerrit:973793|Only include completed transcodes in .m3u8 playlist (T350996)]] (duration: 10m 37s)

ReleaseTaggerBot edited projects, added MW-1.42-notes (1.42.0-wmf.4; 2023-11-07); removed MW-1.42-notes (1.42.0-wmf.5; 2023-11-14).Mon, Nov 13, 10:00 PM
brion closed this task as Resolved.Wed, Nov 15, 10:43 PM

Fix deployed, these'll get cleaned up as transcodes run.

TheDJ added a subscriber: TheDJ.Thu, Nov 16, 10:11 AM

I'm positive this stopped progression of the issue, but is it actually cleaning up ? A lot of the media playback on Commons is broken for me right now.

I'm wondering if the resolution selector is breaking on the m3u8 source.

brion added a comment.Fri, Nov 17, 7:20 PM

It'll require actually updating one of the other transcodes on an affected file to rebuild the m3u8 to work again... I've got several batches running, but I might be able to force a simple regeneration too.

Stashbot added a comment.Fri, Nov 17, 7:51 PM

Mentioned in SAL (#wikimedia-operations) [2023-11-17T19:51:14Z] <bvibber> brion regenerating .m3u8 streaming manifests for all video files on mwmaint2002 (cleanup for T350996)

brion reopened this task as Open.EditedFri, Nov 17, 7:52 PM

Running a one-off rebuild of streaming manifests separately from the active transcodes. This should fix any that were damaged that haven't had a chance to be fixed by other runs yet:

php
<?php
$dbr = wfGetDB( DB_REPLICA );
$types = [ 'VIDEO' ];
$where = [ 'img_media_type' => $types ];
$res = $dbr->newSelectQueryBuilder()->select( [ 'img_name' ] )->from( 'image' )->where( $where )->orderBy( 'img_media_type,img_name', Wikimedia\Rdbms\SelectQueryBuilder::SORT_ASC )->caller( __METHOD__ )->fetchResultSet();
foreach ( $res as $row ) {
  echo "$row->img_name\n";
  $title = Title::newFromText( $row->img_name, NS_FILE );
  $file = \MediaWiki\MediaWikiServices::getInstance()->getRepoGroup()->getLocalRepo()->newFile( $title );
  \MediaWiki\TimedMediaHandler\WebVideoTranscode\WebVideoTranscode::updateStreamingManifests( $file );
}
brion added a comment.Fri, Nov 17, 10:05 PM

Script is up to the "An"s, I expect it to complete within ~24-48 hours.

brion changed the task status from Open to In Progress.Sun, Nov 19, 2:08 AM

Script completed. If no surprises during the next couple days of rebuilds, all should be well to resolve. :D

brion closed this task as Resolved.Tue, Nov 21, 8:52 PM

Closing.

Content licensed under Creative Commons Attribution-ShareAlike (CC BY-SA) 4.0 unless otherwise noted; code licensed under GNU General Public License (GPL) 2.0 or later and other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL