Page MenuHomePhabricator

Clean up video transcode config for speed/bitrate balance
Closed, ResolvedPublic

Description

Planned:

  • use faster VP9 encoding options
  • explicitly check the frame rate to scale bitrate target, instead of using a wildly large target with constrained quality
  • use Vorbis audio instead of Opus to workaround Safari limit on multichannel Opus
  • general cleanup & consistency between available formats

Patch is in progress.

Event Timeline

Change 803308 had a related patch set uploaded (by Brion VIBBER; author: Brion VIBBER):

[mediawiki/extensions/TimedMediaHandler@master] Video transcode config cleanup

https://gerrit.wikimedia.org/r/803308

Change 803308 merged by jenkins-bot:

[mediawiki/extensions/TimedMediaHandler@master] Video transcode config cleanup

https://gerrit.wikimedia.org/r/803308

@brion: Your patch uses -fpsmax, which was only introduced in ffmpeg 4.4 (https://git.ffmpeg.org/gitweb/ffmpeg.git/commit/d99cc1782563672bcdb46fb5ec51135847db8c99), while the video scalers run 4.1.9.

Transcoding failure got filed as T317069 and I did the investigation at T317069#8219995. Basically what Moritz says: our ffmpeg is older.

Change 834530 had a related patch set uploaded (by Jforrester; author: Jforrester):

[mediawiki/extensions/TimedMediaHandler@master] Revert "Video transcode config cleanup"

https://gerrit.wikimedia.org/r/834530

Change 834530 merged by jenkins-bot:

[mediawiki/extensions/TimedMediaHandler@master] Revert "Video transcode config cleanup"

https://gerrit.wikimedia.org/r/834530

Change 836997 had a related patch set uploaded (by Martineznovo; author: Jforrester):

[mediawiki/extensions/TimedMediaHandler@REL1_39] Revert "Video transcode config cleanup"

https://gerrit.wikimedia.org/r/836997

The revert was done on master, but this extension has been already branched for REL1_39. I think it should be reverted on REL1_39 too (see submitted patch), to avoid landing a feature that's not present on master.

If the original patch gets applied again without checking if ffmpeg supports this parameter, it would be nice to document in the extension page that this extension requires ffmpeg >= 4.4

Change 836997 merged by jenkins-bot:

[mediawiki/extensions/TimedMediaHandler@REL1_39] Revert "Video transcode config cleanup"

https://gerrit.wikimedia.org/r/836997

Change 888279 had a related patch set uploaded (by Brion VIBBER; author: Brion VIBBER):

[mediawiki/extensions/TimedMediaHandler@master] WIP cleanup to transcode config, adds old-iOS back-compat

https://gerrit.wikimedia.org/r/888279

Change 888279 merged by jenkins-bot:

[mediawiki/extensions/TimedMediaHandler@master] HTTP Live Streaming (HLS for iOS) and WebM transcode cleanup

https://gerrit.wikimedia.org/r/888279

Change 961864 had a related patch set uploaded (by Brion VIBBER; author: Brion VIBBER):

[operations/mediawiki-config@master] Video transcode update for experimental HLS

https://gerrit.wikimedia.org/r/961864

Change 961864 merged by jenkins-bot:

[operations/mediawiki-config@master] Drop old VP8 video transcodes, enable HLS on testwiki

https://gerrit.wikimedia.org/r/961864

Mentioned in SAL (#wikimedia-operations) [2023-10-05T13:36:54Z] <lucaswerkmeister-wmde@deploy2002> Started scap: Backport for [[gerrit:961864|Drop old VP8 video transcodes, enable HLS on testwiki (T312152 T309823)]]

Mentioned in SAL (#wikimedia-operations) [2023-10-05T13:38:26Z] <lucaswerkmeister-wmde@deploy2002> brion and lucaswerkmeister-wmde: Backport for [[gerrit:961864|Drop old VP8 video transcodes, enable HLS on testwiki (T312152 T309823)]] synced to the testservers (https://wikitech.wikimedia.org/wiki/Mwdebug)

Change 961864 had a related patch set uploaded (by Brion VIBBER; author: Brion VIBBER):

[operations/mediawiki-config@master] Video transcode update for experimental HLS

https://gerrit.wikimedia.org/r/961864

Before this config change:

testwiki (wmf.29)
lucaswerkmeister-wmde@mwmaint2002:~$ mwscript shell testwiki
Psy Shell v0.11.10 (PHP 7.4.33 — cli) by Justin Hileman
> $wgEnabledTranscodeSet
= [
    "160p.webm" => true,
    "240p.webm" => true,
    "360p.webm" => true,
    "480p.webm" => true,
    "720p.webm" => true,
    "1080p.webm" => true,
    "1440p.webm" => false,
    "2160p.webm" => false,
    "120p.vp9.webm" => true,
    "180p.vp9.webm" => true,
    "240p.vp9.webm" => true,
    "360p.vp9.webm" => true,
    "480p.vp9.webm" => true,
    "720p.vp9.webm" => true,
    "1080p.vp9.webm" => true,
    "1440p.vp9.webm" => true,
    "2160p.vp9.webm" => true,
    "160p.mp4" => false,
    "240p.mp4" => false,
    "320p.mp4" => false,
    "360p.mp4" => false,
    "480p.mp4" => false,
    "720p.mp4" => false,
    "1080p.mp4" => false,
    "1440p.mp4" => false,
    "2160p.mp4" => false,
    "stereo.audio.mp3" => false,
    "stereo.audio.opus.mp4" => false,
    "144p.video.mjpeg.mov" => false,
    "240p.video.vp9.mp4" => false,
    "360p.video.vp9.mp4" => false,
    "480p.video.vp9.mp4" => false,
    "720p.video.vp9.mp4" => false,
    "1080p.video.vp9.mp4" => false,
    "1440p.video.vp9.mp4" => false,
    "2160p.video.vp9.mp4" => false,
  ]
enwiki (wmf.28)
lucaswerkmeister-wmde@mwdebug2002:~$ mwscript shell enwiki
Psy Shell v0.11.10 (PHP 7.4.33 — cli) by Justin Hileman
> $wgEnabledTranscodeSet
= [
    "160p.webm" => true,
    "240p.webm" => true,
    "360p.webm" => true,
    "480p.webm" => true,
    "720p.webm" => true,
    "1080p.webm" => true,
    "1440p.webm" => false,
    "2160p.webm" => false,
    "120p.vp9.webm" => true,
    "180p.vp9.webm" => true,
    "240p.vp9.webm" => true,
    "360p.vp9.webm" => true,
    "480p.vp9.webm" => true,
    "720p.vp9.webm" => true,
    "1080p.vp9.webm" => true,
    "1440p.vp9.webm" => true,
    "2160p.vp9.webm" => true,
    "160p.mp4" => false,
    "240p.mp4" => false,
    "320p.mp4" => false,
    "360p.mp4" => false,
    "480p.mp4" => false,
    "720p.mp4" => false,
    "1080p.mp4" => false,
    "1440p.mp4" => false,
    "2160p.mp4" => false,
  ]

After this config change:

testwiki (wmf.29)
lucaswerkmeister-wmde@mwdebug2002:~$ mwscript shell testwiki
Psy Shell v0.11.10 (PHP 7.4.33 — cli) by Justin Hileman
> $wgEnabledTranscodeSet
= [
    "160p.webm" => false,
    "240p.webm" => false,
    "360p.webm" => true,
    "480p.webm" => false,
    "720p.webm" => false,
    "1080p.webm" => false,
    "1440p.webm" => false,
    "2160p.webm" => false,
    "120p.vp9.webm" => false,
    "180p.vp9.webm" => false,
    "240p.vp9.webm" => true,
    "360p.vp9.webm" => true,
    "480p.vp9.webm" => true,
    "720p.vp9.webm" => true,
    "1080p.vp9.webm" => true,
    "1440p.vp9.webm" => true,
    "2160p.vp9.webm" => true,
    "160p.mp4" => false,
    "240p.mp4" => false,
    "320p.mp4" => false,
    "360p.mp4" => false,
    "480p.mp4" => false,
    "720p.mp4" => false,
    "1080p.mp4" => false,
    "1440p.mp4" => false,
    "2160p.mp4" => false,
    "stereo.audio.mp3" => true,
    "stereo.audio.opus.mp4" => true,
    "144p.video.mjpeg.mov" => true,
    "240p.video.vp9.mp4" => true,
    "360p.video.vp9.mp4" => true,
    "480p.video.vp9.mp4" => true,
    "720p.video.vp9.mp4" => true,
    "1080p.video.vp9.mp4" => true,
    "1440p.video.vp9.mp4" => true,
    "2160p.video.vp9.mp4" => true,
  ]
enwiki (wmf.28)
lucaswerkmeister-wmde@mwdebug2002:~$ mwscript shell enwiki
Psy Shell v0.11.10 (PHP 7.4.33 — cli) by Justin Hileman
> $wgEnabledTranscodeSet
= [
    "160p.webm" => false,
    "240p.webm" => false,
    "360p.webm" => true,
    "480p.webm" => false,
    "720p.webm" => false,
    "1080p.webm" => false,
    "1440p.webm" => false,
    "2160p.webm" => false,
    "120p.vp9.webm" => true,
    "180p.vp9.webm" => true,
    "240p.vp9.webm" => true,
    "360p.vp9.webm" => true,
    "480p.vp9.webm" => true,
    "720p.vp9.webm" => true,
    "1080p.vp9.webm" => true,
    "1440p.vp9.webm" => true,
    "2160p.vp9.webm" => true,
    "160p.mp4" => false,
    "240p.mp4" => false,
    "320p.mp4" => false,
    "360p.mp4" => false,
    "480p.mp4" => false,
    "720p.mp4" => false,
    "1080p.mp4" => false,
    "1440p.mp4" => false,
    "2160p.mp4" => false,
  ]

Mentioned in SAL (#wikimedia-operations) [2023-10-05T13:49:01Z] <lucaswerkmeister-wmde@deploy2002> Finished scap: Backport for [[gerrit:961864|Drop old VP8 video transcodes, enable HLS on testwiki (T312152 T309823)]] (duration: 12m 07s)

Mentioned in SAL (#wikimedia-operations) [2023-10-05T16:57:18Z] <bvibber> scaling back batch jobs for T312153 and T312152, will run these in further chunks as the new config rolls out

general changes done, doing misc backfill for other reasons now :D