Page MenuHomePhabricator

VP9-enabled ffmpeg doesn't get installed after reimage of mw job runner/video scaler
Closed, ResolvedPublic


The job runners which are working as video scalers (in the past this was a dedicated role) are using a repository component (ffmpeg-vp9) which installs a custom ffmpeg and lbvpx build which supports the VP9 codec.

After the reimage of the mw* servers the component is added to apt sources, but only the stock ffmpeg version got installed (i.e. the upgrade to the VP9-enabled build didn't happen).

We need to

  • upgrade the reimaged hosts with cumin
  • fix puppet to properly install it after a reimage (can be tested with the codfw ones which are not yet reimaged)
  • find out whether any encodes failed and need a re-spin

Event Timeline

Change 554550 had a related patch set uploaded (by Muehlenhoff; owner: Muehlenhoff):
[operations/puppet@production] Fix apt pinning for VP9-enabled ffmpeg build

Change 554550 merged by Muehlenhoff:
[operations/puppet@production] Fix apt pinning for VP9-enabled ffmpeg build

@brion It looks like all our videoscalers were lacking VP9 codec support. What do you think we should do ?

Ideally: fix the installs ASAP :) If can't be done: disable $wgFFmpegVP9RowMT and that should hopefully work if it reverted to a version that groks vp9 but doesn't know the -row-mt option.

Change 554894 had a related patch set uploaded (by Brion VIBBER; owner: Brion VIBBER):
[operations/mediawiki-config@master] Temporary disable of $wgFFmpegVP9RowMT

Mentioned in SAL (#wikimedia-operations) [2019-12-05T15:43:07Z] <moritzm> upgrading the reimaged video scalers back to the row-mt enabled ffmpeg T239831

Change 554894 abandoned by Brion VIBBER:
Temporary disable of $wgFFmpegVP9RowMT

installation was fixed, so not needed :) thanks all!

Ok, I'll run:

foreachwiki extensions/TimedMediaHandler/maintenance/requeueTranscodes.php --error --throttle

to catch up the errored transcodes.

Mentioned in SAL (#wikimedia-operations) [2019-12-05T16:40:42Z] <brion> running requeueTranscodes.php --error --throttle on mwmaint1002 to clean up T239831-related broken video transcodes. will raise usage on video scalers for a while.

@brion thank you! You can mark this as resolved if there is nothing else to be done