Page MenuHomePhabricator
Create Task
Maniphest T239831

VP9-enabled ffmpeg doesn't get installed after reimage of mw job runner/video scaler
Closed, ResolvedPublic
Actions

Description

The job runners which are working as video scalers (in the past this was a dedicated role) are using a repository component (ffmpeg-vp9) which installs a custom ffmpeg and lbvpx build which supports the VP9 codec.

After the reimage of the mw* servers the component is added to apt sources, but only the stock ffmpeg version got installed (i.e. the upgrade to the VP9-enabled build didn't happen).

We need to

  • upgrade the reimaged hosts with cumin
  • fix puppet to properly install it after a reimage (can be tested with the codfw ones which are not yet reimaged)
  • find out whether any encodes failed and need a re-spin

Details

ProjectBranchLines +/-Subject
operations/mediawiki-configmaster+3 -1Temporary disable of $wgFFmpegVP9RowMT
operations/puppetproduction+6 -0Fix apt pinning for VP9-enabled ffmpeg build
Customize query in gerrit

Related Objects

Event Timeline

MoritzMuehlenhoff created this task.Dec 4 2019, 3:59 PM
Restricted Application added a subscriber: Aklapper. · View Herald TranscriptDec 4 2019, 3:59 PM
Jdforrester-WMF added a subscriber: Jdforrester-WMF.Dec 4 2019, 4:01 PM
MoritzMuehlenhoff mentioned this in T239832: Fix installation of Puppet 5/Facter 3 on new stretch installs/reimages.Dec 4 2019, 4:04 PM
gerritbot added a comment.Dec 4 2019, 4:24 PM

Change 554550 had a related patch set uploaded (by Muehlenhoff; owner: Muehlenhoff):
[operations/puppet@production] Fix apt pinning for VP9-enabled ffmpeg build

https://gerrit.wikimedia.org/r/554550

gerritbot added a project: Patch-For-Review.Dec 4 2019, 4:24 PM
jijiki claimed this task.Dec 4 2019, 4:35 PM
gerritbot added a comment.Dec 5 2019, 8:45 AM

Change 554550 merged by Muehlenhoff:
[operations/puppet@production] Fix apt pinning for VP9-enabled ffmpeg build

https://gerrit.wikimedia.org/r/554550

Maintenance_bot removed a project: Patch-For-Review.Dec 5 2019, 9:10 AM
jijiki added a comment.Dec 5 2019, 12:19 PM

@brion It looks like all our videoscalers were lacking VP9 codec support. What do you think we should do ?

brion added a comment.Dec 5 2019, 3:25 PM

Ideally: fix the installs ASAP :) If can't be done: disable $wgFFmpegVP9RowMT and that should hopefully work if it reverted to a version that groks vp9 but doesn't know the -row-mt option.

gerritbot added a comment.Dec 5 2019, 3:38 PM

Change 554894 had a related patch set uploaded (by Brion VIBBER; owner: Brion VIBBER):
[operations/mediawiki-config@master] Temporary disable of $wgFFmpegVP9RowMT

https://gerrit.wikimedia.org/r/554894

gerritbot added a project: Patch-For-Review.Dec 5 2019, 3:38 PM
Stashbot added a comment.Dec 5 2019, 3:43 PM

Mentioned in SAL (#wikimedia-operations) [2019-12-05T15:43:07Z] <moritzm> upgrading the reimaged video scalers back to the row-mt enabled ffmpeg T239831

gerritbot added a comment.Dec 5 2019, 4:26 PM

Change 554894 abandoned by Brion VIBBER:
Temporary disable of $wgFFmpegVP9RowMT

Reason:
installation was fixed, so not needed :) thanks all!

https://gerrit.wikimedia.org/r/554894

brion added a comment.Dec 5 2019, 4:38 PM

Ok, I'll run:

foreachwiki extensions/TimedMediaHandler/maintenance/requeueTranscodes.php --error --throttle

to catch up the errored transcodes.

Stashbot added a comment.Dec 5 2019, 4:40 PM

Mentioned in SAL (#wikimedia-operations) [2019-12-05T16:40:42Z] <brion> running requeueTranscodes.php --error --throttle on mwmaint1002 to clean up T239831-related broken video transcodes. will raise usage on video scalers for a while.

Maintenance_bot removed a project: Patch-For-Review.Dec 5 2019, 5:10 PM
jijiki added a comment.Dec 5 2019, 6:41 PM

@brion thank you! You can mark this as resolved if there is nothing else to be done

brion closed this task as Resolved.Dec 5 2019, 9:54 PM
Content licensed under Creative Commons Attribution-ShareAlike 3.0 (CC-BY-SA) unless otherwise noted; code licensed under GNU General Public License (GPL) or other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL