Page MenuHomePhabricator

ffmpeg in MW-Docker lacks -row-mt option for improved multithreaded VP9 encoding
Closed, ResolvedPublic

Description

It looks like the MediaWiki-Docker base image is built on Debian 9 (Stretch) which contains an older ffmpeg:

 ffmpeg --version
ffmpeg version 3.2.14-1~deb9u1 Copyright (c) 2000-2019 the FFmpeg developers
  built with gcc 6.3.0 (Debian 6.3.0-18+deb9u1) 20170516

Either this version is too old, or it's compiled against too old a libvpx (or both) to support the -row-mt 1 option for improved multitreaded VP9 encoding when TimedMediaHandler's $wgFFmpegVP9RowMT option is enabled as in production:

Unrecognized option 'row-mt'.
Error splitting the argument list: Option not found

I believe in production environment we use a backported ffmpeg and libvpx, but using a Debian 10 Buster base image should resolve this as well I think.

Impact is relatively low other than complicating testing, as I would have to manually install a custom ffmpeg to confirm things work in production configuration. Without the option, fewer cores are used and it takes longer to run each video transcode but things still work.

Event Timeline

I believe in production environment we use a backported ffmpeg and libvpx, but using a Debian 10 Buster base image should resolve this as well I think.

I think (though we're probably a long ways from it) we're aiming at a very (very) fuzzy kind of production-likeness, so I guess getting that custom build into the stretch image would be a good idea.

It also doesn't seem like too big a deal to supply an (optional-for-now) image based on wikimedia-buster, so I'll that to the list.

@MoritzMuehlenhoff should be able to offer good guidance.

Looking for ffmpeg in Puppet, it turns out that for Stretch we have backported libvpx 1.7.0 and ffmpeg with the row-mt thing. T190333

The packages can be fetched using the apt component component/vp9. Probably something such as:

/etc/apt/sources.list.d/ffmpeg-vp9.list
deb http://apt.wikimedia.org/wikimedia stretch-wikimedia component/vp9

And you will need to add an apt pin configuration to set the priority to 1001 for the packages you want fetched from that component. I can't remember off hand how to do it sorry.

The puppet code:

modules/profile/manifests/mediawiki/videoscaler.pp
class profile::mediawiki::videoscaler()
{
    include ::mediawiki::users

    # Backport of libvpx 1.7 and row-mt support, can be removed once
    # video scalers are migrated to Debian buster
    if os_version('debian == stretch') {

        # ffmpeg has a dependency on the base version of the release, so e.g. (>= 7:3.2.14)
        # This isn't sufficient it make apt pull in our patched version, so explicitly add
        # pinning for the various ffmpeg libraries as well
        $ffmpeg_packages = ['ffmpeg', 'libavcodec57', 'libavdevice57', 'libavfilter6', 'libavformat57',
                            'libavresample3', 'libpostproc54', 'libswresample2', 'libavutil55',
                            'libswscale4']

        apt::package_from_component { 'ffmpeg-vp9':
            component => 'component/vp9',
            packages  => $ffmpeg_packages,
        }
    }
}

0.02€ :]

I should be able to pair on that one ;]

Change 597826 had a related patch set uploaded (by Brennen Bearnes; owner: Brennen Bearnes):
[releng/dev-images@master] add stretch-php72-videoscaler for TimedMediaHandler

https://gerrit.wikimedia.org/r/597826

Change 597826 merged by Jforrester:
[releng/dev-images@master] add jobrunner & tweak settings for TimedMediaHandler

https://gerrit.wikimedia.org/r/597826

brennen moved this task from Waiting Response to Done or Declined on the User-brennen board.
brennen moved this task from In Progress to Done on the MediaWiki-Docker board.