|operations/puppet : production||Use ffmpeg from jessie-backports on jessie-based video scalers|
|Resolved||MoritzMuehlenhoff||T143536 Upgrade all mw* servers to debian jessie|
|Resolved||None||T144006 Move the MW Beta appservers to Debian|
|Resolved||MoritzMuehlenhoff||T145742 Migrate video scalers to jessie|
|Resolved||brion||T172445 Deprecate/remove Ogg Theora video output formats in favor of WebM|
|Resolved||brion||T172444 Enable WebM playback for ogv.js video player shim|
- Mentioned In
- T172445: Deprecate/remove Ogg Theora video output formats in favor of WebM
T172444: Enable WebM playback for ogv.js video player shim
T144006: Move the MW Beta appservers to Debian
- Mentioned Here
- T103335: Investigate impact of switching from ffmpeg to libav (ffmpeg is not in Jessie)
T116816: beta cluster missing vips command needed to render tiffs and pngs, pnmtojpeg for DjVus
T141338: Backport VIPS 8 to Debian Jessie
T137886: ffmpeg/libav on jessie video scalers
On beta one can try upgrading/switching deployment-tmh01.deployment-prep.eqiad.wmflabs (tmh stands for TimedMediaHandler , the mw extension doing scaling).
Status update: The new jessie scaler has been exposed to production traffic and a few files have been identified which crash ffmpeg2theora on jessie (but are working fine on the old trusty setup). I'm in the process of tracking that down and until then mw1260 is stopped from job processing. What I've done so far:
- Rebuilding ffmpeg2theora against the latest 3.2 (didn't change anything)
- It turned out the old trusty setup was using a git snapshot of libtheora (which I missed since it didn't use a wmf prefix). I built that for jessie-wikimedia and rebuilt ffmpeg2theora, but that stil, crashes
Unfortunately there are no -dbg packages for our ffmpeg version, I'll build a local build with debug symbols. I'll also try to reproduce the error with the stock ffmpeg/ffmpeg2theora version in stretch.
Another observation: Using ffmpeg to convert to ogv, the conversion works just fine (tested on stretch, will also repeat on jessie):
ffmpeg -i National_Archaeological_Museum_Kabile_-_near_Yambol.webm -codec:v libtheora -qscale:v 7 -codec:a libvorbis -qscale:a 5 test.ogv
This uses the same libtheora and libvorbis as ffmpeg2theora, so I'm wondering if we're hitting a problem in ffmpeg2theora rather than ffmpeg/libtheora/libvorbis
Also tested to work fine with jessie-wikimedia:
ffmpeg -i National_Archaeological_Museum_Kabile_-_near_Yambol.webm -codec:v libtheora -qscale:v 7 -codec:a libvorbis -qscale:a 5 -s 640:360 test2.ogv
Per email consultation between me & @MoritzMuehlenhoff we're thinking we should go ahead and deprecate the Ogg Theora video output (using ffmpeg2theora) in favor of the more well-tested WebM VP8 output (using ffmpeg). I'll add a task; a small adjustment to the ogv.js shim setup needs to be done to enable WebM for it.
Status update: Now that Theora generation is disabled, I've re-enabled mw1260, the jessie-based video scaler in eqiad. I'll review logs early next week and if no further errors are logged, the remaining video scalers will be reimaged to Debian as well.
With Theora disabled and mw1260 depooled on Friday, there were no further transcoding errors; all 1699 jobs were processed successfully. I've proceed with reimaging the remaining video scalers (one by one).