Page MenuHomePhabricator

Migrate video scalers to jessie
Closed, ResolvedPublic


The video scalers are not yet running jessie. Some prep work was done in T137886, but more tests are needed.

Event Timeline

hashar added a subscriber: hashar.

On beta one can try upgrading/switching deployment-tmh01.deployment-prep.eqiad.wmflabs (tmh stands for TimedMediaHandler , the mw extension doing scaling).

There is a bit of madness with ffmpeg / libav. That has been looked at via T103335 when we added CI slaves to Jessie and there is T137886 as well.

There is also a bunch of packages needed for Jessie such as vips ( T116816, T141338 ) and there are definitely others.

Change 358381 had a related patch set uploaded (by Muehlenhoff; owner: Muehlenhoff):
[operations/puppet@production] Use ffmpeg from jessie-backports on jessie-based video scalers

Change 358381 merged by Muehlenhoff:
[operations/puppet@production] Use ffmpeg from jessie-backports on jessie-based video scalers

Mentioned in SAL (#wikimedia-operations) [2017-07-26T08:26:47Z] <moritzm> reimaging mw2119 to jessie (T145742)

Mentioned in SAL (#wikimedia-operations) [2017-07-26T09:57:05Z] <moritzm> reimaging mw2152 to jessie (T145742)

Mentioned in SAL (#wikimedia-operations) [2017-07-27T10:32:56Z] <moritzm> reimaging mw2246 to jessie (T145742)

Mentioned in SAL (#wikimedia-operations) [2017-07-27T15:03:04Z] <moritzm> stopping jobrunner/jobchron on mw1260 to investigate a few failing ffmpeg2theora invocations (T145742)

Status update: The new jessie scaler has been exposed to production traffic and a few files have been identified which crash ffmpeg2theora on jessie (but are working fine on the old trusty setup). I'm in the process of tracking that down and until then mw1260 is stopped from job processing. What I've done so far:

  • Rebuilding ffmpeg2theora against the latest 3.2 (didn't change anything)
  • It turned out the old trusty setup was using a git snapshot of libtheora (which I missed since it didn't use a wmf prefix). I built that for jessie-wikimedia and rebuilt ffmpeg2theora, but that stil, crashes

Unfortunately there are no -dbg packages for our ffmpeg version, I'll build a local build with debug symbols. I'll also try to reproduce the error with the stock ffmpeg/ffmpeg2theora version in stretch.

Another observation: Using ffmpeg to convert to ogv, the conversion works just fine (tested on stretch, will also repeat on jessie):

ffmpeg -i National_Archaeological_Museum_Kabile_-_near_Yambol.webm -codec:v libtheora -qscale:v 7 -codec:a libvorbis -qscale:a 5 test.ogv

This uses the same libtheora and libvorbis as ffmpeg2theora, so I'm wondering if we're hitting a problem in ffmpeg2theora rather than ffmpeg/libtheora/libvorbis

Also tested to work fine with jessie-wikimedia:
ffmpeg -i National_Archaeological_Museum_Kabile_-_near_Yambol.webm -codec:v libtheora -qscale:v 7 -codec:a libvorbis -qscale:a 5 -s 640:360 test2.ogv

Per email consultation between me & @MoritzMuehlenhoff we're thinking we should go ahead and deprecate the Ogg Theora video output (using ffmpeg2theora) in favor of the more well-tested WebM VP8 output (using ffmpeg). I'll add a task; a small adjustment to the ogv.js shim setup needs to be done to enable WebM for it.

Status update: Now that Theora generation is disabled, I've re-enabled mw1260, the jessie-based video scaler in eqiad. I'll review logs early next week and if no further errors are logged, the remaining video scalers will be reimaged to Debian as well.

With Theora disabled and mw1260 depooled on Friday, there were no further transcoding errors; all 1699 jobs were processed successfully. I've proceed with reimaging the remaining video scalers (one by one).

Mentioned in SAL (#wikimedia-operations) [2017-08-28T07:51:42Z] <moritzm> reimaging mw1259 to jessie (T145742)

Migration to jessie is completed.