Page MenuHomePhabricator

Disable 1440p and 2160p video transcodes until encoding performance is better
Open, Needs TriagePublic

Description

Current encoding performance on 1440p and 2160p videos can be very long. This needs to be tuned to run faster, possibly at cost of higher bitrate, as a batch of uploads of long videos can stick in the queue for a couple of days easily.

Step 1: disable 1440p and 2160p transcodes on live production (both HLS and WebM)

Step 2: re-tune the compression settings and test offline

Step 3: re-enable, preferably consolidating on HLS only (or re-tooling the adaptive streaming to combine with the WebM)

Event Timeline

Change #1049608 had a related patch set uploaded (by Bvibber; author: Bvibber):

[operations/mediawiki-config@master] Temporarily disable '4K' 2160p and mid 1440p transcodes

https://gerrit.wikimedia.org/r/1049608

We found that timeouts didn't seem to be handled correctly:

  • the default timeout was much longer than expected
  • php-fpm timed out _first_ which ended up killing the php script (and thus moving on to the next job queue item) but leaving the ffmpeg process running
  • we ended up with 18 long-running high-res multi-threaded ffmpeg processes all churning away competing for cpu time with every other job ;_;

Taking the high-res ones out for now should help until we reduce the initial timestamp and make sure it's enforced correctly.

The encoding can also be reconfigured to run faster, with slightly higher bitrate output. I'll poke at this shortly.

Change #1049608 merged by jenkins-bot:

[operations/mediawiki-config@master] Temporarily disable '4K' 2160p and mid 1440p transcodes

https://gerrit.wikimedia.org/r/1049608

Mentioned in SAL (#wikimedia-operations) [2024-06-25T20:05:47Z] <cjming@deploy1002> Started scap: Backport for [[gerrit:1049608|Temporarily disable '4K' 2160p and mid 1440p transcodes (T368433)]]

Mentioned in SAL (#wikimedia-operations) [2024-06-25T20:08:39Z] <cjming@deploy1002> cjming, bvibber: Backport for [[gerrit:1049608|Temporarily disable '4K' 2160p and mid 1440p transcodes (T368433)]] synced to the testservers (https://wikitech.wikimedia.org/wiki/Mwdebug)

Mentioned in SAL (#wikimedia-operations) [2024-06-25T20:14:24Z] <cjming@deploy1002> Finished scap: Backport for [[gerrit:1049608|Temporarily disable '4K' 2160p and mid 1440p transcodes (T368433)]] (duration: 08m 36s)