Page MenuHomePhabricator

Use more than 1 thread per transcode
Closed, ResolvedPublic

Description

41 min 45 s, 1,920 × 1,080 (936.69 MB)

For big files such as this (in size, resolution and quality), it would make sense to have more than 1 thread doing hte transcode

In this cases on the wikimedia boxes they have 16 threads (2 x 4 core + HT). Which means a lot of the box is sitting idle, meaning the transcodes take longer, rather than being sped up by using more of the available capacity


Version: master
Severity: enhancement

Details

Reference
bz54060

Event Timeline

bzimport raised the priority of this task from to High.Nov 22 2014, 2:11 AM
bzimport set Reference to bz54060.

jgerber wrote:

problem with using multiple threads is that the memory use per encode multiplies too. whats the amount of ram available in the box, we might be able to use 2 threads per encode now, the first set of encoding boxes did not allow that.

If y'all have to buy more RAM, please buy more RAM. :)

Leaving CPUs idle while transcoding a 40m-minute 1080p video makes no sense at all.

reedy@tmh1001:~$ cat /proc/meminfo
MemTotal: 16419920 kB

so 16GB

See also http://ganglia.wikimedia.org/latest/?r=hour&cs=&ce=&s=by+name&c=Video%2520scalers%2520eqiad&tab=m&vn=

Looks like we're not even using half of the memory of the cluster. CPUs are mostly idle.

Should be plenty of room (slot wise) to upgrade RAM. We might even have some spare at EQIAD. I'll drop a procurement ticket and then it can go through the process.

Any suggestions what we should go for? 64GB in each?

I don't see any reason to also upgrade the PMTPA boxes at the moment.

I suspect we could at least double that one to a 2. The jobs are done synchronously, right? ie one at a time (per job runner)?

In which case, we don't care so much how much memory is used at a time

Thinking about it, the simplest way to fix this bug might be to just make a global. It can then be set for WMF usage as appropriate - even (number of cores - 1) if the boxes only do one at a time.

Code to count cpus or cores in php gets a little scary.

This of course still leaves the potential memory upgrade, and setting of the config as both non issues in the WMF domain

jgerber wrote:

right now we run 10 concurrent jobs(puppet manifests/role/applicationserver.pp) with $wgTranscodeBackgroundMemoryLimit set to 2GB thats 20GB ram. Since a lot of this is mmapped this is working out. Setting threads to 2 we will need to set $wgTranscodeBackgroundMemoryLimit to 4GB, leaving it at 10 concurrent jobs we end up with 40GB ram. Since there are not so many videos to encode in parallel we could reduce the number of concurrent jobs to 5.

Change 84493 had a related patch set uploaded by J:
Make number of threads a configuration option

https://gerrit.wikimedia.org/r/84493

(In reply to comment #6)

right now we run 10 concurrent jobs(puppet
manifests/role/applicationserver.pp)
with $wgTranscodeBackgroundMemoryLimit set to 2GB thats 20GB ram. Since a lot
of this is mmapped this is working out. Setting threads to 2 we will need to
set $wgTranscodeBackgroundMemoryLimit to 4GB, leaving it at 10 concurrent
jobs
we end up with 40GB ram. Since there are not so many videos to encode in
parallel we could reduce the number of concurrent jobs to 5.

That seems a more sensible option to me - fewer jobs running quicker

Any suggestion on ram amounts? Should we double it or quadruple it?

Change 84539 had a related patch set uploaded by J:
Set $wgFFmpegThreads to 2 for faster transcodes

https://gerrit.wikimedia.org/r/84539

Change 84493 merged by jenkins-bot:
Make number of threads a configuration option

https://gerrit.wikimedia.org/r/84493

Change 84775 had a related patch set uploaded by Reedy:
Drop Video Scalers to 5 concurrent jobs

https://gerrit.wikimedia.org/r/84775

Change 84775 merged by Akosiaris:
Drop Video Scalers to 5 concurrent jobs

https://gerrit.wikimedia.org/r/84775

Change 84539 merged by jenkins-bot:
Set $wgFFmpegThreads to 2 for faster transcodes

https://gerrit.wikimedia.org/r/84539

So is more work needed here (if so: what?) or can this be closed as RESOLVED FIXED?

Gilles raised the priority of this task from High to Unbreak Now!.Dec 4 2014, 10:21 AM
Gilles moved this task from Untriaged to Done on the Multimedia board.
Gilles lowered the priority of this task from Unbreak Now! to High.Dec 4 2014, 11:21 AM