Page MenuHomePhabricator

Unify "give me some number of CPUs I can use for subtasks" logic
ClosedPublic

Authored by demon on Oct 19 2017, 6:28 PM.

Details

Reviewers
mmodell
Group Reviewers
Release-Engineering-Team
Commits
rMSCA2561b82f6211: Unify "give me some number of CPUs I can use for subtasks" logic
Patch without arc
git checkout -b D828 && curl -L https://phabricator.wikimedia.org/D828?download=true | git apply
Summary

Basically we always want to use as many cores as possible, leaving like 2 or so to be nice to other processes

Diff Detail

Repository
rMSCA Scap
Lint
Automatic diff as part of commit; lint not applicable.
Unit
Automatic diff as part of commit; unit tests not applicable.

Event Timeline

demon created this revision.Oct 19 2017, 6:28 PM
Restricted Application added a reviewer: mmodell. · View Herald TranscriptOct 19 2017, 6:28 PM
Restricted Application added a reviewer: Release-Engineering-Team. · View Herald Transcript
Restricted Application added a project: Release-Engineering-Team. · View Herald Transcript
mmodell accepted this revision.Oct 19 2017, 7:01 PM

Probably better than my exercise in verbosity:

1def parallelism(denominator=2, reserve_cores=1):
2 """ Calculate the number of parallel jobs to run
3 This is taken from the total available cpu cores divided by denominator
4 reserve_cores allows for reserving a fixed number of unused cores.
5 By default, max(cpus/2, cpus-1)
6 """
7 cores = multiprocessing.cpu_count()
8 fraction = max(cores / denominator, 1)
9 max_cores = cores - reserve_cores
10 return min(fraction, max_cores)

This revision is now accepted and ready to land.Oct 19 2017, 7:01 PM
In D828#16446, @jenkins wrote:

Build has FAILED

/scap/main.py:28:1: F401 'multiprocessing' imported but unused
scap/main.py
28

can be removed

demon updated this revision to Diff 2196.Oct 20 2017, 6:28 PM
  • Remove unused import
mmodell accepted this revision.Oct 20 2017, 6:49 PM
This revision was automatically updated to reflect the committed changes.