Page MenuHomePhabricator
Differential D828

Unify "give me some number of CPUs I can use for subtasks" logic
ClosedPublic
Actions

Authored by demon on Oct 19 2017, 6:28 PM.

Details

Reviewers
mmodell
Group Reviewers
Release-Engineering-Team
Commits
rMSCA2561b82f6211: Unify "give me some number of CPUs I can use for subtasks" logic
Patch without arc
git checkout -b D828 && curl -L https://phabricator.wikimedia.org/D828?download=true | git apply
Summary

Basically we always want to use as many cores as possible, leaving like 2 or so to be nice to other processes

Diff Detail

Repository
rMSCA Scap
Lint
Automatic diff as part of commit; lint not applicable.
Unit
Automatic diff as part of commit; unit tests not applicable.

Event Timeline

demon created this revision.Oct 19 2017, 6:28 PM
Restricted Application added a reviewer: mmodell. · View Herald TranscriptOct 19 2017, 6:28 PM
Restricted Application added a reviewer: Release-Engineering-Team. · View Herald Transcript
Restricted Application added a project: Release-Engineering-Team. · View Herald Transcript
jenkins added a subscriber: jenkins.Oct 19 2017, 6:29 PM

Build has FAILED

Link to build: https://integration.wikimedia.org/ci/job/phabricator-jessie-diffs/777/
See console output for more information: https://integration.wikimedia.org/ci/job/phabricator-jessie-diffs/777/console

Harbormaster failed remote builds in B2373: Diff 2186!Oct 19 2017, 6:29 PM
jenkins added a comment.Oct 19 2017, 6:30 PM

Build has FAILED

Link to build: https://integration.wikimedia.org/ci/job/differential-docker-test/284/
See console output for more information: https://integration.wikimedia.org/ci/job/differential-docker-test/284/console

mmodell accepted this revision.Oct 19 2017, 7:01 PM

Probably better than my exercise in verbosity:

P6156 parallelism
1def parallelism(denominator=2, reserve_cores=1):
2 """ Calculate the number of parallel jobs to run
3 This is taken from the total available cpu cores divided by denominator
4 reserve_cores allows for reserving a fixed number of unused cores.
5 By default, max(cpus/2, cpus-1)
6 """
7 cores = multiprocessing.cpu_count()
8 fraction = max(cores / denominator, 1)
9 max_cores = cores - reserve_cores
10 return min(fraction, max_cores)

This revision is now accepted and ready to land.Oct 19 2017, 7:01 PM
mmodell added a comment.Oct 19 2017, 7:02 PM
In D828#16446, @jenkins wrote:

Build has FAILED

/scap/main.py:28:1: F401 'multiprocessing' imported but unused
scap/main.py
28

can be removed

mmodell added a parent revision: D826: Cache submodules and use --reference to save space.Oct 19 2017, 10:51 PM
mmodell mentioned this in D826: Cache submodules and use --reference to save space.Oct 19 2017, 10:53 PM
demon updated this revision to Diff 2196.Oct 20 2017, 6:28 PM
  • Remove unused import
Harbormaster completed remote builds in B2384: Diff 2196.Oct 20 2017, 6:30 PM
mmodell accepted this revision.Oct 20 2017, 6:49 PM
Closed by commit rMSCA2561b82f6211: Unify "give me some number of CPUs I can use for subtasks" logic (authored by GerritToPhabBot). · Explain WhyOct 20 2017, 7:54 PM
This revision was automatically updated to reflect the committed changes.

Revision Contents
Changeset List

PathSize
scap/
7 lines
4 lines
8 lines

Diff 2197

View Options

scap/main.py

Loading...
View Options

scap/tasks.py

Loading...
View Options

scap/utils.py

Loading...
Content licensed under Creative Commons Attribution-ShareAlike 3.0 (CC-BY-SA) unless otherwise noted; code licensed under GNU General Public License (GPL) or other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL