Differential D828

Unify "give me some number of CPUs I can use for subtasks" logic
ClosedPublic
Actions

Authored by demon on Oct 19 2017, 6:28 PM.

Details

Reviewers

mmodell

Group Reviewers

Release-Engineering-Team

Commits

rMSCA2561b82f6211: Unify "give me some number of CPUs I can use for subtasks" logic

Patch without arc

git checkout -b D828 && curl -L https://phabricator.wikimedia.org/D828?download=true | git apply

Summary

Basically we always want to use as many cores as possible, leaving like 2 or so to be nice to other processes

Diff Detail

Repository

rMSCA Scap

Lint

Automatic diff as part of commit; lint not applicable.

Unit

Automatic diff as part of commit; unit tests not applicable.

Event Timeline

demon created this revision.Oct 19 2017, 6:28 PM

Restricted Application added a reviewer: mmodell. · View Herald TranscriptOct 19 2017, 6:28 PM

Restricted Application added a reviewer: Release-Engineering-Team. · View Herald Transcript

Restricted Application added a project: Release-Engineering-Team. · View Herald Transcript

Build has FAILED

Link to build: https://integration.wikimedia.org/ci/job/phabricator-jessie-diffs/777/
See console output for more information: https://integration.wikimedia.org/ci/job/phabricator-jessie-diffs/777/console

Harbormaster failed remote builds in B2373: Diff 2186!Oct 19 2017, 6:29 PM

Build has FAILED

Link to build: https://integration.wikimedia.org/ci/job/differential-docker-test/284/
See console output for more information: https://integration.wikimedia.org/ci/job/differential-docker-test/284/console

Probably better than my exercise in verbosity:

P6156 parallelism

1	def parallelism(denominator=2, reserve_cores=1):
2	""" Calculate the number of parallel jobs to run
3	This is taken from the total available cpu cores divided by denominator
4	reserve_cores allows for reserving a fixed number of unused cores.
5	By default, max(cpus/2, cpus-1)
6	"""
7	cores = multiprocessing.cpu_count()
8	fraction = max(cores / denominator, 1)
9	max_cores = cores - reserve_cores
10	return min(fraction, max_cores)

This revision is now accepted and ready to land.Oct 19 2017, 7:01 PM

In D828#16446, @jenkins wrote:

Build has FAILED

/scap/main.py:28:1: F401 'multiprocessing' imported but unused

scap/main.py
28	can be removed

mmodell added a parent revision: D826: Cache submodules and use --reference to save space.Oct 19 2017, 10:51 PM

mmodell mentioned this in D826: Cache submodules and use --reference to save space.Oct 19 2017, 10:53 PM

Remove unused import

Harbormaster completed remote builds in B2384: Diff 2196.Oct 20 2017, 6:30 PM

mmodell accepted this revision.Oct 20 2017, 6:49 PM

Closed by commit rMSCA2561b82f6211: Unify "give me some number of CPUs I can use for subtasks" logic (authored by GerritToPhabBot). · Explain WhyOct 20 2017, 7:54 PM

This revision was automatically updated to reflect the committed changes.

Revision Contents
Changeset List

			Path	Packages
M			scap/main.py (7 lines)
M			scap/tasks.py (4 lines)
M			scap/utils.py (8 lines)

Diff	ID	Base	Description	Created	Lint	Unit
Base			Base
Diff 1	2186	9d86135		Oct 19 2017, 6:28 PM	★	★
Diff 2	2196	9d86135	- Remove unused import	Oct 20 2017, 6:28 PM	★	★
Diff 3	2197	53b3bcf	rMSCA2561b82f6211a3a6c84d6bafe587b007df1cab3e	Oct 20 2017, 7:53 PM	★	★

	Status	Author	Revision
	Closed	demon	D828 Unify "give me some number of CPUs I can use for subtasks" logic
	Closed	mmodell	D826 Cache submodules and use --reference to save space

Diff 2197

View Options

scap/main.py

View Options

scap/tasks.py

View Options

Unify "give me some number of CPUs I can use for subtasks" logicClosedPublicActions

Details

Diff Detail

Event Timeline

Revision ContentsChangeset List

Diff 2197

scap/main.py

scap/tasks.py

scap/utils.py

Unify "give me some number of CPUs I can use for subtasks" logic
ClosedPublic
Actions

Revision Contents
Changeset List