Quibble clones each of the repositories serially. It does so by invoking zuul-cloner which has a very lame foreach implementation for cloning.
Potentially we could use a threadpool or use git submodules and delegate parallelization to it.
GOTCHA: there might be name clash when cloning. Eg if one clones mediawiki/extensions/Foo first, that creates the directory ./extensions and then prevent us from cloning mediawiki/core in it (iirc). So order of cloning does matter.
| Status | Subtype | Assigned | Task | ||
|---|---|---|---|---|---|
| Resolved | hashar | T211701 Quibble should clone repositories in parallel | |||
| Resolved | hashar | T219647 Upgrade CI jobs to Quibble 0.0.31 | |||
| Resolved | hashar | T219645 ffmpeg is no more in Jessie, have to rethink CI jobs / containers | |||
| Resolved | None | T219683 Rebuild docker-registry.wikimedia.org/wikimedia-jessie to drop jessie-update/jessie-backports | |||
| Resolved | Joe | T219580 Remove backports from wikimedia-jessie | |||
| Resolved | Joe | T219747 Rebuild production docker images to pick up update in wikimedia-jessie | |||
| Resolved | Jdforrester-WMF | T219748 Rebuild integration/config images based on jessie | |||
| Resolved | bd808 | T219751 Rebuild toollabs docker images based on wikimedia-jessie | |||
| Resolved | hashar | T218735 sury.org packages do not validate gpg | |||
| Resolved | hashar | T219786 Quibble 0.0.30 misses quibble/mediawiki/local_settings.php | |||
| Duplicate | hashar | T219804 Quibble 0.0.30 misses quibble/mediawiki/local_settings.php |
Change 493764 had a related patch set uploaded (by Hashar; owner: Hashar):
[integration/quibble@master] Support to clone repositories in parallel
Change 493764 merged by jenkins-bot:
[integration/quibble@master] Support to clone repositories in parallel
@hashar thanks for adding this feature! Have you considered making the default number of workers 8/16/whatever?
It definitely should defaults to some higher number of workers. Exact value left to be defined.
I have let it default to 1 for now to still rely on the old code:
if workers == 1:
return zuul_cloner.execute()
// else slightly reimplemented code with parallelismSo when doing the upgrade on CI, no change is introduced. Then I can later enable it by turning the feature flag --git-parallel. If that works fine, for sure it should definitely has a useful default (8 sounds good) :]
So I am playing it safe :]
So this is possible since Quibble 0.0.31 (0.0.30 is broken). Still have to update the Jenkins jobs to pass --git-parallel 8.
Change 500689 had a related patch set uploaded (by Hashar; owner: Hashar):
[integration/config@master] Clone repos in parallel in wmf-quibble
Change 500689 merged by jenkins-bot:
[integration/config@master] Clone repos in parallel in wmf-quibble
I have updated the wmf-quibble jobs for a starter. Will see later at generalizing the parameter to all jobs and probably just make it the default in a future version of Quibble.
Change 501398 had a related patch set uploaded (by Hashar; owner: Hashar):
[integration/config@master] Add --git-parallel=8 to most Quibble jobs
Left to do I guess is to have the command default to 8 worker threads which should be fine?! Maybe.
Change 501398 merged by jenkins-bot:
[integration/config@master] Add --git-parallel=8 to most Quibble jobs
Change 503003 had a related patch set uploaded (by Hashar; owner: Hashar):
[integration/quibble@master] Default to use 4 git workers
Change 503003 merged by jenkins-bot:
[integration/quibble@master] Default to use 4 git workers