HomePhabricator

Support to clone repositories in parallel
8cdc0679ee85Unpublished

Authored by hashar on Mar 1 2019, 9:13 PM.

Unpublished Commit · Learn More

Not On Permanent Ref: This commit is not an ancestor of any permanent ref.

Description

Support to clone repositories in parallel

Quibble clones each of the repositories serially. It does so by invoking
zuul.Cloner() which has a very lame foreach implementation for cloning.

When only one worker is requested, use Cloner.execute() as before. This
way the new behavior is feature flagged.
Add a new option --git-parallel, defaulting to one worker.

Logging of the clone operations is multiplexed and hard to gasp:

  • remove Zuul.Repo() DEBUG level, it added little value.
  • each clone logging is now suffixed with the repository name

If mediawiki/core has not been cloned yet, clone it first which I think
might avoid errors due to git clone and paths conflicts.

Timing with a local --git-cache:

git-parallelDuration
1630 seconds
150 seconds

Bug: T211701
Change-Id: If05c28b350365bfca5268c4dc776d5e4f85bffb5

Details

Committed
hasharMar 4 2019, 10:44 AM
Parents
rQUIBBLE6324e1ed2856: Upgrade to flake8 3.7.x
Branches
Unknown
Tags
Unknown
References
refs/changes/64/493764/2
ChangeId
If05c28b350365bfca5268c4dc776d5e4f85bffb5