HomePhabricator

Support to clone repositories in parallel
8cdc0679ee85Unpublished

Unpublished Commit · Learn More

Not On Permanent Ref: This commit is not an ancestor of any permanent ref.
This commit no longer exists in the repository. It may have been part of a branch which was deleted.This commit has been deleted in the repository: it is no longer reachable from any branch, tag, or ref.

Description

Support to clone repositories in parallel

Quibble clones each of the repositories serially. It does so by invoking
zuul.Cloner() which has a very lame foreach implementation for cloning.

When only one worker is requested, use Cloner.execute() as before. This
way the new behavior is feature flagged.
Add a new option --git-parallel, defaulting to one worker.

Logging of the clone operations is multiplexed and hard to gasp:

  • remove Zuul.Repo() DEBUG level, it added little value.
  • each clone logging is now suffixed with the repository name

If mediawiki/core has not been cloned yet, clone it first which I think
might avoid errors due to git clone and paths conflicts.

Timing with a local --git-cache:

git-parallelDuration
1630 seconds
150 seconds

Bug: T211701
Change-Id: If05c28b350365bfca5268c4dc776d5e4f85bffb5

Details

Provenance
hasharAuthored on Mar 1 2019, 9:13 PM
ChangeId
If05c28b350365bfca5268c4dc776d5e4f85bffb5

Commit No Longer Exists

This commit no longer exists in the repository.