Page MenuHomePhabricator

Use git clone depth=1 with zuul?
Open, NormalPublic

Description

While following the instructions for reproducing a CI build with quibble, I noticed that cloning repos is taking a long time and is causing time out problems with the example command. Would it be simple/possible to change the clone depth to 1? Cloning core is ~12 seconds with depth=1, but 1 minute 23 seconds otherwise, and if you are reproducing a build with 38 repos this adds up.

Event Timeline

kostajh created this task.Nov 26 2018, 7:05 PM
Restricted Application added a subscriber: Aklapper. · View Herald TranscriptNov 26 2018, 7:05 PM
hashar edited projects, added Quibble; removed Zuul.Mar 18 2019, 2:44 PM

Sorry for lack of follow up, I have missed the task creation notification.

Quibble reuses the logic from zuul-cloner and it requires a full clone in order to fetch the proper branch and or patch. That is necessary since we do not necessarily know the branch or revision to checkout at time of cloning. From Quibble zuul/lib/cloner.py in prepareRepo:

"""Clone a repository for project at dest and apply a reference
suitable for testing. The reference lookup is attempted in this order:

 1) The indicated revision for specific project
 2) Zuul reference for the indicated branch
 3) Zuul reference for the master branch
 4) The tip of the indicated branch
 5) The tip of the master branch

If an "indicated revision" is specified for this project, and we are
unable to meet this requirement, we stop attempting to check this
repo out and raise a zuul.exceptions.RevNotFound exception.

The "indicated branch" is one of the following:

 A) The project-specific override branch (from project_branches arg)
 B) The user specified branch (from the branch arg)
 C) ZUUL_BRANCH (from the zuul_branch arg)
"""

So that theorically we could test an extension master branch against a specific release branch of mediawiki and extensions. Also at the time of cloning, we do not know which revision or branch will ultimately be fetched and checked out, we thus need a full clone.

Anyway. It is definitely slow to clone from scratch. One want to use --git-cache:

--git-cache GIT_CACHE
                      Path to bare git repositories to speed up git
                      cloneoperation. Passed to zuul-cloner as --cache-dir.
                      In Docker: "/srv/git", else "ref"

Zuul will check under GIT_CACHE whether the repository to clone exists (either bare or non-bare repository) and attempt to clone from it. Then it will update it from gerrit.wikimedia.org.

Locally, I have all my repositories under my $HOME/projects which are cloned matching the Gerrit layout hierarchy. Eg:

/home/hashar/projects/mediawiki/core
/home/hashar/projects/mediawiki/extensions/BoilerPlate
/home/hashar/projects/mediawiki/skins/Vector

And I would then use:

quibble --git-cache /home/hashar/projects mediawiki/extensions/BoilerPlate

It does:

INFO:quibble.cmd:Projects: mediawiki/core, mediawiki/skins/Vector, mediawiki/vendor, mediawiki/extensions/BoilerPlate

# Map each project to destination:
INFO:zuul.CloneMapper:  mediawiki/core -> /tmp/T210435/src
INFO:zuul.CloneMapper:  mediawiki/skins/Vector -> /tmp/T210435/src/skins/Vector
INFO:zuul.CloneMapper:  mediawiki/vendor -> /tmp/T210435/src/vendor
INFO:zuul.CloneMapper:  mediawiki/extensions/BoilerPlate -> /tmp/T210435/src/extensions/BoilerPlate
...
# Clone BoilerPlate from GIT_CACHE
INFO:zuul.Cloner:Creating repo mediawiki/extensions/BoilerPlate from cache /home/hashar/projects/mediawiki/extensions/BoilerPlate

# Update it from the canonical repository (Gerrit):
INFO:zuul.Cloner:Updating origin remote in repo mediawiki/extensions/BoilerPlate to https://gerrit.wikimedia.org/r/p/mediawiki/extensions/BoilerPlate
DEBUG:zuul.Repo:Updating repository /tmp/T210435/src/extensions/BoilerPlate

# Defaults to `master` since I have not mentioned a default branch (`--branch` or `--project-branch mediawiki/extensions/BoilerPlate=XXXX`).
INFO:zuul.Cloner:Falling back to branch master

INFO:zuul.Cloner:Prepared mediawiki/extensions/BoilerPlate repo with branch master at commit 5a6ec84a72405d839446db937f161e9d97a345cf

I guess the documentation is definitely lacking on that front :-/

Thanks, this is very helpful!