While following the instructions for reproducing a CI build with quibble, I noticed that cloning repos is taking a long time and is causing time out problems with the example command. Would it be simple/possible to change the clone depth to 1? Cloning core is ~12 seconds with depth=1, but 1 minute 23 seconds otherwise, and if you are reproducing a build with 38 repos this adds up.
Sorry for lack of follow up, I have missed the task creation notification.
Quibble reuses the logic from zuul-cloner and it requires a full clone in order to fetch the proper branch and or patch. That is necessary since we do not necessarily know the branch or revision to checkout at time of cloning. From Quibble zuul/lib/cloner.py in prepareRepo:
"""Clone a repository for project at dest and apply a reference suitable for testing. The reference lookup is attempted in this order: 1) The indicated revision for specific project 2) Zuul reference for the indicated branch 3) Zuul reference for the master branch 4) The tip of the indicated branch 5) The tip of the master branch If an "indicated revision" is specified for this project, and we are unable to meet this requirement, we stop attempting to check this repo out and raise a zuul.exceptions.RevNotFound exception. The "indicated branch" is one of the following: A) The project-specific override branch (from project_branches arg) B) The user specified branch (from the branch arg) C) ZUUL_BRANCH (from the zuul_branch arg) """
So that theorically we could test an extension master branch against a specific release branch of mediawiki and extensions. Also at the time of cloning, we do not know which revision or branch will ultimately be fetched and checked out, we thus need a full clone.
Anyway. It is definitely slow to clone from scratch. One want to use --git-cache:
--git-cache GIT_CACHE Path to bare git repositories to speed up git cloneoperation. Passed to zuul-cloner as --cache-dir. In Docker: "/srv/git", else "ref"
Zuul will check under GIT_CACHE whether the repository to clone exists (either bare or non-bare repository) and attempt to clone from it. Then it will update it from gerrit.wikimedia.org.
Locally, I have all my repositories under my $HOME/projects which are cloned matching the Gerrit layout hierarchy. Eg:
/home/hashar/projects/mediawiki/core /home/hashar/projects/mediawiki/extensions/BoilerPlate /home/hashar/projects/mediawiki/skins/Vector
And I would then use:
quibble --git-cache /home/hashar/projects mediawiki/extensions/BoilerPlate
INFO:quibble.cmd:Projects: mediawiki/core, mediawiki/skins/Vector, mediawiki/vendor, mediawiki/extensions/BoilerPlate # Map each project to destination: INFO:zuul.CloneMapper: mediawiki/core -> /tmp/T210435/src INFO:zuul.CloneMapper: mediawiki/skins/Vector -> /tmp/T210435/src/skins/Vector INFO:zuul.CloneMapper: mediawiki/vendor -> /tmp/T210435/src/vendor INFO:zuul.CloneMapper: mediawiki/extensions/BoilerPlate -> /tmp/T210435/src/extensions/BoilerPlate ... # Clone BoilerPlate from GIT_CACHE INFO:zuul.Cloner:Creating repo mediawiki/extensions/BoilerPlate from cache /home/hashar/projects/mediawiki/extensions/BoilerPlate # Update it from the canonical repository (Gerrit): INFO:zuul.Cloner:Updating origin remote in repo mediawiki/extensions/BoilerPlate to https://gerrit.wikimedia.org/r/p/mediawiki/extensions/BoilerPlate DEBUG:zuul.Repo:Updating repository /tmp/T210435/src/extensions/BoilerPlate # Defaults to `master` since I have not mentioned a default branch (`--branch` or `--project-branch mediawiki/extensions/BoilerPlate=XXXX`). INFO:zuul.Cloner:Falling back to branch master INFO:zuul.Cloner:Prepared mediawiki/extensions/BoilerPlate repo with branch master at commit 5a6ec84a72405d839446db937f161e9d97a345cf
I guess the documentation is definitely lacking on that front :-/