Page MenuHomePhabricator

Update jobs to use zuul-cloner with git cache
Closed, ResolvedPublic

Description

Once T96687 is resolved, we need to update our use of zuul-cloner to actually pass the --cache-dir.

Event Timeline

Krinkle created this task.EditedApr 24 2015, 12:26 AM
Krinkle raised the priority of this task from to High.
Krinkle updated the task description. (Show Details)
Krinkle added subscribers: Aklapper, greg, hashar, Krinkle.

I've tested this on integration-trusty-slave-1021 with mediawiki/core and noticed that it's not much faster to copy from /mnt/git than to clone from gerrit.wikimedia.org. In some cases it was even faster to clone from gerrit for some reason.

A few tests (each on an empty workspace)

krinkle at integration-slave-trusty-1021.eqiad.wmflabs in /mnt/jenkins-workspace/krinkle1
$ zuul-cloner --color --verbose --map /srv/deployment/integration/slave-scripts/etc/zuul-clonemap.yaml --workspace src --cache-dir /mnt/git https://gerrit.wikimedia.org/r/p mediawiki/core
..
  • 01: Copy from cache-dir on same disk: 5 minutes
  • 02: Copy from cache-dir on same disk: 2 minutes
  • 03: Copy from cache-dir on same disk: 3 minutes
krinkle at integration-slave-trusty-1021.eqiad.wmflabs in /mnt/jenkins-workspace/krinkle4
$ zuul-cloner --color --verbose --map /srv/deployment/integration/slave-scripts/etc/zuul-clonemap.yaml --workspace src https://gerrit.wikimedia.org/r/p mediawiki/core
..
  • 01: Fetch from gerrit and copy to disk: 3 minutes
  • 02: Fetch from gerrit and copy to disk: 5 minutes
  • 03: Fetch from gerrit and copy to disk: 3 minutes

I guess the real benefit of caching isn't visible until we change zuul-cloner to use hard links.

Krinkle renamed this task from Zuul-cloner should use git cache via hard links to Update jobs to use zuul-cloner with git cache via hard links.Apr 24 2015, 4:07 AM
Krinkle set Security to None.

I applied https://review.openstack.org/#/c/117626/4 to zuul on the depooled integration-slave-trusty-1021. A quick test confirms it works properly and speeds up the cloning process significantly:

$ zuul-cloner --color --verbose --map /srv/deployment/integration/slave-scripts/etc/zuul-clonemap.yaml --workspace src --cache-dir /mnt/git https://gerrit.wikimedia.org/r/p mediawiki/core
INFO:zuul.Cloner:Loaded map containing 4 rules
INFO:zuul.CloneMapper:Workspace path set to: src
INFO:zuul.CloneMapper:Mapping projects to workspace...
INFO:zuul.CloneMapper:  mediawiki/core -> src
DEBUG:zuul.CloneMapper:Checking overlap in destination directories...
INFO:zuul.CloneMapper:Expansion completed.
INFO:zuul.Cloner:Preparing 1 repositories
INFO:zuul.Cloner:Creating repo mediawiki/core from cache /mnt/git/mediawiki/core
# ^ Notice the absence of "file://"
INFO:zuul.Cloner:Updating origin remote in repo mediawiki/core to https://gerrit.wikimedia.org/r/p/mediawiki/core
DEBUG:zuul.Repo:Updating repository src
DEBUG:zuul.Cloner:upstream repo has branch master
DEBUG:zuul.Cloner:Fetched ref refs/zuul/master/Zc9f64d96d09b416ca1dcf968f3f10a1b from mediawiki/core
DEBUG:zuul.Repo:Checking out 7c9923f91b48aab41d3b432277b56f36c38710cb
INFO:zuul.Cloner:Prepared mediawiki/core repo with commit 7c9923f91b48aab41d3b432277b56f36c38710cb
INFO:zuul.Cloner:Prepared all repositories

Copy from cache-dir on same disk: 39 seconds

Krinkle renamed this task from Update jobs to use zuul-cloner with git cache via hard links to Update jobs to use zuul-cloner with git cache.Apr 28 2015, 2:23 PM

From T97106#1890742 now has hardlink support from git bare repos simply by using --cache-dir /srv/git on Nodepool instances.

hashar closed this task as Resolved.Nov 3 2016, 9:10 PM
hashar claimed this task.

The macros are using zuul-cloner cache dir or the git plugin reference when relevant.