Update jobs to use zuul-cloner with git cache
Closed, ResolvedPublic
Actions

Assigned To

Authored By

	Krinkle
	Apr 24 2015, 12:26 AM

Description

Once T96687 is resolved, we need to update our use of zuul-cloner to actually pass the --cache-dir.

Related Objects
Search...

Status	Subtype	Assigned	Task
Resolved		Krinkle	T91211 gallium and lanthanum disks full (tracking)
Declined		None	T91707 L10n-bot should not force-merge / override Jenkins (breaks the build)
Declined		hashar	T93703 reduce copies of mediawiki/core in workspaces
Resolved	PRODUCTION ERROR	Krinkle	T86730 Zuul-cloner failing to acquire .git lock sometimes
Declined		hashar	T96627 Jenkins jobs must wipe workspace
Resolved		hashar	T97098 Update jobs to use zuul-cloner with git cache
Declined		hashar	T96687 Set up git replication on integration slaves
Resolved		Krinkle	T97106 Zuul-cloner should use hard links when fetching from cache-dir
Resolved		hashar	T118340 Upload new Zuul packages on apt.wikimedia.org for Precise / Trusty / Jessie
Resolved		Dzahn	T119714 Upload new Zuul packages zuul_2.1.0-60-g1cc37f7-wmf4 on apt.wikimedia.org for Precise / Trusty / Jessie
Resolved		hashar	T119741 integration-zuul-layoutvalidation fails because of python-daemon 2.1
Resolved		hashar	T87294 Nodepool images need Gerrit mirror for git-clone performance
Resolved		Andrew	T113394 Evaluate impact of giant VM images on labs infrastructure

Event Timeline

I've tested this on integration-trusty-slave-1021 with mediawiki/core and noticed that it's not much faster to copy from /mnt/git than to clone from gerrit.wikimedia.org. In some cases it was even faster to clone from gerrit for some reason.

A few tests (each on an empty workspace)

krinkle at integration-slave-trusty-1021.eqiad.wmflabs in /mnt/jenkins-workspace/krinkle1
$ zuul-cloner --color --verbose --map /srv/deployment/integration/slave-scripts/etc/zuul-clonemap.yaml --workspace src --cache-dir /mnt/git https://gerrit.wikimedia.org/r/p mediawiki/core
..

01: Copy from cache-dir on same disk: 5 minutes
02: Copy from cache-dir on same disk: 2 minutes
03: Copy from cache-dir on same disk: 3 minutes

krinkle at integration-slave-trusty-1021.eqiad.wmflabs in /mnt/jenkins-workspace/krinkle4
$ zuul-cloner --color --verbose --map /srv/deployment/integration/slave-scripts/etc/zuul-clonemap.yaml --workspace src https://gerrit.wikimedia.org/r/p mediawiki/core
..

01: Fetch from gerrit and copy to disk: 3 minutes
02: Fetch from gerrit and copy to disk: 5 minutes
03: Fetch from gerrit and copy to disk: 3 minutes

I guess the real benefit of caching isn't visible until we change zuul-cloner to use hard links.

Krinkle added a subtask: T96687: Set up git replication on integration slaves.Apr 24 2015, 12:27 AM

Krinkle mentioned this in T97106: Zuul-cloner should use hard links when fetching from cache-dir .Apr 24 2015, 2:56 AM

I applied https://review.openstack.org/#/c/117626/4 to zuul on the depooled integration-slave-trusty-1021. A quick test confirms it works properly and speeds up the cloning process significantly:

$ zuul-cloner --color --verbose --map /srv/deployment/integration/slave-scripts/etc/zuul-clonemap.yaml --workspace src --cache-dir /mnt/git https://gerrit.wikimedia.org/r/p mediawiki/core
INFO:zuul.Cloner:Loaded map containing 4 rules
INFO:zuul.CloneMapper:Workspace path set to: src
INFO:zuul.CloneMapper:Mapping projects to workspace...
INFO:zuul.CloneMapper:  mediawiki/core -> src
DEBUG:zuul.CloneMapper:Checking overlap in destination directories...
INFO:zuul.CloneMapper:Expansion completed.
INFO:zuul.Cloner:Preparing 1 repositories
INFO:zuul.Cloner:Creating repo mediawiki/core from cache /mnt/git/mediawiki/core
# ^ Notice the absence of "file://"
INFO:zuul.Cloner:Updating origin remote in repo mediawiki/core to https://gerrit.wikimedia.org/r/p/mediawiki/core
DEBUG:zuul.Repo:Updating repository src
DEBUG:zuul.Cloner:upstream repo has branch master
DEBUG:zuul.Cloner:Fetched ref refs/zuul/master/Zc9f64d96d09b416ca1dcf968f3f10a1b from mediawiki/core
DEBUG:zuul.Repo:Checking out 7c9923f91b48aab41d3b432277b56f36c38710cb
INFO:zuul.Cloner:Prepared mediawiki/core repo with commit 7c9923f91b48aab41d3b432277b56f36c38710cb
INFO:zuul.Cloner:Prepared all repositories

Copy from cache-dir on same disk: 39 seconds

Krinkle renamed this task from Update jobs to use zuul-cloner with git cache via hard links to Update jobs to use zuul-cloner with git cache.Apr 28 2015, 2:23 PM

Krinkle moved this task from Untriaged to Next on the Continuous-Integration-Infrastructure board.Apr 28 2015, 2:35 PM

Legoktm mentioned this in T110512: mediawiki-core-phplint clone the whole repo from Zuul and times out.Aug 27 2015, 3:37 PM

hashar closed subtask T96687: Set up git replication on integration slaves as Declined.Oct 6 2015, 12:29 PM

hashar added a subtask: T87294: Nodepool images need Gerrit mirror for git-clone performance.