Page MenuHomePhabricator

Zuul-cloner should use hard links when fetching from cache-dir
Closed, ResolvedPublic

Related Objects

Event Timeline

Krinkle created this task.Apr 24 2015, 2:56 AM
Krinkle raised the priority of this task from to Needs Triage.
Krinkle updated the task description. (Show Details)
Krinkle moved this task to Enhancements on the Zuul board.
Krinkle added a subscriber: Krinkle.
Restricted Application added a subscriber: Aklapper. · View Herald TranscriptApr 24 2015, 2:56 AM
hashar claimed this task.Apr 28 2015, 2:26 PM
hashar added a subscriber: hashar.

Our Zuul .deb packages need to incorporate https://review.openstack.org/#/c/117626/4

I will handle the patch management.

hashar triaged this task as High priority.Apr 28 2015, 2:28 PM

Change 207442 had a related patch set uploaded (by Hashar):
zuul-cloner can now hardlinks from cache-dir

https://gerrit.wikimedia.org/r/207442

Change 207442 merged by Hashar:
zuul-cloner can now hardlinks from cache-dir

https://gerrit.wikimedia.org/r/207442

Change 207444 had a related patch set uploaded (by Hashar):
Merge branch 'debian/precise-wikimedia' into debian/trusty-wikimedia

https://gerrit.wikimedia.org/r/207444

Change 207444 merged by Hashar:
Merge branch 'debian/precise-wikimedia' into debian/trusty-wikimedia

https://gerrit.wikimedia.org/r/207444

hashar added a comment.EditedApr 29 2015, 2:03 PM

I have incorporated upstream patch https://review.openstack.org/#/c/117626/4 to our Precise and Trusty packages. They are available at: http://people.wikimedia.org/~hashar/debs/zuul_2.0.0-304-g685ca22-wmf2/ and I have copied them under our project /root/.

To manually install them on our instances one can:

Precise

sudo dpkg -i /data/project/root/zuul_2.0.0-304-g685ca22-wmf2precise1_amd64.deb

Trusty

sudo dpkg -i /data/project/root/zuul_2.0.0-304-g685ca22-wmf2trusty1_amd64.deb

zuul-cloner --version would show the new version.

From the code that is essentially a noop since there we do not use the git cache dir yet.

Did an install on integration-slave-precise-1011 had to install python-six, maybe the package that originally got installed on that instance did not have it as a dependency. Easy to fix though.

# dpkg -i /data/project/root/zuul_2.0.0-304-g685ca22-wmf2precise1_amd64.deb
...
dpkg: dependency problems prevent configuration of zuul:
 zuul depends on python-six (>= 1.6.0); however:
  Package python-six is not installed.
dpkg: error processing zuul (--install):
 dependency problems - leaving unconfigured
Processing triggers for ureadahead ...
Errors were encountered while processing:
 zuul

python-six is thus registered as a missing package but dpkg does not process it, apt-get would though so:

# apt-get -f install
Reading package lists... Done
Building dependency tree       
Reading state information... Done
Correcting dependencies... Done
The following extra packages will be installed:
  python-six zuul
Suggested packages:
  jenkins
The following NEW packages will be installed:
  python-six
The following packages will be DOWNGRADED:
  zuul
0 upgraded, 1 newly installed, 1 downgraded, 0 to remove and 29 not upgraded.
1 not fully installed or removed.
Need to get 2,930 kB of archives.
After this operation, 81.9 kB of additional disk space will be used.
Do you want to continue [Y/n]? 
Get:1 http://apt.wikimedia.org/wikimedia/ precise-wikimedia/universe python-six all 1.9.0-1~precise1 [10.8 kB]
Get:2 http://apt.wikimedia.org/wikimedia/ precise-wikimedia/main zuul amd64 2.0.0-304-g685ca22-wmf1precise1 [2,920 kB]
Fetched 2,930 kB in 0s (21.8 MB/s)
Selecting previously unselected package python-six.
(Reading database ... 103166 files and directories currently installed.)
Unpacking python-six (from .../python-six_1.9.0-1~precise1_all.deb) ...
Setting up python-six (1.9.0-1~precise1) ...
Setting up zuul (2.0.0-304-g685ca22-wmf2precise1) ...
 * Zuul Server: /etc/default/zuul is not set to START_DAEMON=1: exiting
 * Zuul Merger: /etc/default/zuul-merger is not set to START_DAEMON=1: exiting

And:

$ zuul-cloner --version
Zuul version: 2.0.0-304-g685ca22-wmf2precise1
$ apt-cache policy zuul
zuul:
  Installed: 2.0.0-304-g685ca22-wmf2precise1
  Candidate: 2.0.0-304-g685ca22-wmf1precise1
  Version table:
 *** 2.0.0-304-g685ca22-wmf2precise1 0
        100 /var/lib/dpkg/status
     2.0.0-304-g685ca22-wmf1precise1 0
       1001 http://apt.wikimedia.org/wikimedia/ precise-wikimedia/main amd64 Packages

To revert, just uninstall zuul and reinstall it with apt-get, that will bring the stable version from apt.wikimedia.org.

I have manually upgraded zuul on all slaves. zuul-cloner has:

--cache-no-hardlinks CACHE_NO_HARDLINKS
                      force git-clone to never use hardlinks whenfetching
                      from the cache directory.

I have mentioned it on upstream Gerrit change. Jeremy Stanley noticed the hard linking is skipped when the mirror repository belong to another user.

That is why I proposed to use git clone --shared on T93703#1144542. That set .git/objects/info/alternates to grab objects regardless of file ownership (just need to be readable).

Change 252927 had a related patch set uploaded (by Hashar):
Refresh patch for zuul-cloner --cache-no-hardlinks

https://gerrit.wikimedia.org/r/252927

Change 252927 merged by Hashar:
Refresh patch for zuul-cloner --cache-no-hardlinks

https://gerrit.wikimedia.org/r/252927

Change 252933 had a related patch set uploaded (by Hashar):
Merge branch 'debian/precise-wikimedia' into debian/trusty-wikimedia

https://gerrit.wikimedia.org/r/252933

Change 252933 merged by Hashar:
Merge branch 'debian/precise-wikimedia' into debian/trusty-wikimedia

https://gerrit.wikimedia.org/r/252933

Change 252934 had a related patch set uploaded (by Hashar):
Merge branch 'debian/precise-wikimedia' into debian/jessie-wikimedia

https://gerrit.wikimedia.org/r/252934

When populating the git cache, I went using bare repositories which are suffixed with '.git'. Zuul-cloner does not recognize them, had to patch it up https://review.openstack.org/#/c/249207/

Change 255124 had a related patch set uploaded (by Hashar):
zuul-cloner now recognizes bare repos in cache

https://gerrit.wikimedia.org/r/255124

Change 255124 merged by Hashar:
zuul-cloner now recognizes bare repos in cache

https://gerrit.wikimedia.org/r/255124

Change 252934 merged by Hashar:
Merge branch 'debian/precise-wikimedia' into debian/jessie-wikimedia

https://gerrit.wikimedia.org/r/252934

Change 255541 had a related patch set uploaded (by Hashar):
Merge branch 'debian/precise-wikimedia' into debian/trusty-wikimedia

https://gerrit.wikimedia.org/r/255541

Change 255541 merged by Hashar:
Merge branch 'debian/precise-wikimedia' into debian/trusty-wikimedia

https://gerrit.wikimedia.org/r/255541

Change 255543 had a related patch set uploaded (by Hashar):
Merge branch 'debian/precise-wikimedia' into debian/jessie-wikimedia

https://gerrit.wikimedia.org/r/255543

Change 255543 merged by Hashar:
Merge branch 'debian/precise-wikimedia' into debian/jessie-wikimedia

https://gerrit.wikimedia.org/r/255543

I have cherry picked my patch and regenerated all packages. Now pending publishing to apt.wikimedia.org T119714

T119714 got us zuul_2.1.0-60-g1cc37f7-wmf4 everywhere. So we should now be able to use hard-links.

Nodepool snapshots were not properly upgrading packages from apt and were left with the version that was in the image. Forcing an apt-get update ( https://gerrit.wikimedia.org/r/260040 ) before apt-get upgrade solved it.

That being said, I connected to nodepool instant and used the cache-dir system:

jenkins@ci-jessie-wikimedia-15088:~$ time zuul-cloner --color -v --cache-dir /srv/git --branch=production https://gerrit.wikimedia.org/r/p operations/puppet
INFO:zuul.CloneMapper:Workspace path set to: /home/jenkins
DEBUG:zuul.CloneMapper:Using operations/puppet as destination (unmatched)
INFO:zuul.CloneMapper:Mapping projects to workspace...
INFO:zuul.CloneMapper:  operations/puppet -> /home/jenkins/operations/puppet
DEBUG:zuul.CloneMapper:Checking overlap in destination directories...
INFO:zuul.CloneMapper:Expansion completed.
INFO:zuul.Cloner:Preparing 1 repositories

INFO:zuul.Cloner:Creating repo operations/puppet from cache /srv/git/operations/puppet.git
                                                 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

INFO:zuul.Cloner:Updating origin remote in repo operations/puppet to https://gerrit.wikimedia.org/r/p/operations/puppet
DEBUG:zuul.Repo:Resetting repository /home/jenkins/operations/puppet
DEBUG:zuul.Repo:Updating repository /home/jenkins/operations/puppet
INFO:zuul.Cloner:upstream repo has branch production
DEBUG:zuul.Cloner:Project operations/puppet in Zuul does not have ref production
INFO:zuul.Cloner:Falling back to branch production
DEBUG:zuul.Repo:Checking out remotes/origin/production
INFO:zuul.Cloner:Prepared operations/puppet repo with branch production at commit f386e3ca05066da6b33781c24a893f3cc26d02d3
INFO:zuul.Cloner:Prepared all repositories

real	0m4.636s
user	0m1.868s
sys	0m0.564s

So that is fixed by the patch we have proposed upstream https://review.openstack.org/#/c/117626/ but already included in Wikimedia Debian package.

hashar removed a project: Patch-For-Review.
hashar set Security to None.
hashar moved this task from Backlog to Patch proposed upstream on the Upstream board.
hashar removed hashar as the assignee of this task.Mar 27 2017, 1:47 PM
hashar lowered the priority of this task from High to Low.

That is fixed via a patch we maintain in the Zuul Debian package.

Krinkle removed a subscriber: Krinkle.Mar 27 2017, 10:29 PM
Krinkle closed this task as Resolved.Sep 28 2018, 7:37 PM
Krinkle claimed this task.

This doesn't seem particularly relevant anymore with the newer Docker-based quibble images and git-caches. I don't actually know whether it uses hardlinks or not, but as long as we make shallow clones and have good caches, the overhead is insignificant even if we don't use hardlinks.

On the Docker container we have the git cache mounted read-only from the host into the container as /srv/git. Since they are different partitions, git/linux would always hard copy the files. From the tests I did for Quibble, it is reasonably fast, most probably because all the mirrored repositories are already in the Linux disk cache.

So yeah +1 on declining. One less trouble some custom fix we will be able to drop in our Zuul.