Related discussion in comments on:
Upstream patch (work in progress:
Add cache-hardlink argument to cloner
https://review.openstack.org/#/c/117626/1
| Status | Subtype | Assigned | Task | ||
|---|---|---|---|---|---|
| Resolved | Krinkle | T91211 gallium and lanthanum disks full (tracking) | |||
| Declined | None | T91707 L10n-bot should not force-merge / override Jenkins (breaks the build) | |||
| Declined | hashar | T93703 reduce copies of mediawiki/core in workspaces | |||
| Resolved | PRODUCTION ERROR | Krinkle | T86730 Zuul-cloner failing to acquire .git lock sometimes | ||
| Declined | hashar | T96627 Jenkins jobs must wipe workspace | |||
| Resolved | hashar | T97098 Update jobs to use zuul-cloner with git cache | |||
| Resolved | Krinkle | T97106 Zuul-cloner should use hard links when fetching from cache-dir | |||
| Resolved | hashar | T118340 Upload new Zuul packages on apt.wikimedia.org for Precise / Trusty / Jessie | |||
| Resolved | Dzahn | T119714 Upload new Zuul packages zuul_2.1.0-60-g1cc37f7-wmf4 on apt.wikimedia.org for Precise / Trusty / Jessie | |||
| Resolved | hashar | T119741 integration-zuul-layoutvalidation fails because of python-daemon 2.1 |
Our Zuul .deb packages need to incorporate https://review.openstack.org/#/c/117626/4
I will handle the patch management.
Change 207442 had a related patch set uploaded (by Hashar):
zuul-cloner can now hardlinks from cache-dir
Change 207444 had a related patch set uploaded (by Hashar):
Merge branch 'debian/precise-wikimedia' into debian/trusty-wikimedia
Change 207444 merged by Hashar:
Merge branch 'debian/precise-wikimedia' into debian/trusty-wikimedia
I have incorporated upstream patch https://review.openstack.org/#/c/117626/4 to our Precise and Trusty packages. They are available at: http://people.wikimedia.org/~hashar/debs/zuul_2.0.0-304-g685ca22-wmf2/ and I have copied them under our project /root/.
To manually install them on our instances one can:
Precise
sudo dpkg -i /data/project/root/zuul_2.0.0-304-g685ca22-wmf2precise1_amd64.deb
Trusty
sudo dpkg -i /data/project/root/zuul_2.0.0-304-g685ca22-wmf2trusty1_amd64.deb
zuul-cloner --version would show the new version.
From the code that is essentially a noop since there we do not use the git cache dir yet.
Did an install on integration-slave-precise-1011 had to install python-six, maybe the package that originally got installed on that instance did not have it as a dependency. Easy to fix though.
# dpkg -i /data/project/root/zuul_2.0.0-304-g685ca22-wmf2precise1_amd64.deb ... dpkg: dependency problems prevent configuration of zuul: zuul depends on python-six (>= 1.6.0); however: Package python-six is not installed. dpkg: error processing zuul (--install): dependency problems - leaving unconfigured Processing triggers for ureadahead ... Errors were encountered while processing: zuul
python-six is thus registered as a missing package but dpkg does not process it, apt-get would though so:
# apt-get -f install Reading package lists... Done Building dependency tree Reading state information... Done Correcting dependencies... Done The following extra packages will be installed: python-six zuul Suggested packages: jenkins The following NEW packages will be installed: python-six The following packages will be DOWNGRADED: zuul 0 upgraded, 1 newly installed, 1 downgraded, 0 to remove and 29 not upgraded. 1 not fully installed or removed. Need to get 2,930 kB of archives. After this operation, 81.9 kB of additional disk space will be used. Do you want to continue [Y/n]? Get:1 http://apt.wikimedia.org/wikimedia/ precise-wikimedia/universe python-six all 1.9.0-1~precise1 [10.8 kB] Get:2 http://apt.wikimedia.org/wikimedia/ precise-wikimedia/main zuul amd64 2.0.0-304-g685ca22-wmf1precise1 [2,920 kB] Fetched 2,930 kB in 0s (21.8 MB/s) Selecting previously unselected package python-six. (Reading database ... 103166 files and directories currently installed.) Unpacking python-six (from .../python-six_1.9.0-1~precise1_all.deb) ... Setting up python-six (1.9.0-1~precise1) ... Setting up zuul (2.0.0-304-g685ca22-wmf2precise1) ... * Zuul Server: /etc/default/zuul is not set to START_DAEMON=1: exiting * Zuul Merger: /etc/default/zuul-merger is not set to START_DAEMON=1: exiting
And:
$ zuul-cloner --version Zuul version: 2.0.0-304-g685ca22-wmf2precise1 $ apt-cache policy zuul zuul: Installed: 2.0.0-304-g685ca22-wmf2precise1 Candidate: 2.0.0-304-g685ca22-wmf1precise1 Version table: *** 2.0.0-304-g685ca22-wmf2precise1 0 100 /var/lib/dpkg/status 2.0.0-304-g685ca22-wmf1precise1 0 1001 http://apt.wikimedia.org/wikimedia/ precise-wikimedia/main amd64 Packages
To revert, just uninstall zuul and reinstall it with apt-get, that will bring the stable version from apt.wikimedia.org.
I have manually upgraded zuul on all slaves. zuul-cloner has:
--cache-no-hardlinks CACHE_NO_HARDLINKS
force git-clone to never use hardlinks whenfetching
from the cache directory.I have mentioned it on upstream Gerrit change. Jeremy Stanley noticed the hard linking is skipped when the mirror repository belong to another user.
That is why I proposed to use git clone --shared on T93703#1144542. That set .git/objects/info/alternates to grab objects regardless of file ownership (just need to be readable).
Change 252927 had a related patch set uploaded (by Hashar):
Refresh patch for zuul-cloner --cache-no-hardlinks
Change 252933 had a related patch set uploaded (by Hashar):
Merge branch 'debian/precise-wikimedia' into debian/trusty-wikimedia
Change 252933 merged by Hashar:
Merge branch 'debian/precise-wikimedia' into debian/trusty-wikimedia
Change 252934 had a related patch set uploaded (by Hashar):
Merge branch 'debian/precise-wikimedia' into debian/jessie-wikimedia
When populating the git cache, I went using bare repositories which are suffixed with '.git'. Zuul-cloner does not recognize them, had to patch it up https://review.openstack.org/#/c/249207/
Change 255124 had a related patch set uploaded (by Hashar):
zuul-cloner now recognizes bare repos in cache
Change 252934 merged by Hashar:
Merge branch 'debian/precise-wikimedia' into debian/jessie-wikimedia
Change 255541 had a related patch set uploaded (by Hashar):
Merge branch 'debian/precise-wikimedia' into debian/trusty-wikimedia
Change 255541 merged by Hashar:
Merge branch 'debian/precise-wikimedia' into debian/trusty-wikimedia
Change 255543 had a related patch set uploaded (by Hashar):
Merge branch 'debian/precise-wikimedia' into debian/jessie-wikimedia
Change 255543 merged by Hashar:
Merge branch 'debian/precise-wikimedia' into debian/jessie-wikimedia
I have cherry picked my patch and regenerated all packages. Now pending publishing to apt.wikimedia.org T119714
T119714 got us zuul_2.1.0-60-g1cc37f7-wmf4 everywhere. So we should now be able to use hard-links.
Nodepool snapshots were not properly upgrading packages from apt and were left with the version that was in the image. Forcing an apt-get update ( https://gerrit.wikimedia.org/r/260040 ) before apt-get upgrade solved it.
That being said, I connected to nodepool instant and used the cache-dir system:
jenkins@ci-jessie-wikimedia-15088:~$ time zuul-cloner --color -v --cache-dir /srv/git --branch=production https://gerrit.wikimedia.org/r/p operations/puppet
INFO:zuul.CloneMapper:Workspace path set to: /home/jenkins
DEBUG:zuul.CloneMapper:Using operations/puppet as destination (unmatched)
INFO:zuul.CloneMapper:Mapping projects to workspace...
INFO:zuul.CloneMapper: operations/puppet -> /home/jenkins/operations/puppet
DEBUG:zuul.CloneMapper:Checking overlap in destination directories...
INFO:zuul.CloneMapper:Expansion completed.
INFO:zuul.Cloner:Preparing 1 repositories
INFO:zuul.Cloner:Creating repo operations/puppet from cache /srv/git/operations/puppet.git
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
INFO:zuul.Cloner:Updating origin remote in repo operations/puppet to https://gerrit.wikimedia.org/r/p/operations/puppet
DEBUG:zuul.Repo:Resetting repository /home/jenkins/operations/puppet
DEBUG:zuul.Repo:Updating repository /home/jenkins/operations/puppet
INFO:zuul.Cloner:upstream repo has branch production
DEBUG:zuul.Cloner:Project operations/puppet in Zuul does not have ref production
INFO:zuul.Cloner:Falling back to branch production
DEBUG:zuul.Repo:Checking out remotes/origin/production
INFO:zuul.Cloner:Prepared operations/puppet repo with branch production at commit f386e3ca05066da6b33781c24a893f3cc26d02d3
INFO:zuul.Cloner:Prepared all repositories
real 0m4.636s
user 0m1.868s
sys 0m0.564sSo that is fixed by the patch we have proposed upstream https://review.openstack.org/#/c/117626/ but already included in Wikimedia Debian package.
This doesn't seem particularly relevant anymore with the newer Docker-based quibble images and git-caches. I don't actually know whether it uses hardlinks or not, but as long as we make shallow clones and have good caches, the overhead is insignificant even if we don't use hardlinks.
On the Docker container we have the git cache mounted read-only from the host into the container as /srv/git. Since they are different partitions, git/linux would always hard copy the files. From the tests I did for Quibble, it is reasonably fast, most probably because all the mirrored repositories are already in the Linux disk cache.
So yeah +1 on declining. One less trouble some custom fix we will be able to drop in our Zuul.