Description
Details
| Project | Branch | Lines +/- | Subject | |
|---|---|---|---|---|
| integration/config | master | +9 -4 | nodepool: cleanup git mirror cache |
Event Timeline
Relevant script used by OpenStack for their nodepool instance to set up local git cache:
I created a diskimage-builder element wikimedia-puppet which clones operations/puppet.git on the building host as a bare repository. It is then copied into the image under /srv/git so we can use it as a mirror in instances.
Patch is https://gerrit.wikimedia.org/r/#/c/234975/
Relevant code that keep a cache on building host and inject it in the image filesystem: https://gerrit.wikimedia.org/r/#/c/234975/10/dib/elements/wikimedia-puppet/root.d/01-populate-puppet-repo,unified or in pseudo code:
git clone --bare "https://gerrit.wikimedia.org/r/p/operations/puppet.git" /srv/dib/cache/operations/puppet mkdir -p $IMAGE_ROOT_FS/srv/git/operations/puppet.git cp -a /srv/dib/cache/operations/puppet $IMAGE_ROOT_FS/srv/git/operations/puppet.git
Example usage at top of https://gerrit.wikimedia.org/r/#/c/234975/10/dib/elements/wikimedia-puppet/install.d/05-puppet,unified :
git clone /srv/git/operations/puppet.git /puppet/
We can make the code more generic to inject more repositories.
Potentially we could use diskimage-builder element source-repositories upstream doc. I haven't looked at the code though and not sure it can be abused to match our use case. This way we would just have to provide a yaml file listing what we want copy.
Upstream code: https://github.com/openstack/diskimage-builder/tree/master/elements/source-repositories or:
git clone https://github.com/openstack/diskimage-builder cd diskimage-builder/elements/source-repositories
I could not use source-repositories properly and it does a non-bare clone.
There is some code in https://gerrit.wikimedia.org/r/#/c/239365/ . Given a list of repositories, that clones them as bare repo on the building host. Then the content is copied in the image filesystem under /srv/git/ . We should be able to hint zuul-cloner about it.
Still have to get the script to maintain the cached repos. Namely:
git pack-refs --all git repack -A -d git gc --prune=all
This is more or less in progress. I think I have the base code to handle the mirroring, gotta refactor a bit the DIB manifest.
Change 239909 had a related patch set uploaded (by Hashar):
nodepool: cleanup git mirror cache
The cache is generated when building the image: via dib/elements/wikimedia/root.d/01-mirror-gerrit-repos. For now, it only mirrors:
mediawiki/core.git
operations/puppet.git
integration/config.git
dib/elements/wikimedia/install.d/51-git-mirror-ownership makes them owned by jenkins (the username defined by the devuser element and used ssh into the instance).
The script that prepare snapshot images should probably refresh the mirrors. Should be done via nodepool/scripts/setup_node.sh. Currently when refreshing the snapshot we clone from the mirror then git pull from Gerrit.