Zuul is deployed on contint1001 using a Debian package intended for Jessie. The whole process is unnecessarily complicated and predate a lot of improvement we have accomplished over the last few years with regard to deploying python application.
We need to deploy it using Scap and using wheels to satisfy the requirements.
A potentially helpful overview: https://docs.google.com/drawings/d/1WTnZGO_2WS8OkWUVAVwpd9eg3nXKWVQvDPoUyoPTC1s/edit
Legacy deployment
The repository is https://gerrit.wikimedia.org/g/integration/zuul Our code is a fork of upstream ~ version 2.5 and comes with hotfixes and backports and has Debian packaging information.
Since some of the required python modules are either missing from Jessie or to new, we have the Debian package to fetch dependencies from pypi using dh_virtualenv. Hence the package building requires network access. Zuul is then running from a virtualenv. Some dependencies are fullfilled using the regulard debian/control Depends: field and are thus installed by apt when installing the package.
The branches that matters:
| patch-queue/debian/jessie-wikimedia | Fork of upstream + code hotfixes and HEAD of the repository |
| debian/jessie-wikimedia | Has the ./debian directory for packaging |
The process is roughly:
- patches are pilled up in patch-queue/debian/jessie-wikimedia
- Using git buildpackage patch manager, the patches are exported in as a Debian patch serie into the branch debian/jessie-wikimedia. That is done using gbp pq export
- The package is build using DIST=jessie-wikimedia gbp buildpackage which requires pbuilder hooks installed from operations/puppet.git modules/package_builder and a cowbuilder environment to be setup on the build host.
- pbuilder install the build dependencies then invoke dh_virtualenv reusing system packages and fetch the list in requirements.txt
The outcome is a virtualenv deployed under /usr/share/python/zuul/.
Checking on contint1001, the embedded python modules are:
$ /usr/share/python/zuul/bin/pip list --local APScheduler (3.0.6) futures (3.0.5) gear (0.7.0) gitdb2 (2.0.5) GitPython (2.1.11) lockfile (0.12.2) pip (1.5.6) python-daemon (2.0.6) setuptools (5.5.1) smmap2 (2.0.5) statsd (2.1.2) tzlocal (1.2.2) zuul (2.5.1-wmf11)
debian/control has a list of the requirements. Some would fail on Stretch/Buster such as python-voluptuous which would be to new for our version of Zuul. I guess we will have to proceed by trial and errors to find the proper set of requirements.
New deployment
@Paladox has made an attempt to convert to use wheels. It might not match the systems we have put in place for other software. A few repositories worth looking at for inspiration:
- https://gerrit.wikimedia.org/g/operations/docker-images/docker-pkg/deploy which uses Docker containers to build the wheels. Ensuring they match the target distribution (for example Buster).
- https://gerrit.wikimedia.org/g/operations/software/debmonitor/deploy which has a set of script to generate the wheels.
I guess people from SRE can be involved to help start on the proper rails.
Scap target and services
The Zuul Debian package ships two services which are configured via puppet.git.
zuul is the main daemon, it is only active on contint1001. The instance on contint2001 is masked via systemd.
zuul-merger they process git merge commit of the patches against the target branch. There is one running on each of contint1001 and contint2001.
The Zuul daemon hosts a tiny web server to expose it's internal status as a json file. That is exposed publicly as integration.wikimedia.org/zuul/status.json and pass through an Apache proxy forwarding requests made for https://integration.wikimedia.org/zuul/ .
Migration
Once the Debian package is build and a Buster server is available, the new version can be installed there. Then:
- Change the Apache proxy for integration.wikimedia.org to have requests to zuul to be send to that new server instead of the locally running zuul.
- Stop the old zuul
- Bring back the new one
- have the Jenkins Gearman client to be pointed to that new server
And that might do it.