Overview of how it would work
Scap won't rely anymore on a debian package to install its Python dependencies. Instead, it will use a Python virtual environment populated using pip. This venv will act as a self-contained scap that can then be rsync'd to targets.
For external dependencies/system configuration, scap will rely on Puppet.
Deploy workflow
Once a new release is created (i.e. a new tag has been created), on deploy server:
- On both masters:
- cd /srv/deployment/scap (which is a checkout of the scap git repo)
- git checkout <new tag>
- python3 -m venv /home/scap/scap
- /home/scap/scap/bin/pip install --upgrade /srv/deployment/scap
- scap install-world (note that /usr/bin/scap -> /var/lib/scap/scap/bin/scap), which will:
- Reads a file to collect a list of hosts that should have scap installed
- rsync's ~scap/scap to scap@target:scap/ for each target listed in /etc/dsh/group/scap_targets (masters excluded).
Prereqs
Puppet will configure the deploy servers to provide the following:
- Checkout of the scap git repo at /srv/deployment/scap
- A symlink /usr/bin/scap -> /var/lib/scap/scap/bin/scap
- A way to query the list of scap targets. See https://phabricator.wikimedia.org/T302919#7748986 for a recent list. I assume that cumin is involved. Whatever we use needs to be accessible from deployment.eqiad.wmnet (should be addressed by https://gerrit.wikimedia.org/r/c/operations/puppet/+/771441)
- A user (maybe named scap, and assumed hereafter in this text) that can be ssh'd into on each of those hosts (similar to how deployers can ssh as mwdeploy to mediawiki targets during scap sync operations.. how is that set up? keyholder is used)
- The following dependencies:
- git
- rsync
- bash-completion
- python3
- python3-venv
The scap user, the symlink to the venv and some of the deps need to be provisioned on the scap targets too, not just the deploy servers. The targets also need to be able to access /var/lib/scap/scap on the deploy servers via rsync.
Transition plan
- Run scap install-world to prime the targets. Any uses of scap on these targets will continue to use the code from the current scap deb.
- Apply all the Puppet changes except the /usr/bin/scap -> /var/lib/scap/scap/bin/scap symlink
- Thoroughly test the new scap deploy process
- Apply the Puppet change that creates the symlink
- Uninstall scap deb package
Outstanding notes/problems
- What about beta ? -> Puppet config and scap beta scripts will be updated to also use the new self-installation mechanism
- Installing scap somewhere other than /usr/lib/python3/dist-packages will break scripts that expect to be able to import scap (such as stage-train in the releases repo). Hopefully allowing scap to deploy itself will reduce the need/desire to extend scap outside of scap.
- What about freshly-provisioned hosts? Running scap on them will fail until someone runs scap install-world on the deploy server. This will be resolved by establishing Puppet configuration to make scap targets self-install scap via rsync if it is not installed already (https://gerrit.wikimedia.org/r/c/operations/puppet/+/806397)