= Overview of how it would work =
== AssumptionsScap won't rely anymore on a debian package to install its Python dependencies. Instead, it will use a Python virtual environment populated using `pip`. This venv will act as a self-contained scap that can then be rsync'd to targets.
For external dependencies/system configuration, scap will rely on Puppet.
== Deploy workflow ==
* TherOnce a new release is a user named `scap` that existscreated (i.e. a new tag has been created), on all scap targets.deploy server:
* Members of the release engineering team can `ssh scap@<target>` from deploy1002.* On both masters:
* A relenger has tagged a commit in * cd /srv/deployment/scap for a new release.
== Expected operation ==
On deploy server: (which is a checkout of the scap git repo)
* cd /srv/deployment/scap (which is a * git checkout of the scap git repo)<new tag>
* git checkout <some tag> * python3 -m venv /home/scap/scap
* bin/scap install-scap * /home/scap/scap/bin/pip install --upgrade /srv/deployment/scap
* `scap install-world` (/usr/bin/scap -> /home/scap/scap/bin/scap)
* `scap install-scapworld` then:
* runs some query or reads a file to collect a list of hosts that should have scap installed
* rsync's/srv/deployment /home/scap/scap/ to scap@target:/home/scap/scap/ for each hosttarget host (masters excluded).
== Prereqs ==
For this to work there must be a stub scap deb package preinstalled on all targets which contains a /usr/bin/scap which is a symlink to /home/scap/scap/bin/scap. This package would exclude the stuff currently in /usr/lib/python3/dist-packages/scap.
[] Find a way to query the list of scap targets. See https://phabricator.wikimedia.org/T302919#7748986 for a recent list. I assume that cumin is involved. Whatever we use needs to be accessible from deployment.eqiad.wmnet.Puppet will configure the deploy servers to provide the following:
[] Need a user (maybe named `scap`, and assumed hereafter in this text) that can be ssh'd into on each of those hosts (similar to how deployers can ssh as mwdeploy to mediawiki targets during scap sync operations.. how is that set up? keyholder is used).* Checkout of the scap git repo at /srv/deployment/scap
[] Change scap's deb to just include the * A symlink /usr/bin/scap -> /home/scap/scap/bin/scap symlink.
[] New dependency: rsync* A way to query the list of scap targets. See https://phabricator.wikimedia.org/T302919#7748986 for a recent list. I assume that cumin is involved. Whatever we use needs to be accessible from deployment.eqiad.wmnet (should be addressed by https://gerrit.wikimedia.org/r/c/operations/puppet/+/771441)
[] What's the version number of this package* A user (maybe named `scap`, and assumed hereafter in this text) that can be ssh'd into on each of those hosts (similar to how deployers can ssh as mwdeploy to mediawiki targets during scap sync operations.. how is that set up? It needs to be something that won't be confusing.keyholder is used)
[] Looks like /usr/share/bash-completion/completions/scap should a few other bits should be included as well. Those could be symlinks too.
== Bootstrapping the deploy server ==* The following dependencies:
Use puppet to
* ensure that the scap stub deb is installed. * git
* rsync
* ensure that deploy-servers:/srv/deployment/scap is a clone of the scap git repo. Once checked out * bash-completion
* python3
* python3-venv
The scap user, the symlink to the venv and some of the deps need to be provisioned on the scap targets too, puppet should leave it alone.not just the deploy servers
== Transition plan ==
[]# Run `scap install-scap`world` to prime the targets. Any uses of scap on these targets will continue to use the code from the current scap deb.
[] Merge a commit that makes the scap deb be the stubbed form (just symlinks, as described above). Choose a version number for this deb. The version number selected must make clear that it is unrelated to any particular version of scap.# Apply all the Puppet changes except the /usr/bin/scap -> /home/scap/scap/bin/scap symlink
[] Follow the usual scap release procedure to tag and request# Thoroughly test the new scap deployment by serviceops.y process
[] When serviceops has completed deployment of the stub, /usr/bin/scap on targets will now point to the code previously installed by `scap install-scap`# Apply the Puppet change that creates the symlink
# Uninstall scap deb package
== Outstanding notes/problems==
[] What about beta ?
[] Installing scap somewhere other than /usr/lib/python3/dist-packages will break scripts that expect to be able to `import scap` (such as `stage-train` in the releases repo). Hopefully allowing scap to deploy itself will reduce the need/desire to extend scap outside of scap.
[] What about freshly-provisioned hosts? Running scap on them will fail until someone runs `scap install-scap` on the deploy server.
[] Scap has some python module dependencies such as jinja2, pyaml, requests, and prettytable. They will need to be handledworld` on the deploy server.