This is blocking Trusty deprecation in HW because:
* we need to decommission {T218021} which contains openstack databases we need to save
* I think we can reallocate those databases to `cloudcontrol2001-dev.wikimedia.org`, but for that we need to properly image the server first {T219626}
* we can't properly image the server in mitaka/stretch because the issue described below:
we have all over the roles for HW servers things like `include profile::openstack::$deployment::clientpackages` (or require instead of include).
That profile installs a bunch of python openstack packages (in `openstack::clientpackages::common` at `modules/openstack/manifests/clientpackages/common.pp`, client libs like:
'python-novaclient',
'python-glanceclient',
'python-keystoneclient',
'python-openstackclient',
'python-designateclient',
'python-neutronclient',
when this profile is added, these packages are installed from whatever repo is configured in the system, because `openstack::clientpackages::common` doesn't include any specific repo/pinning for any specific openstack/debian combo.
At a later point in the puppet catalog evaluation, we try to install openstack server components (keystone, glance, nova, neutron, etc) by means of each specific openstack server profile. And for them we do have proper repo/pinning configuration, by means of `openstack::serverpackages::mitaka::stretch` and friends.
But since the first python libs are already installed, this causes apt to conflict in the dependencies, or try to downgrade packages, which can cause either:
* wrong openstack server packages are installed
* apt failure causing puppet agent failure
Both cases are wrong and need to be fixed.
If stuff wasn't complex enough, it turns out that `openstack::clientpackages::common` is used * inside* CloudVPS (i.e, VM instances), so modifying that is really painful.
I already tried in https://gerrit.wikimedia.org/r/c/operations/puppet/+/500797 (already reverted) which resulted in the openstack-mitaka-jessie repo component being added to toolforge, creating apt resolver issues because packages in that component have higher pinning and again apt tries to downgrade stuff. The openstack-mitaka-jessie component is meant for HW servers only, Toolforge dependencies were not considered when I created that component.
I don't know yet how to fix this entanglement, but opened this phab task to brain dump.
Proposal 1:
* let's use `clientpackages` stuff **only** for Cloud VPS instances. Try to clean servers from usage of the profile/class tree
* if a script in server side requires any specific libs not installed by dependencies of server packages, let's create an explicit Package[whatever] resource for them (in the ::serverpackages namespace)