Page MenuHomePhabricator

mwopenstackclients.py reliability
Open, Needs TriagePublic

Description

The modules/openstack/files/clientpackages/mwopenstackclients.py file is critical to the proper functioning of Puppetservers in CloudVPS, if the file is broken puppet runs starts timing out and the puppetserver itself it's not able to self-heal itself, requiring a manual fix.

During the work on the parent task, a patch with an error in mwopenstackclients.py was merged, causing puppet runs to start failing.
We've used cumin to fix it, and for posterity I'll paste below some of the commands used to resolve the issue.

But this task should be to improve the reliability of that script that has ~600 lines of Python code without any test and depends on this hardcoded/imported designate client with a copyright from 2015, imported into puppet in Oct. 2019 with a commit message referring Debian Stretch. It should probably be discarded in favore of an upstream-provided client.
Proper testing should be added, ideally even integration tests, and evaluate the possibility to move the file to a proper library.

Some commands used:

# To find all the puppetservers, then targeted with D{fqdn1,fqdn2,...}
sudo cumin -x 'O{*}' "grep '^server =' /etc/puppet/puppet.conf"

# To check the version of the file (run via cumin)
md5sum /usr/lib/python3/dist-packages/mwopenstackclients.py

# To fix the problem
SSH_AUTH_SOCK=/run/keyholder/proxy.sock scp good_file root@$FQDN:/path/to/bad/file

on both cloudcumin hosts to target both openstack installations.

Related Objects

Event Timeline

This is a classic case of a trivial glue script that has grown way beyond its intended use. I see two ways to move forward here:

  1. Decide that managing our own glue code for openstack access is bad, and replace all uses with direct import and use of upstream libraries. The upstream situation is certainly much better than it was when I first made this script, although I suspect it will still be a bit messy and awkward to replace everywhere it is used.
  1. In most cases the use of individual project-specific libraries (e.g. "from novaclient import client as nova_client") is now discouraged in favor of using the general-purpose openstacksdk. Openstacksdk might be advanced enough now that it can meet all our needs, in which case we're due for a big rewrite of mwopenstackclients, and we'll need tests for /that/ anyway.