The modules/openstack/files/clientpackages/mwopenstackclients.py file is critical to the proper functioning of Puppetservers in CloudVPS, if the file is broken puppet runs starts timing out and the puppetserver itself it's not able to self-heal itself, requiring a manual fix.
During the work on the parent task, a patch with an error in mwopenstackclients.py was merged, causing puppet runs to start failing.
We've used cumin to fix it, and for posterity I'll paste below some of the commands used to resolve the issue.
But this task should be to improve the reliability of that script that has ~600 lines of Python code without any test and depends on this hardcoded/imported designate client with a copyright from 2015, imported into puppet in Oct. 2019 with a commit message referring Debian Stretch. It should probably be discarded in favore of an upstream-provided client.
Proper testing should be added, ideally even integration tests, and evaluate the possibility to move the file to a proper library.
Some commands used:
# To find all the puppetservers, then targeted with D{fqdn1,fqdn2,...}
sudo cumin -x 'O{*}' "grep '^server =' /etc/puppet/puppet.conf"
# To check the version of the file (run via cumin)
md5sum /usr/lib/python3/dist-packages/mwopenstackclients.py
# To fix the problem
SSH_AUTH_SOCK=/run/keyholder/proxy.sock scp good_file root@$FQDN:/path/to/bad/fileon both cloudcumin hosts to target both openstack installations.