cumin 3.0.1-1 is broken on labs master
Closed, ResolvedPublic

Description

On`integration-cumin.integration.eqiad.wmflabs`:

$ sudo cumin --force 'name:docker' 'docker ps -a'
Caught InvalidQueryError exception: Default backend 'openstack' is not registered: {
'F': Backend(keyword='F', name='cumin_file_backend', cls=<class 'cumin_file_backend.HostsFileQuery'>),
'D': Backend(keyword='D', name='direct', cls=<class 'cumin.backends.direct.DirectQuery'>),
'P': Backend(keyword='P', name='puppetdb', cls=<class 'cumin.backends.puppetdb.PuppetDBQuery'>),
'K': Backend(keyword='K', name='knownhosts', cls=<class 'cumin.backends.knownhosts.KnownHostsQuery'>)}

/etc/cumin/config.yaml has:

default_backend: openstack
plugins:
    backends:
        - cumin_file_backend

Via strace it does open openstack.py but that is apparently not enabled in the configuration :]


The original issue below has been fixed by https://gerrit.wikimedia.org/r/#/c/419131/

$ sudo cumin --force 'name:docker' 'docker ps -a'
Caught CuminError exception: Unable to import backend 'cumin_file_backend': No module named 'cumin_file_backend'

From my bash history, it was still working on 2018-02-14 09:41:00.

Cumin got upgraded via unattended-upgrade:

Start-Date: 2018-02-20  06:51:07
Commandline: /usr/bin/unattended-upgrade
Install: ... 
Upgrade: ..., cumin:amd64 (2.0.0-1, 3.0.1-1)

The issue is the file backend (introduced by 88c43843c3d886b90b5756ad7fb6a7ee4d84da7e for T185967) is installed /usr/local/lib/python2.7/dist-packages/cumin_file_backend.py . Cumin now uses python3 and thus it is unable to lookup the file :-(

May I suggest that cumin_file_backend to be included in cumin itself instead of being provided by puppet? :]

hashar created this task.Feb 23 2018, 4:53 PM
Restricted Application added a subscriber: Aklapper. · View Herald TranscriptFeb 23 2018, 4:53 PM

This will be fixed by https://gerrit.wikimedia.org/r/c/412894/ , that is pending the full release of Cumin 3.0.1 in prod that is waiting the full release of conftool 1.0.0 in prod, that is pending final testing and also an issue with the python3-etcd debian package.

Having unattended upgrades to upgrade also internal packages of internal softwares that are strictly coupled with our Puppet configuration seems a broken design to me.

If you want to be unblocked feel free to cherry-pick and merge those two files from the CR linked above (or I'll probably do it on Monday if I have time):

modules/profile/manifests/openstack/main/cumin/master.pp
modules/profile/files/openstack/main/cumin/cumin_file_backend.py

May I suggest that cumin_file_backend to be included in cumin itself instead of being provided by puppet? :]

No, the file backend will not be part of Cumin, it's a (temporary) hack that exists only for WMCS Puppetmasters because they need to target a set of hosts that cannot be generated neither with the OpenStack backend nor the Puppet browser (for which I offered to make a backend).
A file backend is against Cumin core principle to have backends that provide dynamic host lists.

Mentioned in SAL (#wikimedia-releng) [2018-03-13T08:09:28Z] <hashar> integration: cherry picking https://gerrit.wikimedia.org/r/c/412894/ to fix cumin 3.0.1 | T188112

Mentioned in SAL (#wikimedia-releng) [2018-03-13T08:13:40Z] <hashar> integration-cumin: ln -s /usr/local/lib/python3.4 /usr/local/lib/python3 (HACK) | T188112

I've split the WMCS part into a separate CR that can be merged independently of production: https://gerrit.wikimedia.org/r/c/419131

Change 419131 had a related patch set uploaded (by Hashar; owner: Volans):
[operations/puppet@production] Cumin masters in WMCS: upgrade to python3

https://gerrit.wikimedia.org/r/419131

hashar updated the task description. (Show Details)Mar 13 2018, 10:00 AM
hashar added a comment.Apr 7 2018, 7:50 AM

Left over note from March 13th

Caught InvalidQueryError exception: Default backend 'openstack' is not registered

Theoretically addressed by installing the cumin package with --install-suggests ( https://gerrit.wikimedia.org/r/#/c/419131/2..3/modules/profile/manifests/openstack/main/cumin/master.pp ). Though somehow suggests do not work :(

In theory the message can be solved by MANUALLY installing the packages:

python3-keystoneauth1 python3-keystoneclient python3-novaclient
Volans added a comment.Apr 9 2018, 8:29 AM

Patch updated to overcome this problem, once reviewed and merged it should solve the issue.

https://gerrit.wikimedia.org/r/#/c/operations/puppet/+/419131/ fixed it on integration-cumin.integration.eqiad.wmflabs !

Change 419131 merged by Volans:
[operations/puppet@production] Cumin masters in WMCS: upgrade to python3

https://gerrit.wikimedia.org/r/419131

Change 446757 had a related patch set uploaded (by Volans; owner: Volans):
[operations/puppet@production] cumin: pin also python3-urllib3 for WMCS

https://gerrit.wikimedia.org/r/446757

Change 446757 merged by Volans:
[operations/puppet@production] cumin: pin also python3-urllib3 for WMCS

https://gerrit.wikimedia.org/r/446757

hashar assigned this task to Volans.Jul 19 2018, 9:15 AM
hashar closed this task as Resolved.

Works fine now and all patches have been merged. Thank you!

Krenair added a subscriber: Krenair.EditedJul 28 2018, 3:07 PM

Broken again in T200608, with a related unattended-upgrade occurring not 24 hours after this was closed?