Page MenuHomePhabricator

CloudVPS: keystone bugs in Queens (wmfkeystonehooks missing role_api and LDAP encoding issues)
Closed, ResolvedPublic

Description

A keystone error in our custom hook is preventing new users from being correctly set.

How to trigger the error:

aborrero@cloudcontrol1004:~ $ sudo wmcs-openstack role add --user jayme --project bastion user
[.. no output ..]

Then, in /var/log/keystone/keystone.log:

(oslo_messaging.notify.notifier): 2020-04-06 10:12:27,806 ERROR Problem ''KeystoneHooks' object has no attribute 'role_api'' attempting to send to notification system. Payload={'user': 'jayme', 'target': {'typeURI': 'service/security/account/user', 'id': 'dde439dc-367c-5408-90e6-7c5aa6242153'}, 'inherited_to_projects': False, 'id': '98b78886-c655-5015-b2b4-1565726d3663', 'typeURI': 'http://schemas.dmtf.org/cloud/audit/1.0/event', 'eventType': 'activity', 'action': 'created.role_assignment', 'observer': {'typeURI': 'service/security', 'id': '11fe8a810dde48a38dd6cbc93a2af3fb'}, 'initiator': {'typeURI': 'service/security/account/user', 'id': '40014189417c59a5b59e4e3ab338d172', 'project_id': 'admin', 'host': {'agent': 'python-keystoneclient', 'address': '208.80.154.23'}, 'user_id': 'novaadmin'}, 'outcome': 'success', 'project': 'bastion', 'role': 'f473273fac7146b3bdbf22e5d4504f95', 'eventTime': '2020-04-06T10:12:27.583786+0000'}
Traceback (most recent call last):
  File "/usr/lib/python3/dist-packages/oslo_messaging/notify/notifier.py", line 315, in do_notify
    ext.obj.notify(ctxt, msg, priority, retry or self.retry)
  File "/usr/lib/python3/dist-packages/wmfkeystonehooks/wmfkeystonehooks.py", line 370, in notify
    self._on_member_update(project_id)
  File "/usr/lib/python3/dist-packages/wmfkeystonehooks/wmfkeystonehooks.py", line 132, in _on_member_update
    assignments = self._get_current_assignments(project_id)
  File "/usr/lib/python3/dist-packages/wmfkeystonehooks/wmfkeystonehooks.py", line 115, in _get_current_assignments
    reverseroledict = dict((v, k) for k, v in self._get_role_dict().items())
  File "/usr/lib/python3/dist-packages/wmfkeystonehooks/wmfkeystonehooks.py", line 106, in _get_role_dict
    rolelist = self.role_api.list_roles()
AttributeError: 'KeystoneHooks' object has no attribute 'role_api'

Event Timeline

aborrero moved this task from Inbox to Doing on the cloud-services-team (Kanban) board.

Change 586330 had a related patch set uploaded (by Arturo Borrero Gonzalez; owner: Arturo Borrero Gonzalez):
[operations/puppet@production] openstack: queens: wmfkeystonehooks: refresh code to use provider_api

https://gerrit.wikimedia.org/r/586330

Change 586330 had a related patch set uploaded (by Arturo Borrero Gonzalez; owner: Arturo Borrero Gonzalez):
[operations/puppet@production] openstack: queens: wmfkeystonehooks: refresh code to use provider_api

https://gerrit.wikimedia.org/r/586330

This patch apparently results in another unrelated error:

(oslo_messaging.notify.notifier): 2020-04-06 10:54:20,364 ERROR Problem '('Tuple_to_LDAPMod(): expected a byte string in the list', 'groupOfNames')' attempting to send to notification system. Payload={'project': 'bastion', 'observer': {'typeURI': 'service/security', 'id': '11fe8a810dde48a38dd6cbc93a2af3fb'}, 'eventType': 'activity', 'initiator': {'host': {'agent': 'python-keystoneclient', 'address': '208.80.154.23'}, 'project_id': 'admin', 'typeURI': 'service/security/account/user', 'id': '40014189417c59a5b59e4e3ab338d172', 'user_id': 'novaadmin'}, 'eventTime': '2020-04-06T10:54:19.315487+0000', 'action': 'created.role_assignment', 'role': 'f473273fac7146b3bdbf22e5d4504f95', 'inherited_to_projects': False, 'typeURI': 'http://schemas.dmtf.org/cloud/audit/1.0/event', 'outcome': 'success', 'target': {'typeURI': 'service/security/account/user', 'id': '6538ef44-a8f5-51cc-a694-52117a332dd8'}, 'user': 'jayme', 'id': '9f13641f-40a5-5224-81cd-9afdb48ad9f7'}
Traceback (most recent call last):
  File "/usr/lib/python3/dist-packages/oslo_messaging/notify/notifier.py", line 315, in do_notify
    ext.obj.notify(ctxt, msg, priority, retry or self.retry)
  File "/usr/lib/python3/dist-packages/wmfkeystonehooks/wmfkeystonehooks.py", line 369, in notify
    self._on_member_update(project_id)
  File "/usr/lib/python3/dist-packages/wmfkeystonehooks/wmfkeystonehooks.py", line 132, in _on_member_update
    ldapgroups.sync_ldap_project_group(project_id, assignments)
  File "/usr/lib/python3/dist-packages/wmfkeystonehooks/ldapgroups.py", line 180, in sync_ldap_project_group
    ds.add_s(dn, modlist)
  File "/usr/lib/python3/dist-packages/ldap/ldapobject.py", line 428, in add_s
    return self.add_ext_s(dn,modlist,None,None)
  File "/usr/lib/python3/dist-packages/ldap/ldapobject.py", line 413, in add_ext_s
    msgid = self.add_ext(dn,modlist,serverctrls,clientctrls)
  File "/usr/lib/python3/dist-packages/ldap/ldapobject.py", line 410, in add_ext
    return self._ldap_call(self._l.add_ext,dn,modlist,RequestControlTuples(serverctrls),RequestControlTuples(clientctrls))
  File "/usr/lib/python3/dist-packages/ldap/ldapobject.py", line 313, in _ldap_call
    result = func(*args,**kwargs)
TypeError: ('Tuple_to_LDAPMod(): expected a byte string in the list', 'groupOfNames')

Change 586341 had a related patch set uploaded (by Arturo Borrero Gonzalez; owner: Arturo Borrero Gonzalez):
[operations/puppet@production] openstack: keystone: queens: fix encoding issues in our custom LDAP handler

https://gerrit.wikimedia.org/r/586341

aborrero renamed this task from CloudVPS: keystone bug wmfkeystonehooks missing role_api to CloudVPS: keystone bugs in Queens (wmfkeystonehooks missing role_api and LDAP encoding issues).Apr 6 2020, 12:17 PM

Change 586330 merged by Arturo Borrero Gonzalez:
[operations/puppet@production] openstack: queens: wmfkeystonehooks: refresh code to use provider_api

https://gerrit.wikimedia.org/r/586330

Change 586341 merged by Arturo Borrero Gonzalez:
[operations/puppet@production] openstack: keystone: fix encoding issues in our custom LDAP handler

https://gerrit.wikimedia.org/r/586341

Mentioned in SAL (#wikimedia-cloud) [2020-04-06T16:23:22Z] <arturo> restarting apache2 in cloudcontrol1003/1004 to pick up latest wmfkeystonehooks changes T249494

aborrero claimed this task.

Seems to be working now.

Change 586446 had a related patch set uploaded (by Andrew Bogott; owner: Andrew Bogott):
[operations/puppet@production] openstack: keystone: update encoding in our custom LDAP handler

https://gerrit.wikimedia.org/r/586446

Change 586446 merged by Andrew Bogott:
[operations/puppet@production] openstack: keystone: update encoding in our custom LDAP handler

https://gerrit.wikimedia.org/r/586446

Change 586449 had a related patch set uploaded (by Andrew Bogott; owner: Andrew Bogott):
[operations/puppet@production] openstack: keystone: update encoding in our custom LDAP handler

https://gerrit.wikimedia.org/r/586449

Change 586449 merged by Andrew Bogott:
[operations/puppet@production] openstack: keystone: update encoding in our custom LDAP handler

https://gerrit.wikimedia.org/r/586449

Change 586454 had a related patch set uploaded (by Andrew Bogott; owner: Andrew Bogott):
[operations/puppet@production] wmfkeystonehooks: encode member names as utf8

https://gerrit.wikimedia.org/r/586454

Change 586457 had a related patch set uploaded (by Andrew Bogott; owner: Andrew Bogott):
[operations/puppet@production] wmfkeystonehooks sudo group: encode a bunch of ldap values as utf8

https://gerrit.wikimedia.org/r/586457

Change 586454 merged by Andrew Bogott:
[operations/puppet@production] wmfkeystonehooks: encode member names as utf8

https://gerrit.wikimedia.org/r/586454

Change 586457 merged by Andrew Bogott:
[operations/puppet@production] wmfkeystonehooks sudo group: encode a bunch of ldap values as utf8

https://gerrit.wikimedia.org/r/586457

Change 586476 had a related patch set uploaded (by Andrew Bogott; owner: Andrew Bogott):
[operations/puppet@production] wmfkeystonehooks sudo group: encode yet more ldap values as utf8

https://gerrit.wikimedia.org/r/586476

Change 586476 merged by Andrew Bogott:
[operations/puppet@production] wmfkeystonehooks sudo group: encode yet more ldap values as utf8

https://gerrit.wikimedia.org/r/586476