On cloudcontrol2001-dev, keystone is sometimes running and sometimes not... it seems like a coin-toss. Explicitly starting it with 'service keystone start' generally does not result in it running.
It's also the case that puppet never settles down into a stable state on that system. I suspect these facts are related.
I've confirmed that puppet runs are stable on cloudcontrol1003 which has a similar catalog but for Jessie.
| Status | Subtype | Assigned | Task | ||
|---|---|---|---|---|---|
| Resolved | Andrew | T237749 Upgrade wmcs OpenStack version to Ocata | |||
| Resolved | Andrew | T210715 cloudvps: PDNS 3.x vs 4.x | |||
| Open | None | T132225 Add SSHFP dns records to bastions | |||
| Resolved | MoritzMuehlenhoff | T224549 Track remaining jessie systems in production | |||
| Open | None | T224708 Drop most of mwopenstackclients.DnsManager in favour of designateclient | |||
| Resolved | aborrero | T212302 CloudVPS: upgrade: jessie -> stretch & mitaka -> newton | |||
| Resolved | aborrero | T215407 cloudvps: cloudcontrol support for mitaka/stretch | |||
| Resolved | aborrero | T223832 puppet vs. stretch vs. keystone |
This seems to be some kind of conflict in the packages? between python-ldap and keystone. I will investigate.
I originally tried to solve this with this patch https://gerrit.wikimedia.org/r/c/operations/puppet/+/500014 because python-pyldap was a fork of python-ldap.
It turns out that python-ldap is installed also by:
class ldap::client::utils($ldapconfig) {
require_package('python-ldap', 'python-pycurl')aborrero@cumin1001:~ $ sudo cumin "P{R:Class = ldap::client::utils}"
15 hosts will be targeted:
cloudcontrol[2001,2003]-dev.wikimedia.org,cloudcontrol[1003-1004].wikimedia.org,cloudservices[1003-1004].wikimedia.org,cloudstore[1008-1009].wikimedia.org,labstore[1003-1005].eqiad.wmnet,labweb[1001-1002].wikimedia.org,mwmaint2001.codfw.wmnet,mwmaint1002.eqiad.wmnet
DRY-RUN mode enabled, abortingSince python-ldap can be replaced by python-pyldap my proposal is to simply drop python-ldap from ldap::client::utils.
But wait! this seems to be a bit more complex:
14:30 <arturo> hey zigo, can you share some hints on `python-pyldap` vs `python-ldap`? 14:30 <zigo> arturo: I can't remember the exact details, but basically, the fork got remerged. 14:30 <zigo> I can't remember what the direction is, sorry ... 14:30 <arturo> -_- 14:30 <zigo> But the one with version 3 has the re-merge.
So it seems we have another super special snowflake here.
I should investigate now how to deal with this combo... CC'ing @jbond and @MoritzMuehlenhoff just in case.
Change 511413 had a related patch set uploaded (by Arturo Borrero Gonzalez; owner: Arturo Borrero Gonzalez):
[operations/puppet@production] ldap::client::utils: don't require python-ldap if already declared
Change 511413 merged by Andrew Bogott:
[operations/puppet@production] ldap::client::utils: don't require python-ldap if already declared
Change 511460 had a related patch set uploaded (by Andrew Bogott; owner: Andrew Bogott):
[operations/puppet@production] keystone: set logdir group to 'keystone'
Change 511460 merged by Andrew Bogott:
[operations/puppet@production] keystone: set logdir group to 'keystone'