Page MenuHomePhabricator
Create Task
Maniphest T340047

LDAP problems following cloudservices2005-dev reimage
Closed, ResolvedPublic
Actions

Description

I'm observing keystone is having problems connecting to LDAP @ cloudservices2004-dev following latest cloudservices2005-dev reimage works.

ERROR ldappool ldap.INVALID_CREDENTIALS: {'desc': 'Invalid credentials'}

I've turned on debug mode on the server:

Jun 21 15:05:58 cloudservices2004-dev slapd[2867781]: conn=1026 fd=40 ACCEPT from IP=172.20.5.7:37510 (IP=0.0.0.0:389)
[..]
Jun 21 15:05:58 cloudservices2004-dev slapd[2867781]: conn=1026 op=0 BIND dn="uid=novaobserver,ou=people,dc=wikimedia,dc=org" method=128
Jun 21 15:05:58 cloudservices2004-dev slapd[2867781]: conn=1026 op=0 RESULT tag=97 err=49 text=

error 49 is indeed auth failure.

Details

SubjectRepoBranchLines +/-
codfw1dev: services: override cloudcontrol FQDNoperations/puppetproduction+9 -0
openldap: main-acls.erb: support keystone hosts without AAAAoperations/puppetproduction+24 -7
Customize query in gerrit

Related Objects
Search...

StatusSubtypeAssignedTask
 ResolvedaborreroT296411 cloud: decide on general idea for having cloud-dedicated hardware provide service in the cloud realm & the internet
 ResolvedaborreroT297596 have cloud hardware servers in the cloud realm using a dedicated LB layer
 ResolvedaborreroT324992 cloudlb: create PoC on codfw
 ResolvedaborreroT327908 rename cloudgw2001-dev into cloudlb2001-dev
 ResolvedayounsiT327930 Is Vlan 2122 cloud-support1-b-codfw required?
 ResolvedcmooneyT327919 Configure cloudsw1-b1-codfw and migrate cloud hosts in codfw B1 to it
 ResolvedPapaulT331470 Run 2x1G links from asw-b1-codfw to cloudsw1-b1-codfw
Restricted Task
 ResolvedcmooneyT333316 Homer unable to commit config to cloudsw1-b1-codfw (QFX5120 21.4R3.16)
 ResolvedcmooneyT336368 cloudgw: review security policy for edge network
 ResolvedaborreroT332153 cloudlb PoC: prepare backends
 ResolvedaborreroT336963 cloudcontrol2001-dev can't reach cloud-vps public IPs
 ResolvedaborreroT335759 cloud-private subnet: introduce new domain
 ResolvedtaaviT336566 cloud: failures resolving some `wikimedia.cloud` domains
 OpencmooneyT346428 Netbox: Add support for our complex host network setups in provision script
 ResolvedaborreroT346410 need private IPs for cloudvirt200[4-6]-dev.codfw.wmnet
 OpencmooneyT347375 Netbox device location information not available on the first Puppet run of a device
 ResolvedaborreroT335760 Modify Bird module to allow source IP to be passed to template
 ResolvedaborreroT336071 cloudlb: figure out routing
 ResolvedaborreroT337758 puppet: interface::route resource does not persists settings
 ResolvedaborreroT338936 cloudlb: figure out plans for eqiad1
 ResolvedaborreroT338937 cloudlb: review swift/radosgw status
ResolvedAndrewT341380 Open swift port (28080) to the public internet
 ResolvedAndrewT341484 Horizon object store UI doesn't allow uploading of files
 ResolvedAndrewT341640 Move Horizon deploy to a Docker container
 OpenAndrewT341509 radosgw+keystone chokes on projects with '-' in their id
 OpenAndrewT343158 Support OpenStack projects where project name != project id
 ResolvedAndrewT347927 Better swift error messages for projects with invalid chars
 ResolvedaborreroT340536 cloud-private: figure out, implement and test cross-DC traffic
 OpenNoneT317177 [tracking] Don't keep on the public vlans hosts that don't require it
 OpenNoneT340446 Cloud VPS Designate setup improvements
 ResolvedaborreroT307357 Move cloud vps ns-recursor IPs to host/row-independent addressing
 ResolvedcmooneyT336587 cloudservices[2004/2005]-dev & cloudweb2002-dev: connect them to cloudsw so they can have cloud-private vlan
 ResolvedaborreroT338779 cloudservices2005-dev: reimage into new network setup
 ResolvedaborreroT340047 LDAP problems following cloudservices2005-dev reimage
 OpenNoneT340127 systemctl restart keystone doesn't actually restart keystone sometimes

Event Timeline

aborrero created this task.Jun 21 2023, 3:27 PM
aborrero added a comment.Jun 21 2023, 3:40 PM

The observer password is set in hiera to lt-RiBeyokCO81bVvcX (this is a public password).

In the slapd database, this password is stored as d2aeac3d76cd790ea2ca017869b2c8394c08cb47 which:

aborrero@cloudservices2004-dev:~ $ echo e1NIQX1OWWRyUld1M2g1bDdhTlFPTXY0SXVjS2RCQXc9 | base64 -d
{SHA}NYdrRWu3h5l7aNQOMv4IucKdBAw=
aborrero@cloudservices2004-dev:~ $ sudo slappasswd -h {SHA} -s lt-RiBeyokCO81bVvcX
{SHA}NYdrRWu3h5l7aNQOMv4IucKdBAw=
gerritbot added a comment.Jun 21 2023, 3:51 PM

Change 931964 had a related patch set uploaded (by Arturo Borrero Gonzalez; author: Arturo Borrero Gonzalez):

[operations/puppet@production] codfw1dev: services: override cloudcontrol FQDN

https://gerrit.wikimedia.org/r/931964

gerritbot added a project: Patch-For-Review.Jun 21 2023, 3:51 PM
gerritbot added a comment.Jun 21 2023, 4:03 PM

Change 931968 had a related patch set uploaded (by Arturo Borrero Gonzalez; author: Arturo Borrero Gonzalez):

[operations/puppet@production] openldap: main-acls.erb: support keystone hosts without AAAA

https://gerrit.wikimedia.org/r/931968

aborrero added a comment.Jun 21 2023, 5:07 PM

The problem is LDAP IP-based ACLs.

aborrero changed the task status from Open to In Progress.Jun 21 2023, 5:07 PM
aborrero triaged this task as High priority.
aborrero moved this task from Backlog to Doing on the User-aborrero board.
gerritbot added a comment.Jun 22 2023, 9:33 AM

Change 931968 merged by Arturo Borrero Gonzalez:

[operations/puppet@production] openldap: main-acls.erb: support keystone hosts without AAAA

https://gerrit.wikimedia.org/r/931968

gerritbot added a comment.Jun 22 2023, 9:34 AM

Change 931964 merged by Arturo Borrero Gonzalez:

[operations/puppet@production] codfw1dev: services: override cloudcontrol FQDN

https://gerrit.wikimedia.org/r/931964

Maintenance_bot removed a project: Patch-For-Review.Jun 22 2023, 10:11 AM
aborrero added a comment.Jun 22 2023, 12:00 PM

The ACLs seem fixed, now getting:

ERROR ldappool ldap.SERVER_DOWN: {'desc': "Can't contact LDAP server", 'errno': 22, 'info': 'Invalid argument'}
Andrew added a comment.EditedJun 22 2023, 3:36 PM

This is hardly a minimal unit test, but my go-to ldap exploration routine works:

root@cloudcontrol2001-dev:~# ldapvi -h ldap://cloudservices2004-dev.codfw.wmnet:389 --user uid=novaadmin,ou=people,dc=wikimedia,dc=org -b ou=groups,dc=wikimedia,dc=org

It also works with the service address:

root@cloudcontrol2001-dev:~# ldapvi -h ldap://cloudservices2004-dev.private.codfw.wikimedia.cloud:389 --user uid=novaadmin,ou=people,dc=wikimedia,dc=org -b ou=groups,dc=wikimedia,dc=org
aborrero added a comment.Jun 22 2023, 3:57 PM

After @Andrew magic touch this is now working apparently.

The suspicion is this is an instance of T340127: systemctl restart keystone doesn't actually restart keystone sometimes happening.

aborrero closed this task as Resolved.Jun 26 2023, 9:53 AM
Content licensed under Creative Commons Attribution-ShareAlike (CC BY-SA) 4.0 unless otherwise noted; code licensed under GNU General Public License (GPL) 2.0 or later and other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL