Page MenuHomePhabricator
Create Task
Maniphest T347555

[openstack] LDAP is broken in codfw
Closed, ResolvedPublic
Actions

Description

slapd is running in cloudservices2004-dev and cloudservices2005-dev, and is configured to sync information bidirectionally between the two servers.

Looking at the slapd logs (journalctl -u slapd) it looks like synchronization was broken from before I reimaged 2004 to bookworm (reimage happened on Sep 18):

Sep 07 13:13:33 cloudservices2005-dev slapd[2400376]: slap_client_connect: URI=ldap://cloudservices2004-dev.codfw.wmnet:389 Error, ldap_start_tls failed (-11)

Disabling TLS in /etc/ldap/slapd.conf (starttls=no) results in a different error that seems to indicate invalid credentials:

Sep 28 09:22:46 cloudservices2005-dev slapd[3980547]: slap_client_connect: URI=ldap://cloudservices2004-dev.codfw.wmnet:389 DN="cn=repluser,dc=wikimedia,dc=org" ldap_sasl_bind_s failed (49)

Details

SubjectRepoBranchLines +/-
codfw1dev: disable cloudservices2004-dev LDAP serveroperations/puppetproduction+0 -1
cloudservices[2004,2005]-dev: refresh their counterpart FQDNoperations/puppetproduction+4 -3
Customize query in gerrit

Related Objects
Search...

Event Timeline

fnegri changed the task status from Open to In Progress.Sep 28 2023, 9:27 AM
fnegri created this task.
fnegri moved this task from Backlog to In progress on the cloud-services-team (FY2023/2024-Q1-Q2) board.
aborrero added a comment.Sep 28 2023, 9:43 AM

We may want to check the certificate in use, it may be missing some alt-name or similar. It should be in ops/puppet, maybe using acmechief, which doesn't make a lot of sense for that .codfw.wmnet address.

Maybe they should be contacting each other using the .private.codfw.wikimedia.cloud address.

aborrero added a comment.Sep 28 2023, 11:58 AM

In hieradata/role/common/acme_chief.yaml we have:

ldap-codfw1dev:
    CN: 'ns-recursor.openstack.codfw1dev.wikimediacloud.org'
    SNI:
    - 'ns0.openstack.codfw1dev.wikimediacloud.org'
    - 'ns1.openstack.codfw1dev.wikimediacloud.org'
    - 'cloudservices2004-dev.private.codfw.wikimedia.cloud'
    - 'cloudservices2005-dev.private.codfw.wikimedia.cloud'
    challenge: dns-01
    authorized_hosts:
    - 'cloudservices2004-dev.codfw.wmnet'
    - 'cloudservices2005-dev.codfw.wmnet'
gerritbot added a comment.Sep 28 2023, 12:04 PM

Change 961780 had a related patch set uploaded (by Arturo Borrero Gonzalez; author: Arturo Borrero Gonzalez):

[operations/puppet@production] cloudservices[2004,2005]-dev: refresh their counterpart FQDN

https://gerrit.wikimedia.org/r/961780

gerritbot added a project: Patch-For-Review.Sep 28 2023, 12:04 PM
gerritbot added a comment.Sep 28 2023, 12:12 PM

Change 961780 merged by Arturo Borrero Gonzalez:

[operations/puppet@production] cloudservices[2004,2005]-dev: refresh their counterpart FQDN

https://gerrit.wikimedia.org/r/961780

aborrero added a comment.Sep 28 2023, 12:22 PM

The patch seems to solve the TLS problem.

Now it seems we are missing something related to https://gerrit.wikimedia.org/r/c/operations/puppet/+/961066

I'll let @Andrew handle that one.

Maintenance_bot removed a project: Patch-For-Review.Sep 28 2023, 12:30 PM
fnegri triaged this task as High priority.Sep 28 2023, 1:12 PM
gerritbot added a comment.Sep 28 2023, 3:40 PM

Change 961843 had a related patch set uploaded (by Arturo Borrero Gonzalez; author: Arturo Borrero Gonzalez):

[operations/puppet@production] codfw1dev: disable cloudservices2004-dev LDAP server

https://gerrit.wikimedia.org/r/961843

gerritbot added a project: Patch-For-Review.Sep 28 2023, 3:40 PM
fnegri reassigned this task from fnegri to Andrew.Sep 28 2023, 3:52 PM
Andrew closed this task as Resolved.Sep 28 2023, 10:46 PM

This is now resolved. I rebuilt cloudservices2005-dev with bookworm and slapd seems to be working there too.

fnegri mentioned this in T345810: [openstack] Upgrade codfw hosts to bookworm.Sep 29 2023, 4:40 PM
fnegri moved this task from In progress to Done on the cloud-services-team (FY2023/2024-Q1-Q2) board.Oct 3 2023, 5:59 PM
Content licensed under Creative Commons Attribution-ShareAlike (CC BY-SA) 4.0 unless otherwise noted; code licensed under GNU General Public License (GPL) 2.0 or later and other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL