Page MenuHomePhabricator

`cn=tools.mzmcbride,ou=servicegroups,dc=wikimedia,dc=org` record stuck on ldap-ro.eqiad.wikimedia.org
Closed, ResolvedPublicBUG REPORT

Description

I updated P35381 to exclude tools that are currently disabled pending deletion (https://disabled-tools.toolforge.org/). I have now disabled all of the legitimate tools reported by the script with one exception. There is a dangling cn=tools.mzmcbride,ou=servicegroups,dc=wikimedia,dc=org record that the script will continue to report until somebody manually removes it from the LDAP directory.

It turns out that the cn=tools.mzmcbride,ou=servicegroups,dc=wikimedia,dc=org record is not present on the primary LDAP server:

$ ldapsearch -xLLL -b cn=tools.mzmcbride,ou=servicegroups,dc=wikimedia,dc=org -H ldap://ldap-rw.eqiad.wikimedia.org:389
No such object (32)
Matched DN: ou=servicegroups,dc=wikimedia,dc=org

But it is present on the replica used in Cloud VPS:

$ ldapsearch -xLLL -b cn=tools.mzmcbride,ou=servicegroups,dc=wikimedia,dc=org -H ldap://ldap-ro.eqiad.wikimedia.org:389
dn: cn=tools.mzmcbride,ou=servicegroups,dc=wikimedia,dc=org
objectClass: groupOfNames
objectClass: posixGroup
objectClass: top
member: uid=mzmcbride,ou=people,dc=wikimedia,dc=org
cn: tools.mzmcbride
gidNumber: 51334

Attempting to delete the record directly from ldap-ro.eqiad.wikimedia.org fails (as hoped honestly):

delete member:
        uid=mzmcbride,ou=people,dc=wikimedia,dc=org
modifying entry "cn=tools.mzmcbride,ou=servicegroups,dc=wikimedia,dc=org"
ldap_modify: Server is unwilling to perform (53)
        additional info: operation restricted

Can some local LDAP wizard like @MoritzMuehlenhoff or @akosiaris help figure out how to re-sync ldap-ro.eqiad.wikimedia.org with ldap-rw.eqiad.wikimedia.org?

Event Timeline

bd808 renamed this task from `cn=tools.mzmcbride,ou=servicegroups,dc=wikimedia,dc=org` not on to `cn=tools.mzmcbride,ou=servicegroups,dc=wikimedia,dc=org` record stuck on ldap-ro.eqiad.wikimedia.org.Jan 12 2024, 12:54 AM
akosiaris claimed this task.

For some reason, which I can't know since apparently this entry was deleted back in November 2022 (per the audit log[1]) and system logs are not longer around, the replication had apparently failed on the 2 hosts powering ldap-ro.eqiad.

I re-added the entry on ldap-rw, waited 2 mins for it to propagate fully, and then re-deleted and ...

$ ldapsearch -xLLL -b cn=tools.mzmcbride,ou=servicegroups,dc=wikimedia,dc=org
No such object (32)

Hopefully this was a one off, but if it isn't, the same process should work.

[1]

# delete 1668970896 dc=wikimedia,dc=org uid=novaadmin,ou=people,dc=wikimedia,dc=org IP=10.64.37.19:53222 conn=36352
dn: cn=tools.mzmcbride,ou=servicegroups,dc=wikimedia,dc=org
changetype: delete
# end delete 1668970896