Change 295357 had a related patch set uploaded (by Faidon Liambotis):
openldap: enable the memberof overlay

https://gerrit.wikimedia.org/r/295357

Change 306905 had a related patch set uploaded (by Muehlenhoff):
Ship a script to rewrite group memberships after enabling the memberof overlay

https://gerrit.wikimedia.org/r/306905

Change 306905 merged by Muehlenhoff:
Ship a script to rewrite group memberships after enabling the memberof overlay

https://gerrit.wikimedia.org/r/306905

Change 295357 merged by Muehlenhoff:
openldap: enable the memberof overlay

https://gerrit.wikimedia.org/r/295357

This is now enabled, but there seems to be a problem of the memberof overlay in combination with mirror mode/syncrepl: Changes which were made against seaborgium/serpens are only effected on the local server, but not replicated to the other endpoint.

The manpage for slapo-memberof states that the overlay should be run on both endpoints (as we do):

The memberof overlay may be used with any backend that provides full read-write functionality, but it is mainly intended for use with local storage backends. The maintenance operations it performs are internal to the server on which the overlay is configured and are never replicated. Replica servers should be configured with their own instances of the memberOf overlay if it is desired to maintain these memberOf attributes on the replicas.

The order of the configuration of the overlays also seems to be correct (syncprov is configured after memberof):

`Overlays are pushed onto a stack over the database, and so they will execute in the reverse of the order in which they were configured and the database itself will receive control last of all.
`

I did manage to get a memberOf relayed from seaborgium to serpens. The process was:

• Check that memberOf is present on seaborgium but NOT on serpens for a set of users (dzahn in this case)

akosiaris@seaborgium:~$ ldapsearch -LLL -x uid=dzahn memberof
dn: uid=dzahn,ou=people,dc=wikimedia,dc=org
memberOf: cn=packaging,ou=projects,dc=wikimedia,dc=org

akosiaris@serpens:~$ ldapsearch -LLL -x uid=dzahn memberof
dn: uid=dzahn,ou=people,dc=wikimedia,dc=org

• Construct an LDIF changetype: modify change like the following:

dn: cn=packaging,ou=projects,dc=wikimedia,dc=org
changetype: modify
replace: member
member: uid=mark,ou=people,dc=wikimedia,dc=org

that effectively deletes everyone from the group (mark there is a placeholder and was not in the original dataset). We can also delete: member but that also requires remove the groupofnames which I wanted to avoid in my tests. But in the migration script it is probably the more prudent approach.

• Wait 60secs for the change to propagate. The logs on serpens will output something like

Sep 16 08:58:33 serpens slapd[29881]: conn=-1 op=0: memberof_value_modify DN="uid=dzahn,ou=people,dc=wikimedia,dc=org" delete memberOf="cn=packaging,ou=projects,dc=wikimedia,dc=org" failed err=16

Effectively pointing out that the object does NOT have a memberof attribute, and so it can not delete it. We did know that already of course.

• Construct a change using the original LDIF like the following

dn: cn=packaging,ou=projects,dc=wikimedia,dc=org
changetype: modify
replace: member
member: uid=ben,ou=people,dc=wikimedia,dc=org
member: uid=diederik,ou=people,dc=wikimedia,dc=org
member: uid=novaadmin,ou=people,dc=wikimedia,dc=org
member: uid=tstarling,ou=people,dc=wikimedia,dc=org
member: uid=laner,ou=people,dc=wikimedia,dc=org
member: uid=dzahn,ou=people,dc=wikimedia,dc=org
member: uid=yuvipanda,ou=people,dc=wikimedia,dc=org
member: uid=andrew,ou=people,dc=wikimedia,dc=org
member: uid=matanya,ou=people,dc=wikimedia,dc=org
member: uid=akosiaris,ou=people,dc=wikimedia,dc=org
member: uid=ori,ou=people,dc=wikimedia,dc=org
member: uid=marc,ou=people,dc=wikimedia,dc=org

That just basically readds everyone to the group.

• Wait 60 secs and check

and voila

ldapsearch -LLL -x uid=dzahn memberof
dn: uid=dzahn,ou=people,dc=wikimedia,dc=org
memberOf: cn=packaging,ou=projects,dc=wikimedia,dc=org

That 60 secs wait seems to be crucial. If I do the removal/add in one go, nothing happens. Which I can not yet explain (but will try to).

That poses an interesting problem for everyday use of course, but I am starting to think it will be only corner cases.

@MoritzMuehlenhoff Should I write a script that does the above for every group ?

Interesting! But it that limited to rewriting existing groups? Or does it also affect new memberOf attributes being added as a consequence of new group additions/changes. Right now seaborgium has 6280 memberOf attributes and 196 only on serpens, so this could likely also affect the day-to-day group changes.

Wrt scripting, the existing https://gerrit.wikimedia.org/r/#/c/306905/ should have this almost, only needs a time.sleep(60) and a tweak to empty_group to set a dummy user instead of emptying it.

In T142817#2643111, @MoritzMuehlenhoff wrote:

Interesting! But it that limited to rewriting existing groups? Or does it also affect new memberOf attributes being added as a consequence of new group additions/changes. Right now seaborgium has 6280 memberOf attributes and 196 only on serpens, so this could likely also affect the day-to-day group changes.

I 've just added a project called testproject via wikitech and waited it out and

akosiaris@seaborgium:~$ ldapsearch -LLL -x uid=akosiaris memberof
dn: uid=akosiaris,ou=people,dc=wikimedia,dc=org
memberOf: cn=project-toolsbeta,ou=groups,dc=wikimedia,dc=org
memberOf: cn=project-netdata,ou=groups,dc=wikimedia,dc=org
memberOf: cn=project-tools,ou=groups,dc=wikimedia,dc=org
memberOf: cn=project-bastion,ou=groups,dc=wikimedia,dc=org
memberOf: cn=packaging,ou=projects,dc=wikimedia,dc=org
memberOf: cn=project-testproject,ou=groups,dc=wikimedia,dc=org

akosiaris@serpens:~$ ldapsearch -LLL -x uid=akosiaris memberof
dn: uid=akosiaris,ou=people,dc=wikimedia,dc=org
memberOf: cn=project-toolsbeta,ou=groups,dc=wikimedia,dc=org
memberOf: cn=project-tools,ou=groups,dc=wikimedia,dc=org
memberOf: cn=project-bastion,ou=groups,dc=wikimedia,dc=org
memberOf: cn=project-netdata,ou=groups,dc=wikimedia,dc=org
memberOf: cn=packaging,ou=projects,dc=wikimedia,dc=org
memberOf: cn=project-testproject,ou=groups,dc=wikimedia,dc=org

I also added Giuseppe to project netdata and everything worked fine as well.

So it seems like if we have problems in day-to-day operations cause of this, it will probably be some edge-cases with race conditions.

Wrt scripting, the existing https://gerrit.wikimedia.org/r/#/c/306905/ should have this almost, only needs a time.sleep(60) and a tweak to empty_group to set a dummy user instead of emptying it.

Great!

Ok, I'll tweak my earlier script and we can convert the core groups next week. And let's take this feature with a grain of salt and compare memberOf attributes from time to time until we're reasonably confident that standard replication works fine.

I checked how changes to the nda group have trickled in and for more than half of the members the change only affected the slapd database on seaborgium: https://phabricator.wikimedia.org/P4452

I'll ask for clarification on openldap-technical, maybe we're missing something in our configuration.

@MoritzMuehlenhoff, any news from openldap-technical or in general about this?

No yet, no.

I've revisited this and after some digging it's apparent that maintaining the memberOf attribute of all newly added groups is in fact working fine. I've also fixed my rewrite script to sleep for 65 seconds and will use that to update all privileged LDAP groups.

All privileged groups have been updated with the exception of cn=wmf and cn=ops, I will these during an upcoming early morning to minimise disruption.