A second though, maybe the l10nupdate and mwdeploy User definitions in puppet should be given UID/GID that matches the one from LDAP.

(In reply to Antoine "hashar" Musso from comment #1)

A second though, maybe the l10nupdate and mwdeploy User definitions in
puppet should be given UID/GID that matches the one from LDAP.

Renumbering is going to be a pain, but it would less painful to ensure that the gid/uid pairs used in LDAP match the gid/uid pairs found in the production cluster. It's interesting to me that the mwdeploy user and group do not have explicit uid/gid in puppet. I wonder how that actually works in practice across production.

See also T79786: mwdeploy does not have the same user ID on all Apaches and T119165: l10nupdate user uid mismatch between tin and mira

Same happens with groups as @thcipriani found out on the beta cluster. A project-deployment-prep local user group ended up shadowing up the LDAP one....

Mentioned in SAL (#wikimedia-releng) [2019-12-13T11:08:27Z] <hashar> deployment-mediawiki07 : removing faulty entry mwdeploy:x:497:498::/var/lib/mwdeploy:/bin/bash in /etc/passwd # T73480

This likely is really an upstream production puppet issue. That said, switching non-buster VMs to sssd might help.

Mentioned in SAL (#wikimedia-releng) [2021-06-11T14:14:31Z] <hashar> deployment-imagescaler03: delete local mwdeploy user with uid 497 # T73480

Change 699427 had a related patch set uploaded (by Hashar; author: Hashar):

[operations/puppet@production] mwdeploy user is provided by LDAP on WMCS

https://gerrit.wikimedia.org/r/699427

Change 699427 abandoned by Hashar:

[operations/puppet@production] mwdeploy user is provided by LDAP on WMCS

Reason:

https://gerrit.wikimedia.org/r/699427