Marc-André / Andrew B. created such users previously for the l10nupdate and mwdeploy system account.

I just added users Apache with uid/gid 48/48. I'm not 100% positive this won't cause conflicts on some systems due to the unusually low uid, so please approach with caution.

Should this be closed? Or does it need more consultation?

In T76086#807844, @Cmcmahon wrote:

Should this be closed? Or does it need more consultation?

@Andrew said he created the user in ldap, but

deployment-logstash1:~
bd808$ id apache
id: apache: No such user

So something is not quite right yet. I wonder if the low uid/gid is filtered out from the ldap config on the host or something?

Confirmed -- when I move the apache user to id 607 everything works normally.

Since those low numbers are typically reserved for package users, ldap/pam apparently does not assign them. Is 607 OK?

The user and group are definitely in:

$ ldaplist -l passwd apache

dn: uid=apache,ou=people,dc=wikimedia,dc=org
	uid: apache
	objectClass: person
	objectClass: organizationalPerson
	objectClass: inetorgperson
	objectClass: ldappublickey
	objectClass: shadowaccount
	objectClass: posixaccount
	objectClass: top
	loginShell: /usr/local/bin/sillyshell
	uidNumber: 48
	gidNumber: 48
	sn: apache
	homeDirectory: /home/apache
	cn: apache

$ ldaplist -l group apache

dn: cn=apache,ou=groups,dc=wikimedia,dc=org
	objectClass: groupOfNames
	objectClass: posixGroup
	objectClass: top
	member: uid=apache,ou=people,dc=wikimedia,dc=org
	gidNumber: 48
	cn: apache

Bryan assumption about low UID being ignored is right:

nslcd.conf:nss_min_uid 499

So should we revert the LDAP change and use puppet to ensure we have apache:apache with 48:48 ? The Precise instances seems to have UID = 48 but the Trusty ones have UID = 997.

:(

In T76086#808381, @Andrew wrote:

Confirmed -- when I move the apache user to id 607 everything works normally.

Since those low numbers are typically reserved for package users, ldap/pam apparently does not assign them. Is 607 OK?

In T76086#808382, @hashar wrote:

So should we revert the LDAP change and use puppet to ensure we have apache:apache with 48:48 ? The Precise instances seems to have UID = 48 but the Trusty ones have UID = 997.

Creating the user everywhere (including the NFS server) via puppet should work, but using a different uid/gid wouldn't be the end of the work either. We'd just have to chown files on the local servers to match the new uid/gid as we scrubbed the entries from the local /etc/passwd and /etc/group files. I don't think it amounts to much more work on the beta servers to renumber and it's easier to have @Andrew make the ldap change for us than to push this sort of thing through operations/puppet.git. My vote is for the larger number in ldap.

Hmm, it's causing puppet failures already:

Error: /Stage[main]/Mediawiki::Users/Group[apache]/gid: change from 607 to 48 failed: Could not set gid on group[apache]: Execution of '/usr/sbin/groupmod -g 48 apache' returned 4: groupmod: GID '48' already exists

on deployment-videoscaler01.

In T76086#809300, @yuvipanda wrote:

Hmm, it's causing puppet failures already:

Error: /Stage[main]/Mediawiki::Users/Group[apache]/gid: change from 607 to 48 failed: Could not set gid on group[apache]: Execution of '/usr/sbin/groupmod -g 48 apache' returned 4: groupmod: GID '48' already exists

on deployment-videoscaler01.

blerg. Well if the gid is 48 in puppet we should probably make it 48 in ldap. I guess we must not be setting the uid in puppet since that drifts between the precise and trusty instances.

hooray for making gid/uid's consistent between ldap and puppet.

Guys, sorry I didn't notice this thread earlier. You can't actually add in LDAP users that are normally managed by packages -- apt-get really doesn't like this at all!

You have to add the user on the NFS servers themselves -- not in LDAP. NFS4 works with usernames (and not user IDs) so that so long as a user exists on the server with that name everything will work fine. That's not a limitation of our setup but of the debian packaging system (it refuses to use an existing user if the package defines one of the same name, even if you use a directory service).

Just so that it is clear: the actual UID does not need to be the same. You can add the user in the node /labstore100[12]\.eqiad\.wmnet/ in site.pp; though now that I'm readying labstore servers for codfw that configuration should probably be spun off into a role.

ok... I'm happy to do whatever's needed but I'm now confused. It sounds like the right solution is something like:

a) remove the apache user from ldap entirely
b) add the user to labstore* in puppet (using what uid?)
c) ...?

Ok, I've removed the apache user from ldap. And...

https://gerrit.wikimedia.org/r/#/c/177584/

Change 177584 had a related patch set uploaded (by Tim Landscheidt):
Add an 'apache' user to eqiad labstores.

https://gerrit.wikimedia.org/r/177584

Patch-For-Review

UID 48 for Apache should be correct, yep
see this for our kind of standard:

https://wikitech.wikimedia.org/wiki/UID

The associated patch was merged. Can someone confirm that this is now resolved?

Not yet

On a Trusty instance which knows apache with uid 997 I create a file:

hashar@deployment-mediawiki01:~$ id apache
uid=997(apache) gid=48(apache) groups=48(apache)
hashar@deployment-mediawiki01:~$ sudo -u apache touch /data/project/upload7/trusty-997
hashar@deployment-mediawiki01:~$ stat /data/project/upload7/trusty-997|fgrep Uid
Access: (0644/-rw-r--r--)  Uid: (  997/  apache)   Gid: (    0/    root)

On a Precise instance, the file belong to the pybal-check (997) user:

hashar@deployment-jobrunner01:~$ stat /data/project/upload7/trusty-997|fgrep Uid
Access: (0644/-rw-r--r--)  Uid: (  997/pybal-check)   Gid: (    0/    root)
hashar@deployment-jobrunner01:~$ sudo -u apache touch /data/project/upload7/trusty-997
touch: cannot touch `/data/project/upload7/trusty-997': Permission denied
hashar@deployment-jobrunner01:~$

The other way around:

Precise:

hashar@deployment-jobrunner01:~$ sudo -u apache touch /data/project/upload7/precise-48
hashar@deployment-jobrunner01:~$ stat /data/project/upload7/precise-48 |fgrep Uid
Access: (0644/-rw-r--r--)  Uid: (   48/  apache)   Gid: (    0/    root)

Trusty:

hashar@deployment-mediawiki01:~$ stat  /data/project/upload7/precise-48|fgrep Uid
Access: (0644/-rw-r--r--)  Uid: (   48/ UNKNOWN)   Gid: (    0/    root)
hashar@deployment-mediawiki01:~$ sudo -u apache touch /data/project/upload7/precise-48
touch: cannot touch ‘/data/project/upload7/precise-48’: Permission denied

It works fine for the mwdeploy user:

Trusty:

hashar@deployment-mediawiki01:~$ id mwdeploy
uid=603(mwdeploy) gid=603(mwdeploy) groups=603(mwdeploy)
hashar@deployment-mediawiki01:~$ sudo -u mwdeploy touch /data/project/upload7/trusty-mwdeploy
hashar@deployment-mediawiki01:~$ stat /data/project/upload7/trusty-mwdeploy|fgrep Uid
Access: (0644/-rw-r--r--)  Uid: (  603/mwdeploy)   Gid: (    0/    root)
hashar@deployment-mediawiki01:~$

Precise:

hashar@deployment-jobrunner01:~$ stat /data/project/upload7/trusty-mwdeploy|fgrep Uid
Access: (0644/-rw-r--r--)  Uid: (  603/mwdeploy)   Gid: (    0/    root)

It works as well when touching a file from Precise as mwdeploy and looking it up on Trusty instance.

So seems NFS does rely on the UID :-/

In T76086#834996, @hashar wrote:

Not yet

On a Trusty instance which knows apache with uid 997 I create a file:

hashar@deployment-mediawiki01:~$ id apache
uid=997(apache) gid=48(apache) groups=48(apache)
hashar@deployment-mediawiki01:~$ sudo -u apache touch /data/project/upload7/trusty-997
hashar@deployment-mediawiki01:~$ stat /data/project/upload7/trusty-997|fgrep Uid
Access: (0644/-rw-r--r--)  Uid: (  997/  apache)   Gid: (    0/    root)

On a Precise instance, the file belong to the pybal-check (997) user:

hashar@deployment-jobrunner01:~$ stat /data/project/upload7/trusty-997|fgrep Uid
Access: (0644/-rw-r--r--)  Uid: (  997/pybal-check)   Gid: (    0/    root)
hashar@deployment-jobrunner01:~$ sudo -u apache touch /data/project/upload7/trusty-997
touch: cannot touch `/data/project/upload7/trusty-997': Permission denied
hashar@deployment-jobrunner01:~$

[snip]

It works as well when touching a file from Precise as mwdeploy and looking it up on Trusty instance.

So seems NFS does rely on the UID :-/

The fact that you can touch the file now shows that the NFS server side is fixed. Now we need to renumber the user account on our local instances so that UID for apache is 48 everywhere for things to be consistent. I think the safest way to do this is to renumber manually and then introduce puppet code that ensures that the UID is 48 for new instances.

In T76086#835034, @bd808 wrote:

The fact that you can touch the file now shows that the NFS server side is fixed.

One step at a time \O/

In T76086#835034, @bd808 wrote:

Now we need to renumber the user account on our local instances so that UID for apache is 48 everywhere for things to be consistent. I think the safest way to do this is to renumber manually and then introduce puppet code that ensures that the UID is 48 for new instances.

Does anyone have any idea where the 997 uid is defined? The puppet class mediawiki::users creates apache:apache 48:48. So I am not sure why the Trusty instances ends up with a consistent 997.