Page MenuHomePhabricator

Create labs ldap user and group "apache" to be used for NFS4 access in deployment-prep project
Closed, ResolvedPublic

Description

The MediaWiki installs use the user "apache" as the runtime user for MediaWiki. We would like this user to be able to write content to the NFS4 shared mounts in the deployment-prep labs project. This in turn requires that the user and group are available on the NFS server.

The current uid of this user seems to vary from deployment-prep host to host:

  • deployment-bastion: uid=48(apache) gid=48(apache) groups=48(apache)
  • deployment-jobrunner01: uid=48(apache) gid=48(apache) groups=48(apache)
  • deployment-mediawiki01: uid=997(apache) gid=48(apache) groups=48(apache)
  • deployment-mediawiki02: uid=997(apache) gid=48(apache) groups=48(apache)
  • deployment-mediawiki03: uid=997(apache) gid=48(apache) groups=48(apache)
  • deployment-rsync01: uid=48(apache) gid=48(apache) groups=48(apache)
  • deployment-upload: uid=48(apache) gid=48(apache) groups=48(apache)

It looks like on Precise hosts we ended up with 48:48 and on Trusty hosts we have 997:48. 48:48 would work for LDAP I think if that UID:GID is not already taken.

Event Timeline

bd808 raised the priority of this task from to High.
bd808 updated the task description. (Show Details)
bd808 changed Security from none to None.
bd808 added subscribers: bd808, greg, Reedy and 2 others.
bd808 moved this task from Scheduled to Blocked on the Scrum-of-Scrums board.Nov 26 2014, 8:49 PM

Marc-André / Andrew B. created such users previously for the l10nupdate and mwdeploy system account.

Andrew added a comment.Dec 2 2014, 5:51 PM

I just added users Apache with uid/gid 48/48. I'm not 100% positive this won't cause conflicts on some systems due to the unusually low uid, so please approach with caution.

Should this be closed? Or does it need more consultation?

Should this be closed? Or does it need more consultation?

@Andrew said he created the user in ldap, but

deployment-logstash1:~
bd808$ id apache
id: apache: No such user

So something is not quite right yet. I wonder if the low uid/gid is filtered out from the ldap config on the host or something?

Andrew added a comment.Dec 3 2014, 8:29 PM

Confirmed -- when I move the apache user to id 607 everything works normally.

Since those low numbers are typically reserved for package users, ldap/pam apparently does not assign them. Is 607 OK?

hashar added a comment.Dec 3 2014, 8:29 PM

The user and group are definitely in:

$ ldaplist -l passwd apache

dn: uid=apache,ou=people,dc=wikimedia,dc=org
	uid: apache
	objectClass: person
	objectClass: organizationalPerson
	objectClass: inetorgperson
	objectClass: ldappublickey
	objectClass: shadowaccount
	objectClass: posixaccount
	objectClass: top
	loginShell: /usr/local/bin/sillyshell
	uidNumber: 48
	gidNumber: 48
	sn: apache
	homeDirectory: /home/apache
	cn: apache

$ ldaplist -l group apache

dn: cn=apache,ou=groups,dc=wikimedia,dc=org
	objectClass: groupOfNames
	objectClass: posixGroup
	objectClass: top
	member: uid=apache,ou=people,dc=wikimedia,dc=org
	gidNumber: 48
	cn: apache

Bryan assumption about low UID being ignored is right:

nslcd.conf:nss_min_uid 499

So should we revert the LDAP change and use puppet to ensure we have apache:apache with 48:48 ? The Precise instances seems to have UID = 48 but the Trusty ones have UID = 997.

:(

Confirmed -- when I move the apache user to id 607 everything works normally.
Since those low numbers are typically reserved for package users, ldap/pam apparently does not assign them. Is 607 OK?

So should we revert the LDAP change and use puppet to ensure we have apache:apache with 48:48 ? The Precise instances seems to have UID = 48 but the Trusty ones have UID = 997.

Creating the user everywhere (including the NFS server) via puppet should work, but using a different uid/gid wouldn't be the end of the work either. We'd just have to chown files on the local servers to match the new uid/gid as we scrubbed the entries from the local /etc/passwd and /etc/group files. I don't think it amounts to much more work on the beta servers to renumber and it's easier to have @Andrew make the ldap change for us than to push this sort of thing through operations/puppet.git. My vote is for the larger number in ldap.

Hmm, it's causing puppet failures already:

Error: /Stage[main]/Mediawiki::Users/Group[apache]/gid: change from 607 to 48 failed: Could not set gid on group[apache]: Execution of '/usr/sbin/groupmod -g 48 apache' returned 4: groupmod: GID '48' already exists

on deployment-videoscaler01.

Hmm, it's causing puppet failures already:

Error: /Stage[main]/Mediawiki::Users/Group[apache]/gid: change from 607 to 48 failed: Could not set gid on group[apache]: Execution of '/usr/sbin/groupmod -g 48 apache' returned 4: groupmod: GID '48' already exists

on deployment-videoscaler01.

blerg. Well if the gid is 48 in puppet we should probably make it 48 in ldap. I guess we must not be setting the uid in puppet since that drifts between the precise and trusty instances.

chasemp added a subscriber: chasemp.Dec 4 2014, 4:53 PM

hooray for making gid/uid's consistent between ldap and puppet.

coren added a comment.Dec 4 2014, 5:05 PM

Guys, sorry I didn't notice this thread earlier. You can't actually add in LDAP users that are normally managed by packages -- apt-get really doesn't like this at all!

You have to add the user on the NFS servers themselves -- not in LDAP. NFS4 works with usernames (and not user IDs) so that so long as a user exists on the server with that name everything will work fine. That's not a limitation of our setup but of the debian packaging system (it refuses to use an existing user if the package defines one of the same name, even if you use a directory service).

coren added a comment.Dec 4 2014, 5:08 PM

Just so that it is clear: the actual UID does not need to be the same. You can add the user in the node /labstore100[12]\.eqiad\.wmnet/ in site.pp; though now that I'm readying labstore servers for codfw that configuration should probably be spun off into a role.

Andrew added a comment.Dec 4 2014, 6:18 PM

ok... I'm happy to do whatever's needed but I'm now confused. It sounds like the right solution is something like:

a) remove the apache user from ldap entirely
b) add the user to labstore* in puppet (using what uid?)
c) ...?

Andrew added a comment.Dec 4 2014, 6:49 PM

Ok, I've removed the apache user from ldap. And...

https://gerrit.wikimedia.org/r/#/c/177584/

Change 177584 had a related patch set uploaded (by Tim Landscheidt):
Add an 'apache' user to eqiad labstores.

https://gerrit.wikimedia.org/r/177584

Patch-For-Review

Dzahn added a subscriber: Dzahn.Dec 5 2014, 2:30 PM

UID 48 for Apache should be correct, yep
see this for our kind of standard:

https://wikitech.wikimedia.org/wiki/UID

Andrew added a comment.Dec 9 2014, 5:21 PM

The associated patch was merged. Can someone confirm that this is now resolved?

hashar added a comment.Dec 9 2014, 9:56 PM

Not yet

On a Trusty instance which knows apache with uid 997 I create a file:

hashar@deployment-mediawiki01:~$ id apache
uid=997(apache) gid=48(apache) groups=48(apache)
hashar@deployment-mediawiki01:~$ sudo -u apache touch /data/project/upload7/trusty-997
hashar@deployment-mediawiki01:~$ stat /data/project/upload7/trusty-997|fgrep Uid
Access: (0644/-rw-r--r--)  Uid: (  997/  apache)   Gid: (    0/    root)

On a Precise instance, the file belong to the pybal-check (997) user:

hashar@deployment-jobrunner01:~$ stat /data/project/upload7/trusty-997|fgrep Uid
Access: (0644/-rw-r--r--)  Uid: (  997/pybal-check)   Gid: (    0/    root)
hashar@deployment-jobrunner01:~$ sudo -u apache touch /data/project/upload7/trusty-997
touch: cannot touch `/data/project/upload7/trusty-997': Permission denied
hashar@deployment-jobrunner01:~$

The other way around:

Precise:

hashar@deployment-jobrunner01:~$ sudo -u apache touch /data/project/upload7/precise-48
hashar@deployment-jobrunner01:~$ stat /data/project/upload7/precise-48 |fgrep Uid
Access: (0644/-rw-r--r--)  Uid: (   48/  apache)   Gid: (    0/    root)

Trusty:

hashar@deployment-mediawiki01:~$ stat  /data/project/upload7/precise-48|fgrep Uid
Access: (0644/-rw-r--r--)  Uid: (   48/ UNKNOWN)   Gid: (    0/    root)
hashar@deployment-mediawiki01:~$ sudo -u apache touch /data/project/upload7/precise-48
touch: cannot touch ‘/data/project/upload7/precise-48’: Permission denied


It works fine for the mwdeploy user:

Trusty:

hashar@deployment-mediawiki01:~$ id mwdeploy
uid=603(mwdeploy) gid=603(mwdeploy) groups=603(mwdeploy)
hashar@deployment-mediawiki01:~$ sudo -u mwdeploy touch /data/project/upload7/trusty-mwdeploy
hashar@deployment-mediawiki01:~$ stat /data/project/upload7/trusty-mwdeploy|fgrep Uid
Access: (0644/-rw-r--r--)  Uid: (  603/mwdeploy)   Gid: (    0/    root)
hashar@deployment-mediawiki01:~$

Precise:

hashar@deployment-jobrunner01:~$ stat /data/project/upload7/trusty-mwdeploy|fgrep Uid
Access: (0644/-rw-r--r--)  Uid: (  603/mwdeploy)   Gid: (    0/    root)

It works as well when touching a file from Precise as mwdeploy and looking it up on Trusty instance.

So seems NFS does rely on the UID :-/

Not yet
On a Trusty instance which knows apache with uid 997 I create a file:

hashar@deployment-mediawiki01:~$ id apache
uid=997(apache) gid=48(apache) groups=48(apache)
hashar@deployment-mediawiki01:~$ sudo -u apache touch /data/project/upload7/trusty-997
hashar@deployment-mediawiki01:~$ stat /data/project/upload7/trusty-997|fgrep Uid
Access: (0644/-rw-r--r--)  Uid: (  997/  apache)   Gid: (    0/    root)

On a Precise instance, the file belong to the pybal-check (997) user:

hashar@deployment-jobrunner01:~$ stat /data/project/upload7/trusty-997|fgrep Uid
Access: (0644/-rw-r--r--)  Uid: (  997/pybal-check)   Gid: (    0/    root)
hashar@deployment-jobrunner01:~$ sudo -u apache touch /data/project/upload7/trusty-997
touch: cannot touch `/data/project/upload7/trusty-997': Permission denied
hashar@deployment-jobrunner01:~$

[snip]

It works as well when touching a file from Precise as mwdeploy and looking it up on Trusty instance.
So seems NFS does rely on the UID :-/

The fact that you can touch the file now shows that the NFS server side is fixed. Now we need to renumber the user account on our local instances so that UID for apache is 48 everywhere for things to be consistent. I think the safest way to do this is to renumber manually and then introduce puppet code that ensures that the UID is 48 for new instances.

The fact that you can touch the file now shows that the NFS server side is fixed.

One step at a time \O/

Now we need to renumber the user account on our local instances so that UID for apache is 48 everywhere for things to be consistent. I think the safest way to do this is to renumber manually and then introduce puppet code that ensures that the UID is 48 for new instances.

Does anyone have any idea where the 997 uid is defined? The puppet class mediawiki::users creates apache:apache 48:48. So I am not sure why the Trusty instances ends up with a consistent 997.

bd808 closed this task as Resolved.Dec 9 2014, 10:23 PM
bd808 claimed this task.

I opened T78076: Make www-data the web-serving user (is currently apache) to track the needed local uid changes.

Thanks for the Ops help!

hashar moved this task from Blocked to Done on the Scrum-of-Scrums board.
Andrew mentioned this in Unknown Object (Diffusion Commit).Dec 10 2014, 8:38 PM
Restricted Application added a project: User-bd808. · View Herald TranscriptJun 7 2017, 6:57 PM