Ori has given a -2 to https://gerrit.wikimedia.org/r/134519 due to the need for a work around for this bug.

Adding Andrew Bogott as a cc. I think the best solution to attempt for this would be to change the shell of mwdeploy in labs LDAP to /bin/false. We may have to adjust some bits of the deploy pipeline as a result of this, but that should be relatively easy (famous last words).

In ldap, user mwdeploy now has loginShell: /bin/false

When I revert the local change in the mediawiki::users class that sets the shell for User['mwdeploy'] to /bin/bash and let shell => /bin/false as is done in production, puppet still fails:

Error: /Stage[main]/Mediawiki::Users/User[mwdeploy]/shell: change from /bin/bash to /bin/false failed: Could not set shell on user[mwdeploy]: Execution of '/usr/sbin/usermod -s /bin/false mwdeploy' returned 6: usermod: user 'mwdeploy' does not exist in /etc/passwd

On deployment-apache02, getent passwd mwdeploy is still returning /bin/bash as the login shell for the mwdeploy user:

mwdeploy:x:603:603:mwdeploy:/var/lib/mwdeploy:/bin/bash

This result persists after restarting both nslcd and nscd to clear any locally cached info for the user. This seems consistent with the prior behavior as Andrew said that before his change loginShell for the mwdeploy user was set to /usr/local/bin/sillyshell and not /bin/bash as puppet sees.

$ ldapsearch -x uid=mwdeploy \* +

1. extended LDIF #
2. LDAPv3
3. base <dc=wikimedia,dc=org> (default) with scope subtree
4. filter: uid=mwdeploy
5. requesting: * + #

mwdeploy, people, wikimedia.org

dn: uid=mwdeploy,ou=people,dc=wikimedia,dc=org
objectClass: person
objectClass: inetorgperson
objectClass: organizationalPerson
objectClass: ldappublickey
objectClass: shadowaccount
objectClass: posixaccount
objectClass: top
uid: mwdeploy
cn: mwdeploy
sn: mwdeploy
loginShell: /bin/false
homeDirectory: /var/lib/mwdeploy
uidNumber: 603
gidNumber: 603
etag: 000000002dd998e4
pwdPolicySubentry: cn=Default Password Policy,cn=Password Policies,cn=config
subschemaSubentry: cn=schema
isMemberOf: cn=mwdeploy,ou=groups,dc=wikimedia,dc=org
createTimestamp: 20140314143125Z
numSubordinates: 0
structuralObjectClass: inetOrgPerson
ds-sync-hist: loginshell:00000146ed8542853a2500002d59:repl:/bin/false
ds-sync-hist: modifytimestamp:00000146ed8542853a2500002d59:repl:20140630160432
Z
ds-sync-hist: modifiersname:00000146ed8542853a2500002d59:repl:uid=andrew,ou=pe
ople,dc=wikimedia,dc=org
hasSubordinates: false
entryDN: uid=mwdeploy,ou=people,dc=wikimedia,dc=org
entryUUID: bab89dfb-2f85-4379-9797-f2f9d0d4e267
creatorsName: uid=novaadmin,ou=people,dc=wikimedia,dc=org
modifyTimestamp: 20140630160432Z
modifiersName: uid=andrew,ou=people,dc=wikimedia,dc=org

search result

search: 2
result: 0 Success

1. numResponses: 2
2. numEntries: 1

$ getent passwd mwdeploy
mwdeploy:x:603:603:mwdeploy:/var/lib/mwdeploy:/bin/bash

From deployment-salt:

$ getent passwd|cut -d: -f7|sort|uniq -c

4519 /bin/bash
  11 /bin/false
  18 /bin/sh
   1 /bin/sync
   2 /usr/sbin/nologin

None of the 11 users with /bin/false shells are in ldap. They all come from the local password file.

$ ldapsearch -x uid=bd808 \* + | grep loginShell
loginShell: /usr/local/bin/sillyshell

$ getent passwd bd808
bd808:x:3518:500:BryanDavis:/home/bd808:/bin/bash

So loginShell in ldap does not seem to be the source of the shell for a user. The string "bash" does not appear in ldapsearch output for my user. Magic?

On tools-login, /etc/ldap.conf has:

I haven't read up on that, but that looks like the culprit.

The line (in essence) was introduced by Ryan's commit f8724e60664a33a37a327434f5c3cb71837f4c20 (Sep 13 00:26:36 2011) with the message "Update from svn repo". Ryan, do you remember why you set that configuration, and if the reasoning still holds up?

(In reply to Tim Landscheidt from comment #8)

The line (in essence) was introduced by Ryan's commit
f8724e60664a33a37a327434f5c3cb71837f4c20 (Sep 13 00:26:36 2011) with the
message "Update from svn repo". Ryan, do you remember why you set that
configuration, and if the reasoning still holds up?

I imagine it has something to do with:

$ ldapsearch -x uid=bd808 \* + | grep loginShell
loginShell: /usr/local/bin/sillyshell

Ah. And that seems to relate to the subversion module where users apparently were only allowed to execute "svnserve -t".

So to migrate safely we would need to:

1. Add the override to /usr/local/bin/sillyshell in Subversion server instances,
2. replace the LDAP attributes that point to /usr/local/bin/sillyshell with /bin/bash, and
3. remove the override for not(Subversion server) instances?

(In reply to myself from comment #10)

[...]
So to migrate safely we would need to:

1. Add the override to /usr/local/bin/sillyshell in Subversion server

instances,

2. replace the LDAP attributes that point to /usr/local/bin/sillyshell with

/bin/bash, and

3. remove the override for not(Subversion server) instances?

Step #1 could maybe lock out ops from the Subversion server, so that needs more thought/a workaround/etc.

This is recurring now since someone has accidentally let the local commits die.

A local hack for this is currently applied on the beta cluster:

[LOCAL HACK] Bug 65591: User['mwdeploy'] shell => /bin/bash

In modules/beta/manifests/scap/target.pp

  class beta::scap::target {
     include ::beta::config
     include ::mediawiki::scap
+    include ::mediawiki::users

Change 185409 had a related patch set uploaded (by Yuvipanda):
beta: Ensure that mw related users are present in scap targets

https://gerrit.wikimedia.org/r/185409

Patch-For-Review

Change 185409 merged by Yuvipanda:
beta: Ensure that mw related users are present in scap targets

https://gerrit.wikimedia.org/r/185409