The striker-deploy03.striker.eqiad.wmflabs instance serves two roles in the Striker Cloud VPS project:
The Puppet configuration for deploying a scap3 server has changed such that it only works on Debian hosts (specifically Debian Stretch?) so this role at least needs to move to a new instance.
| Subject | Repo | Branch | Lines +/- | |
|---|---|---|---|---|
| ssh-key-ldap-lookup: handle missing users | operations/puppet | production | +7 -4 |
I built striker-deploy04.striker.eqiad.wmflabs as a Stretch instance with profile::mediawiki::deployment::server applied to it. There were some failures with the initial provisioning of the /srv/deployment directories, but I worked through them until Puppet ran cleanly.
scap deploy --verbose is failing due to ssh key rejection:
$ cd /srv/deployment/striker/deploy/ $ scap deploy --verbose ... 16:54:04 ['/usr/bin/scap', 'deploy-local', '-v', '--repo', 'striker/deploy', '-g', 'default', 'fetch', '--refresh-config'] on striker-uwsgi03.striker.eqiad.wmflabs returned [255]: OpenSSH_7.4p1 Debian-10+deb9u3, OpenSSL 1.0.2l 25 May 2017 debug1: Reading configuration data /dev/null debug1: Connecting to striker-uwsgi03.striker.eqiad.wmflabs [10.68.16.203] port 22. debug1: Connection established. debug1: identity file /etc/keyholder.d/deploy_service.pub type 1 debug1: key_load_public: No such file or directory debug1: identity file /etc/keyholder.d/deploy_service.pub-cert type -1 debug1: Enabling compatibility mode for protocol 2.0 debug1: Local version string SSH-2.0-OpenSSH_7.4p1 Debian-10+deb9u3 debug1: Remote protocol version 2.0, remote software version OpenSSH_7.4p1 Debian-10+deb9u2 debug1: match: OpenSSH_7.4p1 Debian-10+deb9u2 pat OpenSSH* compat 0x04000000 debug1: Authenticating to striker-uwsgi03.striker.eqiad.wmflabs:22 as 'deploy-service' debug1: SSH2_MSG_KEXINIT sent debug1: SSH2_MSG_KEXINIT received debug1: kex: algorithm: curve25519-sha256@libssh.org debug1: kex: host key algorithm: ecdsa-sha2-nistp256 debug1: kex: server->client cipher: chacha20-poly1305@openssh.com MAC: <implicit> compression: none debug1: kex: client->server cipher: chacha20-poly1305@openssh.com MAC: <implicit> compression: none debug1: expecting SSH2_MSG_KEX_ECDH_REPLY debug1: Server host key: ecdsa-sha2-nistp256 SHA256:HlIsZ5PvvlQ6vQjcOxQLCGQ3KNukao9QeYi0georBwc debug1: Host 'striker-uwsgi03.striker.eqiad.wmflabs' is known and matches the ECDSA host key. debug1: Found key in /home/bd808/.ssh/known_hosts:1 debug1: rekey after 134217728 blocks debug1: SSH2_MSG_NEWKEYS sent debug1: expecting SSH2_MSG_NEWKEYS debug1: SSH2_MSG_NEWKEYS received debug1: rekey after 134217728 blocks debug1: SSH2_MSG_EXT_INFO received debug1: kex_input_ext_info: server-sig-algs=<ssh-ed25519,ssh-rsa,ssh-dss,ecdsa-sha2-nistp256,ecdsa-sha2-nistp384,ecdsa-sha2-nistp521> debug1: SSH2_MSG_SERVICE_ACCEPT received debug1: Authentications that can continue: publickey debug1: Next authentication method: publickey debug1: Offering RSA public key: /etc/keyholder.d/deploy_service.pub debug1: Server accepts key: pkalg ssh-rsa blen 535 Authentication failed. 16:54:04 connection to striker-uwsgi03.striker.eqiad.wmflabs failed and future stages will not be attempted for this target
From /var/log/auth.log on striker-uwsgi03:
Aug 6 16:54:03 striker-uwsgi03 sshd[10258]: error: AuthorizedKeysCommand /usr/sbin/ssh-key-ldap-lookup deploy-service failed, status 1 Aug 6 16:54:03 striker-uwsgi03 sshd[10258]: Postponed publickey for deploy-service from 10.68.20.78 port 60194 ssh2 [preauth] Aug 6 16:54:04 striker-uwsgi03 sshd[10258]: error: AuthorizedKeysCommand /usr/sbin/ssh-key-ldap-lookup deploy-service failed, status 1 Aug 6 16:54:04 striker-uwsgi03 sshd[10258]: pam_access(sshd:account): access denied for user `deploy-service' from `10.68.20.78' Aug 6 16:54:04 striker-uwsgi03 sshd[10258]: Failed publickey for deploy-service from 10.68.20.78 port 60194 ssh2: RSA SHA256:BkaF2U4ziL7sesuRSV9UaEoQzK3GvoWZWgGCrKxUYBo Aug 6 16:54:04 striker-uwsgi03 sshd[10258]: fatal: Access denied for user deploy-service by PAM account configuration [preauth]
The deploy-service user exists locally and the expected public key is in /etc/ssh/userkeys/deploy-service. I would expect the /usr/sbin/ssh-key-ldap-lookup script to fail since the deploy-service user is local and not in the LDAP directory. Is there something missing from the local sshd or PAM configuration on striker-uwsgi03 that used to make this work?
Change 481343 had a related patch set uploaded (by BryanDavis; owner: Bryan Davis):
[operations/puppet@production] ssh-key-ldap-lookup: handle missing users
Change 481343 merged by Andrew Bogott:
[operations/puppet@production] ssh-key-ldap-lookup: handle missing users