Page MenuHomePhabricator

Can't deploy Citoid in Beta
Closed, ResolvedPublic0 Story Points

Description

It seems the deploy-service user cannot connect to target nodes in Beta. From deploy-log:

deploy-log
-- Opening log file: '/mnt/srv/deployment/citoid/deploy/scap/log/scap-sync-2016-04-14-0001.log'
08:15:30 [deployment-tin] Started Deploy: citoid/deploy
08:15:30 [deployment-tin] 
== DEFAULT ==
:* deployment-sca02.eqiad.wmflabs
:* deployment-sca01.eqiad.wmflabs
08:15:30 [deployment-tin] [u'/usr/bin/deploy-local', u'-v', u'--repo', u'citoid/deploy', u'--force', u'-g', u'default', u'fetch'] on deployment-sca02.eqiad.wmflabs returned [255]: Host key verification failed.

08:15:30 [deployment-tin] [u'/usr/bin/deploy-local', u'-v', u'--repo', u'citoid/deploy', u'--force', u'-g', u'default', u'fetch'] on deployment-sca01.eqiad.wmflabs returned [255]: Host key verification failed.

08:15:30 [deployment-tin] 2 targets had deploy errors
08:15:36 [deployment-tin] Finished Deploy: citoid/deploy (duration: 00m 05s)

Event Timeline

mobrovac created this task.Apr 14 2016, 8:28 AM
Restricted Application added a project: VisualEditor. · View Herald TranscriptApr 14 2016, 8:28 AM
Restricted Application added a subscriber: Aklapper. · View Herald Transcript
hashar added a subscriber: hashar.Apr 14 2016, 2:00 PM

Seems like one has to manually accept the remote ssh host key. On production the various host keys are collected via puppet but that is not available on labs / beta (yet?).

Should be as easy as sudo deploy-service, then ssh to the various host and accepting the ssh fingerprint for the host.

hashar triaged this task as Normal priority.Apr 14 2016, 2:00 PM

Actually, it turns out the user doesn't even exist there:

root@deployment-tin:~# groups deploy-service
groups: deploy-service: no such user

How is that possible?

Seems like one has to manually accept the remote ssh host key. On production the various host keys are collected via puppet but that is not available on labs / beta (yet?).

It's T72792: Set up puppet exported resources to collect ssh host keys for beta

Actually, it turns out the user doesn't even exist there:

root@deployment-tin:~# groups deploy-service
groups: deploy-service: no such user

How is that possible?

There's no user but it exists as a group:

krenair@deployment-tin:~$ getent group deploy-service
deploy-service:x:52946:thcipriani,ladsgroup,mobrovac

The Group exists:

thcipriani@deployment-tin:~$ getent group deploy-service
deploy-service:x:52946:thcipriani,ladsgroup,mobrovac

It was added ad-hoc. I've never been super clear on how groups work on beta since it doesn't use the admin module.

The error you're seeing is just host-key checking via ssh without an input method (e.g. ssh -oBatchMode=yes). So rather than being prompted to accept the key, it just fails. Since you're sshing to the target machine as your user, you just need to accept the host-keys for the targets as your user.

You can see the actual error if you run the ssh command manually:

SSH_AUTH_SOCK=/run/keyholder/proxy.sock ssh -l deploy-service deployment-sca01.deployment-prep.eqiad.wmflabs
thcipriani@deployment-tin:~$ SSH_AUTH_SOCK=/run/keyholder/proxy.sock ssh -l deploy-service deployment-sca01.deployment-prep.eqiad.wmflabs
The authenticity of host 'deployment-sca01.deployment-prep.eqiad.wmflabs (10.68.20.183)' can't be established.
ECDSA key fingerprint is 22:a1:ab:5f:3d:9d:b2:e0:a1:52:5e:3d:72:f0:49:b0.
Are you sure you want to continue connecting (yes/no)? ^C
thcipriani@deployment-tin:~$ SSH_AUTH_SOCK=/run/keyholder/proxy.sock ssh -oBatchMode=yes -l deploy-service deployment-sca01.deployment-prep.eqiad.wmflabs
Host key verification failed.

After accepting keys it should work:

thcipriani@deployment-tin:~$ while read host; do ssh-keyscan -H "$host" >> ~/.ssh/known_hosts; done < /srv/deployment/citoid/deploy/scap/betacluster
# deployment-sca01.deployment-prep.eqiad.wmflabs SSH-2.0-OpenSSH_6.7p1 Debian-5+deb8u1
# deployment-sca01.deployment-prep.eqiad.wmflabs SSH-2.0-OpenSSH_6.7p1 Debian-5+deb8u1
# deployment-sca02.deployment-prep.eqiad.wmflabs SSH-2.0-OpenSSH_6.7p1 Debian-5+deb8u1
# deployment-sca02.deployment-prep.eqiad.wmflabs SSH-2.0-OpenSSH_6.7p1 Debian-5+deb8u1
thcipriani@deployment-tin:~$ SSH_AUTH_SOCK=/run/keyholder/proxy.sock ssh -oBatchMode=yes -l deploy-service deployment-sca01.deployment-prep.eqiad.wmflabs
Warning: Permanently added the ECDSA host key for IP address '10.68.20.183' to the list of known hosts.
Linux deployment-sca01 3.19.0-2-amd64 #1 SMP Debian 3.19.3-9 (2016-01-04) x86_64
Debian GNU/Linux 8.3 (jessie)
deployment-sca01 is a Puppet client of deployment-puppetmaster.deployment-prep.eqiad.wmflabs (puppetclient)
deployment-sca01 is role::citoid
deployment-sca01 is a node.js service converting graph definitions into PNG (role::graphoid)
The last Puppet run was at Thu Apr 14 14:54:40 UTC 2016 (4 minutes ago). 
Last login: Thu Apr 14 14:58:00 2016 from deployment-tin.deployment-prep.eqiad.wmflabs
deploy-service@deployment-sca01:~$

Duh, thnx @thcipriani! For deployment-sca01 it works, but I get asked for my password for deployment-sca02:

mobrovac@deployment-tin:/srv/deployment/citoid/deploy$ SSH_AUTH_SOCK=/run/keyholder/proxy.sock ssh -l deploy-service deployment-sca02.deployment-prep.eqiad.wmflabs
The authenticity of host 'deployment-sca02.deployment-prep.eqiad.wmflabs (10.68.20.153)' can't be established.
ECDSA key fingerprint is db:8a:b3:82:c1:a6:1d:93:e6:74:3d:c3:60:00:7a:cf.
Are you sure you want to continue connecting (yes/no)? yes
Warning: Permanently added 'deployment-sca02.deployment-prep.eqiad.wmflabs,10.68.20.153' (ECDSA) to the list of known hosts.
Password:

:(

root@deployment-sca02:/etc/ssh/userkeys# cat deploy-service 
ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCvbV8H7VzyH+NZuCMakT/YYIZH8qyzi1VefDuidplENGZgnXf1whCASFKpgE/y2aZOhHOFQR4jg42dRMDbmEQQGWFm/0Ve4gjmdv87ZNIPdZiGKgWBPnt1+XaUdRc+RvTS9uJg67XhKA5mxyk4uj5wj9OZq65C1okQIC3FuowhZCTkdHryjrQ1JxrayAvI6gED2rALIgScqfvw+UwJ/guoS9XH/yb07SNcuHmF9H5nRTsAYzfvv71/rVTFFTyrnSBUNDIvXsHJjKTwD4SX3FsZPsQ0mK/Uu98YypLN+hsIEFF6q8YTxxab2T2kV59MI7XBJgQi+MptDezA2y2nqp0L servicedeploy_rsa
root@deployment-sca01:/etc/ssh/userkeys# cat deploy-service
ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQCm0Ayo5jaTYKUBCLG6TBs/KSbaN+zT3YGXmaMDeAj0404eBgp1lPoskr/wKM2yEgA6P2rm6OXBOgoY8gdMU0ZNECykeVe5lPAYHjmkmxJRPUQknvWlPEAQ7tBw2EXeRAcgZHbwdhLT3rq8jNNYzIqYPu0OrNnqL3QMYmZkJrclittR1IQOmZ40lLdv2n0E+zO7cTGKyhZHG9hZEJwEERY0xq1uNUsisw3heMeUFtxfDb1iOx8hV7ZK64sYufMMfzuTDwIzWWLPwFGWW+frl1vAex3XoC+AtF+SaZbT185dyPljCWEZNSczSsdF6+MJ41jl0ds9VJDEyy1mc0gK/yB5t/oN+rtau+trtRyqdpx12obuZ1wLb6rJ6f3uDGz/0WY+JgopE3Jb0JHng70IhogLWK/CubczHIAHH90oxMKbPwVc8TOIDKt7RH98y5DvGBbCDW8z9HNqADTCYVx7rH4bDJqkN8nJRELMok0Y9ltJIEM4HCQCBnhLMgqckKrPInRdHgWntxucIi7TuF6QHrikVycDo1klK7J3ST5vy0wtwKCzxSzNbu/cyYURxoZeIZq8Ud1fSypVgdoUkcu10+PY3TaOSEGQT8vsZ+jci1MkWc3yZ3cIBGQhyRDfVyanUASPE9ToPG9Is2CHtbSqUtUu+xjpzcLrs6vI4b7n8fBzpQ== servicedeploy

Hmmm, so I guess Puppet doesn't manage the keys? I'll just copy it from sca01

Hmmm, so I guess Puppet doesn't manage the keys? I'll just copy it from sca01

I think what's happening is that the key is correct because ORES is also using sca01.

service::node is just using the public key in modules/service/files/servicedeploy_rsa.pub

$ ssh-keygen -l -E md5 -f modules/service/files/servicedeploy_rsa.pub 
2048 MD5:6d:54:92:8b:39:10:f5:9b:84:40:36:ef:3c:9a:6d:d8 servicedeploy_rsa (RSA)
root@deployment-sca02:~# ssh-keygen -l -f /etc/ssh/userkeys/deploy-service 
2048 6d:54:92:8b:39:10:f5:9b:84:40:36:ef:3c:9a:6d:d8 /etc/ssh/userkeys/deploy-service (RSA)

We need to make the service-key path for a service::node configurable via hiera.

So where is the key on sca01 coming from?

Oh, you mean ORES somehow forces the key to be correct on sca01 via some method we already configured in beta?

Related discussion. Please correct me if I'm wrong but the key I'm defining is actually the one that the keyholder uses for "service-deploy" user

Oh, you mean ORES somehow forces the key to be correct on sca01 via some method we already configured in beta?

Exactly.

ORES is using the correct key for beta, which means that citoid can be deployed to sca01 but not sca02.

After talking with @mobrovac, it seems like the problem that has to be solved here is that our puppet setup shouldn't allow scap::target to redefine different keys for the same user. The act of defining a deploy_user in a scap::target should lock down the public key that is used in a smart way. Multiple public keys using the same user, while normal for some systems, seems problematic in this instance. It could lead to vulnerabilities if patches get merged without a wide view of the system.

After talking with @mobrovac, it seems like the problem that has to be solved here is that our puppet setup shouldn't allow scap::target to redefine different keys for the same user. The act of defining a deploy_user in a scap::target should lock down the public key that is used in a smart way. Multiple public keys using the same user, while normal for some systems, seems problematic in this instance. It could lead to vulnerabilities if patches get merged without a wide view of the system.

Filed as T132747: scap::target shouldn't allow users to redefine the user's key

Restricted Application added a subscriber: TerraCodes. · View Herald TranscriptApr 19 2016, 7:13 PM
Jdforrester-WMF set the point value for this task to 0.Apr 28 2016, 1:25 AM