Page MenuHomePhabricator

Unable to ssh to ws-web.wikistream.eqiad.wmflabs as user edsu
Closed, ResolvedPublic

Description

I'm an admin on the wikistream project and am unable to ssh to ws-web to debug why the service is currently unavailable.

edsu@bastion-eqiad1-01:~$ ssh ws-web
Permission denied (publickey).

Event Timeline

edsu created this task.Tue, Sep 10, 2:45 PM
Restricted Application added a subscriber: Aklapper. · View Herald TranscriptTue, Sep 10, 2:45 PM
Aklapper renamed this task from Unable to ssh to ws-web to Unable to ssh to ws-web as an admin.Tue, Sep 10, 2:48 PM

@edsu: Could you run the command with verbose debug output please? (somewhere between ssh -v and ssh -vvv)

edsu added a comment.Tue, Sep 10, 3:08 PM

Thanks for the quick response! Here you go:

edsu@bastion-eqiad1-01:~$ ssh -v ws-web
OpenSSH_7.4p1 Debian-10+deb9u6, OpenSSL 1.0.2s  28 May 2019
debug1: Reading configuration data /etc/ssh/ssh_config
debug1: /etc/ssh/ssh_config line 19: Applying options for *
debug1: Connecting to ws-web [172.16.6.146] port 22.
debug1: Connection established.
debug1: key_load_public: No such file or directory
debug1: identity file /home/edsu/.ssh/id_rsa type -1
debug1: key_load_public: No such file or directory
debug1: identity file /home/edsu/.ssh/id_rsa-cert type -1
debug1: key_load_public: No such file or directory
debug1: identity file /home/edsu/.ssh/id_dsa type -1
debug1: key_load_public: No such file or directory
debug1: identity file /home/edsu/.ssh/id_dsa-cert type -1
debug1: key_load_public: No such file or directory
debug1: identity file /home/edsu/.ssh/id_ecdsa type -1
debug1: key_load_public: No such file or directory
debug1: identity file /home/edsu/.ssh/id_ecdsa-cert type -1
debug1: key_load_public: No such file or directory
debug1: identity file /home/edsu/.ssh/id_ed25519 type -1
debug1: key_load_public: No such file or directory
debug1: identity file /home/edsu/.ssh/id_ed25519-cert type -1
debug1: Enabling compatibility mode for protocol 2.0
debug1: Local version string SSH-2.0-OpenSSH_7.4p1 Debian-10+deb9u6
debug1: Remote protocol version 2.0, remote software version OpenSSH_6.7p1 Debian-5+deb8u8
debug1: match: OpenSSH_6.7p1 Debian-5+deb8u8 pat OpenSSH* compat 0x04000000
debug1: Authenticating to ws-web:22 as 'edsu'
debug1: SSH2_MSG_KEXINIT sent
debug1: SSH2_MSG_KEXINIT received
debug1: kex: algorithm: curve25519-sha256@libssh.org
debug1: kex: host key algorithm: ecdsa-sha2-nistp256
debug1: kex: server->client cipher: chacha20-poly1305@openssh.com MAC: <implicit> compression: none
debug1: kex: client->server cipher: chacha20-poly1305@openssh.com MAC: <implicit> compression: none
debug1: expecting SSH2_MSG_KEX_ECDH_REPLY
debug1: Server host key: ecdsa-sha2-nistp256 SHA256:zYBIo3+IoJW4Jm13gUDO8YKvBVzivJ1MC7+Huzdo6hM
debug1: Host 'ws-web' is known and matches the ECDSA host key.
debug1: Found key in /home/edsu/.ssh/known_hosts:1
debug1: rekey after 134217728 blocks
debug1: SSH2_MSG_NEWKEYS sent
debug1: expecting SSH2_MSG_NEWKEYS
debug1: SSH2_MSG_NEWKEYS received
debug1: rekey after 134217728 blocks
debug1: SSH2_MSG_SERVICE_ACCEPT received
debug1: Authentications that can continue: publickey
debug1: Next authentication method: publickey
debug1: Trying private key: /home/edsu/.ssh/id_rsa
debug1: Trying private key: /home/edsu/.ssh/id_dsa
debug1: Trying private key: /home/edsu/.ssh/id_ecdsa
debug1: Trying private key: /home/edsu/.ssh/id_ed25519
debug1: No more authentication methods to try.
Permission denied (publickey).
edsu added a comment.Tue, Sep 10, 3:37 PM

Hmm, looking at the docs perhaps it's no longer possible to ssh to bastion.wmflabs.org and then ssh on to ws-web? Should I follow the instructions at this page to set up bast1002.wikimedia.org as a proxy?

bd808 claimed this task.Tue, Sep 10, 9:43 PM
bd808 triaged this task as Normal priority.
bd808 added a subscriber: bd808.

Hmm, looking at the docs perhaps it's no longer possible to ssh to bastion.wmflabs.org and then ssh on to ws-web? Should I follow the instructions at this page to set up bast1002.wikimedia.org as a proxy?

Bast1002 is a bastion for accessing hosts inside the production network at Wikimedia rather than virtual instances in Cloud VPS. The bastion-eqiad1-01.bastion.eqiad.wmflabs (also known as bastion.wmflabs.org) is the correct bastion to use to access instances in the wikistream Cloud VPS project.

I will poke around in the log files on ws-web.wikistream.eqiad.wmflabs to see if I can find a reason for the auth failure.

srodlund added a subscriber: srodlund.
/var/log/auth
Sep 10 15:08:11 ws-web sshd[17813]: Set /proc/self/oom_score_adj to 0
Sep 10 15:08:11 ws-web sshd[17813]: Connection from 172.16.1.136 port 43756 on 172.16.6.146 port 22
Sep 10 15:08:11 ws-web sshd[17813]: Connection closed by 172.16.1.136 [preauth]

Not super helpful logging. I did see some logging in a vaguely similar time period about nslcd reloading. That service brokers lookups of your ssh public key in LDAP among other things. It is possible that you just had really bad luck and the
instance was unable to talk with the LDAP server at the time you tried to ssh in.

@edsu Could you try logging in again? It would be helpful to have a ssh -vvv output if you are still not able to connect successfully. At that level of debug output hopefully we will be able to see the the fingerprint(s) of the ssh public keys that your ssh-agent is offering so we can check them against the keys you have authorized through wikitech/toolsadmin to access your account. If that still doesn't get us anywhere we may have to sync up for live debugging on the #wikimedia-cloud Freenode IRC channel where I watch the logs in various places as you try additional logins.

Trying to be helpful while we work on the ssh issue, I believe I have gotten the wikistream service back up and running for the moment.

[22:25]  <    bd808>	!log wikistream Restarted varnish (T232486)
[22:34]  <    bd808>	!log wikistream Manually started `HOME=/var/tmp/ NODE_ENV=production /usr/bin/nodejs /opt/wikistream/app.js >/dev/null 2>&1 &` (T232486)
[22:35]  <    bd808>	crap. stashbot never came back because k8s is sick

The /etc/init/wikistream.conf upstart process really needs to be converted to a systemd unit to make this easier. Some docs on https://wikitech.wikimedia.org/wiki/Nova_Resource:Wikistream about how to restart both varnish and the node process would be extra nice.

bd808 renamed this task from Unable to ssh to ws-web as an admin to Unable to ssh to ws-web.wikistream.eqiad.wmflabs as user edsu.Tue, Sep 10, 11:21 PM
bd808 added a project: VPS-Projects.
edsu added a comment.Tue, Sep 10, 11:39 PM

Thanks for getting wikistream back working again. I'm still unable to ssh in however.

edsu@bastion-eqiad1-01:~$ ssh -vvvv ws-web
OpenSSH_7.4p1 Debian-10+deb9u6, OpenSSL 1.0.2s  28 May 2019
debug1: Reading configuration data /etc/ssh/ssh_config
debug1: /etc/ssh/ssh_config line 19: Applying options for *
debug2: resolving "ws-web" port 22
debug2: ssh_connect_direct: needpriv 0
debug1: Connecting to ws-web [172.16.6.146] port 22.
debug1: Connection established.
debug1: key_load_public: No such file or directory
debug1: identity file /home/edsu/.ssh/id_rsa type -1
debug1: key_load_public: No such file or directory
debug1: identity file /home/edsu/.ssh/id_rsa-cert type -1
debug1: key_load_public: No such file or directory
debug1: identity file /home/edsu/.ssh/id_dsa type -1
debug1: key_load_public: No such file or directory
debug1: identity file /home/edsu/.ssh/id_dsa-cert type -1
debug1: key_load_public: No such file or directory
debug1: identity file /home/edsu/.ssh/id_ecdsa type -1
debug1: key_load_public: No such file or directory
debug1: identity file /home/edsu/.ssh/id_ecdsa-cert type -1
debug1: key_load_public: No such file or directory
debug1: identity file /home/edsu/.ssh/id_ed25519 type -1
debug1: key_load_public: No such file or directory
debug1: identity file /home/edsu/.ssh/id_ed25519-cert type -1
debug1: Enabling compatibility mode for protocol 2.0
debug1: Local version string SSH-2.0-OpenSSH_7.4p1 Debian-10+deb9u6
debug1: Remote protocol version 2.0, remote software version OpenSSH_6.7p1 Debian-5+deb8u8
debug1: match: OpenSSH_6.7p1 Debian-5+deb8u8 pat OpenSSH* compat 0x04000000
debug2: fd 3 setting O_NONBLOCK
debug1: Authenticating to ws-web:22 as 'edsu'
debug3: hostkeys_foreach: reading file "/home/edsu/.ssh/known_hosts"
debug3: record_hostkey: found key type ECDSA in file /home/edsu/.ssh/known_hosts:1
debug3: load_hostkeys: loaded 1 keys from ws-web
debug3: order_hostkeyalgs: prefer hostkeyalgs: ecdsa-sha2-nistp256-cert-v01@openssh.com,ecdsa-sha2-nistp384-cert-v01@openssh.com,ecdsa-sha2-nistp521-cert-v01@openssh.com,ecdsa-sha2-nistp256,ecdsa-sha2-nistp384,ecdsa-sha2-nistp521
debug3: send packet: type 20
debug1: SSH2_MSG_KEXINIT sent
debug3: receive packet: type 20
debug1: SSH2_MSG_KEXINIT received
debug2: local client KEXINIT proposal
debug2: KEX algorithms: curve25519-sha256,curve25519-sha256@libssh.org,ecdh-sha2-nistp256,ecdh-sha2-nistp384,ecdh-sha2-nistp521,diffie-hellman-group-exchange-sha256,diffie-hellman-group16-sha512,diffie-hellman-group18-sha512,diffie-hellman-group-exchange-sha1,diffie-hellman-group14-sha256,diffie-hellman-group14-sha1,ext-info-c
debug2: host key algorithms: ecdsa-sha2-nistp256-cert-v01@openssh.com,ecdsa-sha2-nistp384-cert-v01@openssh.com,ecdsa-sha2-nistp521-cert-v01@openssh.com,ecdsa-sha2-nistp256,ecdsa-sha2-nistp384,ecdsa-sha2-nistp521,ssh-ed25519-cert-v01@openssh.com,ssh-rsa-cert-v01@openssh.com,ssh-ed25519,rsa-sha2-512,rsa-sha2-256,ssh-rsa
debug2: ciphers ctos: chacha20-poly1305@openssh.com,aes128-ctr,aes192-ctr,aes256-ctr,aes128-gcm@openssh.com,aes256-gcm@openssh.com,aes128-cbc,aes192-cbc,aes256-cbc
debug2: ciphers stoc: chacha20-poly1305@openssh.com,aes128-ctr,aes192-ctr,aes256-ctr,aes128-gcm@openssh.com,aes256-gcm@openssh.com,aes128-cbc,aes192-cbc,aes256-cbc
debug2: MACs ctos: umac-64-etm@openssh.com,umac-128-etm@openssh.com,hmac-sha2-256-etm@openssh.com,hmac-sha2-512-etm@openssh.com,hmac-sha1-etm@openssh.com,umac-64@openssh.com,umac-128@openssh.com,hmac-sha2-256,hmac-sha2-512,hmac-sha1
debug2: MACs stoc: umac-64-etm@openssh.com,umac-128-etm@openssh.com,hmac-sha2-256-etm@openssh.com,hmac-sha2-512-etm@openssh.com,hmac-sha1-etm@openssh.com,umac-64@openssh.com,umac-128@openssh.com,hmac-sha2-256,hmac-sha2-512,hmac-sha1
debug2: compression ctos: none,zlib@openssh.com,zlib
debug2: compression stoc: none,zlib@openssh.com,zlib
debug2: languages ctos: 
debug2: languages stoc: 
debug2: first_kex_follows 0 
debug2: reserved 0 
debug2: peer server KEXINIT proposal
debug2: KEX algorithms: curve25519-sha256@libssh.org,diffie-hellman-group-exchange-sha256
debug2: host key algorithms: ssh-rsa,ssh-dss,ecdsa-sha2-nistp256,ssh-ed25519
debug2: ciphers ctos: chacha20-poly1305@openssh.com,aes256-gcm@openssh.com,aes128-gcm@openssh.com,aes256-ctr,aes192-ctr,aes128-ctr
debug2: ciphers stoc: chacha20-poly1305@openssh.com,aes256-gcm@openssh.com,aes128-gcm@openssh.com,aes256-ctr,aes192-ctr,aes128-ctr
debug2: MACs ctos: hmac-sha2-512-etm@openssh.com,hmac-sha2-256-etm@openssh.com,umac-128-etm@openssh.com,hmac-sha2-512,hmac-sha2-256,umac-128@openssh.com
debug2: MACs stoc: hmac-sha2-512-etm@openssh.com,hmac-sha2-256-etm@openssh.com,umac-128-etm@openssh.com,hmac-sha2-512,hmac-sha2-256,umac-128@openssh.com
debug2: compression ctos: none,zlib@openssh.com
debug2: compression stoc: none,zlib@openssh.com
debug2: languages ctos: 
debug2: languages stoc: 
debug2: first_kex_follows 0 
debug2: reserved 0 
debug1: kex: algorithm: curve25519-sha256@libssh.org
debug1: kex: host key algorithm: ecdsa-sha2-nistp256
debug1: kex: server->client cipher: chacha20-poly1305@openssh.com MAC: <implicit> compression: none
debug1: kex: client->server cipher: chacha20-poly1305@openssh.com MAC: <implicit> compression: none
debug3: send packet: type 30
debug1: expecting SSH2_MSG_KEX_ECDH_REPLY
debug3: receive packet: type 31
debug1: Server host key: ecdsa-sha2-nistp256 SHA256:zYBIo3+IoJW4Jm13gUDO8YKvBVzivJ1MC7+Huzdo6hM
debug3: hostkeys_foreach: reading file "/home/edsu/.ssh/known_hosts"
debug3: record_hostkey: found key type ECDSA in file /home/edsu/.ssh/known_hosts:1
debug3: load_hostkeys: loaded 1 keys from ws-web
debug3: hostkeys_foreach: reading file "/home/edsu/.ssh/known_hosts"
debug3: record_hostkey: found key type ECDSA in file /home/edsu/.ssh/known_hosts:2
debug3: load_hostkeys: loaded 1 keys from 172.16.6.146
debug1: Host 'ws-web' is known and matches the ECDSA host key.
debug1: Found key in /home/edsu/.ssh/known_hosts:1
debug3: send packet: type 21
debug2: set_newkeys: mode 1
debug1: rekey after 134217728 blocks
debug1: SSH2_MSG_NEWKEYS sent
debug1: expecting SSH2_MSG_NEWKEYS
debug3: receive packet: type 21
debug1: SSH2_MSG_NEWKEYS received
debug2: set_newkeys: mode 0
debug1: rekey after 134217728 blocks
debug2: key: /home/edsu/.ssh/id_rsa ((nil))
debug2: key: /home/edsu/.ssh/id_dsa ((nil))
debug2: key: /home/edsu/.ssh/id_ecdsa ((nil))
debug2: key: /home/edsu/.ssh/id_ed25519 ((nil))
debug3: send packet: type 5
debug3: receive packet: type 6
debug2: service_accept: ssh-userauth
debug1: SSH2_MSG_SERVICE_ACCEPT received
debug3: send packet: type 50
debug3: receive packet: type 51
debug1: Authentications that can continue: publickey
debug3: start over, passed a different list publickey
debug3: preferred gssapi-keyex,gssapi-with-mic,publickey,keyboard-interactive,password
debug3: authmethod_lookup publickey
debug3: remaining preferred: keyboard-interactive,password
debug3: authmethod_is_enabled publickey
debug1: Next authentication method: publickey
debug1: Trying private key: /home/edsu/.ssh/id_rsa
debug3: no such identity: /home/edsu/.ssh/id_rsa: No such file or directory
debug1: Trying private key: /home/edsu/.ssh/id_dsa
debug3: no such identity: /home/edsu/.ssh/id_dsa: No such file or directory
debug1: Trying private key: /home/edsu/.ssh/id_ecdsa
debug3: no such identity: /home/edsu/.ssh/id_ecdsa: No such file or directory
debug1: Trying private key: /home/edsu/.ssh/id_ed25519
debug3: no such identity: /home/edsu/.ssh/id_ed25519: No such file or directory
debug2: we did not send a packet, disable method
debug1: No more authentication methods to try.
Permission denied (publickey).

Thanks for getting wikistream back working again. I'm still unable to ssh in however.

edsu@bastion-eqiad1-01:~$ ssh -vvvv ws-web
...snip...
debug3: authmethod_is_enabled publickey
debug1: Next authentication method: publickey
debug1: Trying private key: /home/edsu/.ssh/id_rsa
debug3: no such identity: /home/edsu/.ssh/id_rsa: No such file or directory
debug1: Trying private key: /home/edsu/.ssh/id_dsa
debug3: no such identity: /home/edsu/.ssh/id_dsa: No such file or directory
debug1: Trying private key: /home/edsu/.ssh/id_ecdsa
debug3: no such identity: /home/edsu/.ssh/id_ecdsa: No such file or directory
debug1: Trying private key: /home/edsu/.ssh/id_ed25519
debug3: no such identity: /home/edsu/.ssh/id_ed25519: No such file or directory
debug2: we did not send a packet, disable method
debug1: No more authentication methods to try.
Permission denied (publickey).

It appears that you are doing an ssh from your personal computer to bastion.wmflabs.org, and then from the shell on bastion.wmflabs.org trying to ssh to ws-web.wikistream.eqiad.wmflabs. This would only work if you are using ssh agent forwarding so that the ssh keys on your personal computer are available to the ssh process on bastion.wmflabs.org or you have ssh private keys stored in $HOME/.ssh/ on bastion.wmflabs.org. Based on the -vvv output you gave I believe neither is the case. That's actually not horrible as neither practice would be ideal. Agent forwarding has a potential to expose your ssh agent to root users on the intermediate host or worse to anyone who finds a local root exploit on those same hosts. Storing ssh private keys on any Cloud VPS instance, whether it is a bastion or not, is not a great idea either for the same reason of potential exposure of the private key to anyone with authorized or unauthorized root access to the filesystem.

The "best practice" method of accessing instances via ssh is by using the bastions as a "jump host" rather than with an ssh-in-ssh session. https://wikitech.wikimedia.org/wiki/Help:Access#Accessing_instances_with_ProxyJump_ssh_option_(recommended) documents the easiest way to set this up if you have a modern openssh client. There are also instructions there for older openssh clients which implement the ProxyCommand directive but not the ProxyJump directive. If you are using PuTTY on windows, see https://wikitech.wikimedia.org/wiki/Help:Access_to_Cloud_VPS_instances_with_PuTTY_and_WinSCP#How_to_set_up_PuTTY_for_proxying_through_bastion.wmflabs.org_to_your_instance

edsu added a comment.Wed, Sep 11, 2:51 PM

@bd808 thank you! I only was doing the ssh-within-ssh because it worked for me in the past. But I've got ProxyJump working fine now, and can access ws-web, so I appreciate you taking the time to explain this solution.

edsu closed this task as Resolved.Wed, Sep 11, 2:51 PM

Thanks again for everyone's help.