Page MenuHomePhabricator
Create Task
Maniphest T232486

Unable to ssh to ws-web.wikistream.eqiad.wmflabs as user edsu
Closed, ResolvedPublic
Actions

Description

I'm an admin on the wikistream project and am unable to ssh to ws-web to debug why the service is currently unavailable.

edsu@bastion-eqiad1-01:~$ ssh ws-web
Permission denied (publickey).

Related Objects

Event Timeline

edsu created this task.Sep 10 2019, 2:45 PM
Restricted Application added a subscriber: Aklapper. · View Herald TranscriptSep 10 2019, 2:45 PM
Reedy added a project: Cloud-VPS.Sep 10 2019, 2:47 PM
Aklapper renamed this task from Unable to ssh to ws-web to Unable to ssh to ws-web as an admin.Sep 10 2019, 2:48 PM
Aklapper added a comment.Sep 10 2019, 2:50 PM

@edsu: Could you run the command with verbose debug output please? (somewhere between ssh -v and ssh -vvv)

edsu added a comment.Sep 10 2019, 3:08 PM

Thanks for the quick response! Here you go:

edsu@bastion-eqiad1-01:~$ ssh -v ws-web
OpenSSH_7.4p1 Debian-10+deb9u6, OpenSSL 1.0.2s  28 May 2019
debug1: Reading configuration data /etc/ssh/ssh_config
debug1: /etc/ssh/ssh_config line 19: Applying options for *
debug1: Connecting to ws-web [172.16.6.146] port 22.
debug1: Connection established.
debug1: key_load_public: No such file or directory
debug1: identity file /home/edsu/.ssh/id_rsa type -1
debug1: key_load_public: No such file or directory
debug1: identity file /home/edsu/.ssh/id_rsa-cert type -1
debug1: key_load_public: No such file or directory
debug1: identity file /home/edsu/.ssh/id_dsa type -1
debug1: key_load_public: No such file or directory
debug1: identity file /home/edsu/.ssh/id_dsa-cert type -1
debug1: key_load_public: No such file or directory
debug1: identity file /home/edsu/.ssh/id_ecdsa type -1
debug1: key_load_public: No such file or directory
debug1: identity file /home/edsu/.ssh/id_ecdsa-cert type -1
debug1: key_load_public: No such file or directory
debug1: identity file /home/edsu/.ssh/id_ed25519 type -1
debug1: key_load_public: No such file or directory
debug1: identity file /home/edsu/.ssh/id_ed25519-cert type -1
debug1: Enabling compatibility mode for protocol 2.0
debug1: Local version string SSH-2.0-OpenSSH_7.4p1 Debian-10+deb9u6
debug1: Remote protocol version 2.0, remote software version OpenSSH_6.7p1 Debian-5+deb8u8
debug1: match: OpenSSH_6.7p1 Debian-5+deb8u8 pat OpenSSH* compat 0x04000000
debug1: Authenticating to ws-web:22 as 'edsu'
debug1: SSH2_MSG_KEXINIT sent
debug1: SSH2_MSG_KEXINIT received
debug1: kex: algorithm: curve25519-sha256@libssh.org
debug1: kex: host key algorithm: ecdsa-sha2-nistp256
debug1: kex: server->client cipher: chacha20-poly1305@openssh.com MAC: <implicit> compression: none
debug1: kex: client->server cipher: chacha20-poly1305@openssh.com MAC: <implicit> compression: none
debug1: expecting SSH2_MSG_KEX_ECDH_REPLY
debug1: Server host key: ecdsa-sha2-nistp256 SHA256:zYBIo3+IoJW4Jm13gUDO8YKvBVzivJ1MC7+Huzdo6hM
debug1: Host 'ws-web' is known and matches the ECDSA host key.
debug1: Found key in /home/edsu/.ssh/known_hosts:1
debug1: rekey after 134217728 blocks
debug1: SSH2_MSG_NEWKEYS sent
debug1: expecting SSH2_MSG_NEWKEYS
debug1: SSH2_MSG_NEWKEYS received
debug1: rekey after 134217728 blocks
debug1: SSH2_MSG_SERVICE_ACCEPT received
debug1: Authentications that can continue: publickey
debug1: Next authentication method: publickey
debug1: Trying private key: /home/edsu/.ssh/id_rsa
debug1: Trying private key: /home/edsu/.ssh/id_dsa
debug1: Trying private key: /home/edsu/.ssh/id_ecdsa
debug1: Trying private key: /home/edsu/.ssh/id_ed25519
debug1: No more authentication methods to try.
Permission denied (publickey).
edsu added a comment.Sep 10 2019, 3:37 PM

Hmm, looking at the docs perhaps it's no longer possible to ssh to bastion.wmflabs.org and then ssh on to ws-web? Should I follow the instructions at this page to set up bast1002.wikimedia.org as a proxy?

bd808 claimed this task.Sep 10 2019, 9:43 PM
bd808 triaged this task as Medium priority.
bd808 added a project: cloud-services-team (Kanban).
bd808 subscribed.
In T232486#5479457, @edsu wrote:

Hmm, looking at the docs perhaps it's no longer possible to ssh to bastion.wmflabs.org and then ssh on to ws-web? Should I follow the instructions at this page to set up bast1002.wikimedia.org as a proxy?

Bast1002 is a bastion for accessing hosts inside the production network at Wikimedia rather than virtual instances in Cloud VPS. The bastion-eqiad1-01.bastion.eqiad.wmflabs (also known as bastion.wmflabs.org) is the correct bastion to use to access instances in the wikistream Cloud VPS project.

I will poke around in the log files on ws-web.wikistream.eqiad.wmflabs to see if I can find a reason for the auth failure.

bd808 moved this task from Inbox to Clinic Duty on the cloud-services-team (Kanban) board.Sep 10 2019, 9:43 PM
• srodlund added a project: Documentation.Sep 10 2019, 10:19 PM
• srodlund subscribed.
bd808 added a comment.Sep 10 2019, 10:20 PM
/var/log/auth
Sep 10 15:08:11 ws-web sshd[17813]: Set /proc/self/oom_score_adj to 0
Sep 10 15:08:11 ws-web sshd[17813]: Connection from 172.16.1.136 port 43756 on 172.16.6.146 port 22
Sep 10 15:08:11 ws-web sshd[17813]: Connection closed by 172.16.1.136 [preauth]

Not super helpful logging. I did see some logging in a vaguely similar time period about nslcd reloading. That service brokers lookups of your ssh public key in LDAP among other things. It is possible that you just had really bad luck and the
instance was unable to talk with the LDAP server at the time you tried to ssh in.

@edsu Could you try logging in again? It would be helpful to have a ssh -vvv output if you are still not able to connect successfully. At that level of debug output hopefully we will be able to see the the fingerprint(s) of the ssh public keys that your ssh-agent is offering so we can check them against the keys you have authorized through wikitech/toolsadmin to access your account. If that still doesn't get us anywhere we may have to sync up for live debugging on the #wikimedia-cloud Freenode IRC channel where I watch the logs in various places as you try additional logins.

bd808 added a comment.Sep 10 2019, 10:38 PM

Trying to be helpful while we work on the ssh issue, I believe I have gotten the wikistream service back up and running for the moment.

[22:25]  <    bd808>	!log wikistream Restarted varnish (T232486)
[22:34]  <    bd808>	!log wikistream Manually started `HOME=/var/tmp/ NODE_ENV=production /usr/bin/nodejs /opt/wikistream/app.js >/dev/null 2>&1 &` (T232486)
[22:35]  <    bd808>	crap. stashbot never came back because k8s is sick

The /etc/init/wikistream.conf upstart process really needs to be converted to a systemd unit to make this easier. Some docs on https://wikitech.wikimedia.org/wiki/Nova_Resource:Wikistream about how to restart both varnish and the node process would be extra nice.

• srodlund mentioned this in T232547: Create docs on how to restart both varnish and the node process for wikistream.Sep 10 2019, 10:52 PM

I created task for this and added it to the documentation workboard: https://phabricator.wikimedia.org/T232547

bd808 renamed this task from Unable to ssh to ws-web as an admin to Unable to ssh to ws-web.wikistream.eqiad.wmflabs as user edsu.Sep 10 2019, 11:21 PM
bd808 added a project: VPS-Projects.
edsu added a comment.Sep 10 2019, 11:39 PM

Thanks for getting wikistream back working again. I'm still unable to ssh in however.

edsu@bastion-eqiad1-01:~$ ssh -vvvv ws-web
OpenSSH_7.4p1 Debian-10+deb9u6, OpenSSL 1.0.2s  28 May 2019
debug1: Reading configuration data /etc/ssh/ssh_config
debug1: /etc/ssh/ssh_config line 19: Applying options for *
debug2: resolving "ws-web" port 22
debug2: ssh_connect_direct: needpriv 0
debug1: Connecting to ws-web [172.16.6.146] port 22.
debug1: Connection established.
debug1: key_load_public: No such file or directory
debug1: identity file /home/edsu/.ssh/id_rsa type -1
debug1: key_load_public: No such file or directory
debug1: identity file /home/edsu/.ssh/id_rsa-cert type -1
debug1: key_load_public: No such file or directory
debug1: identity file /home/edsu/.ssh/id_dsa type -1
debug1: key_load_public: No such file or directory
debug1: identity file /home/edsu/.ssh/id_dsa-cert type -1
debug1: key_load_public: No such file or directory
debug1: identity file /home/edsu/.ssh/id_ecdsa type -1
debug1: key_load_public: No such file or directory
debug1: identity file /home/edsu/.ssh/id_ecdsa-cert type -1
debug1: key_load_public: No such file or directory
debug1: identity file /home/edsu/.ssh/id_ed25519 type -1
debug1: key_load_public: No such file or directory
debug1: identity file /home/edsu/.ssh/id_ed25519-cert type -1
debug1: Enabling compatibility mode for protocol 2.0
debug1: Local version string SSH-2.0-OpenSSH_7.4p1 Debian-10+deb9u6
debug1: Remote protocol version 2.0, remote software version OpenSSH_6.7p1 Debian-5+deb8u8
debug1: match: OpenSSH_6.7p1 Debian-5+deb8u8 pat OpenSSH* compat 0x04000000
debug2: fd 3 setting O_NONBLOCK
debug1: Authenticating to ws-web:22 as 'edsu'
debug3: hostkeys_foreach: reading file "/home/edsu/.ssh/known_hosts"
debug3: record_hostkey: found key type ECDSA in file /home/edsu/.ssh/known_hosts:1
debug3: load_hostkeys: loaded 1 keys from ws-web
debug3: order_hostkeyalgs: prefer hostkeyalgs: ecdsa-sha2-nistp256-cert-v01@openssh.com,ecdsa-sha2-nistp384-cert-v01@openssh.com,ecdsa-sha2-nistp521-cert-v01@openssh.com,ecdsa-sha2-nistp256,ecdsa-sha2-nistp384,ecdsa-sha2-nistp521
debug3: send packet: type 20
debug1: SSH2_MSG_KEXINIT sent
debug3: receive packet: type 20
debug1: SSH2_MSG_KEXINIT received
debug2: local client KEXINIT proposal
debug2: KEX algorithms: curve25519-sha256,curve25519-sha256@libssh.org,ecdh-sha2-nistp256,ecdh-sha2-nistp384,ecdh-sha2-nistp521,diffie-hellman-group-exchange-sha256,diffie-hellman-group16-sha512,diffie-hellman-group18-sha512,diffie-hellman-group-exchange-sha1,diffie-hellman-group14-sha256,diffie-hellman-group14-sha1,ext-info-c
debug2: host key algorithms: ecdsa-sha2-nistp256-cert-v01@openssh.com,ecdsa-sha2-nistp384-cert-v01@openssh.com,ecdsa-sha2-nistp521-cert-v01@openssh.com,ecdsa-sha2-nistp256,ecdsa-sha2-nistp384,ecdsa-sha2-nistp521,ssh-ed25519-cert-v01@openssh.com,ssh-rsa-cert-v01@openssh.com,ssh-ed25519,rsa-sha2-512,rsa-sha2-256,ssh-rsa
debug2: ciphers ctos: chacha20-poly1305@openssh.com,aes128-ctr,aes192-ctr,aes256-ctr,aes128-gcm@openssh.com,aes256-gcm@openssh.com,aes128-cbc,aes192-cbc,aes256-cbc
debug2: ciphers stoc: chacha20-poly1305@openssh.com,aes128-ctr,aes192-ctr,aes256-ctr,aes128-gcm@openssh.com,aes256-gcm@openssh.com,aes128-cbc,aes192-cbc,aes256-cbc
debug2: MACs ctos: umac-64-etm@openssh.com,umac-128-etm@openssh.com,hmac-sha2-256-etm@openssh.com,hmac-sha2-512-etm@openssh.com,hmac-sha1-etm@openssh.com,umac-64@openssh.com,umac-128@openssh.com,hmac-sha2-256,hmac-sha2-512,hmac-sha1
debug2: MACs stoc: umac-64-etm@openssh.com,umac-128-etm@openssh.com,hmac-sha2-256-etm@openssh.com,hmac-sha2-512-etm@openssh.com,hmac-sha1-etm@openssh.com,umac-64@openssh.com,umac-128@openssh.com,hmac-sha2-256,hmac-sha2-512,hmac-sha1
debug2: compression ctos: none,zlib@openssh.com,zlib
debug2: compression stoc: none,zlib@openssh.com,zlib
debug2: languages ctos: 
debug2: languages stoc: 
debug2: first_kex_follows 0 
debug2: reserved 0 
debug2: peer server KEXINIT proposal
debug2: KEX algorithms: curve25519-sha256@libssh.org,diffie-hellman-group-exchange-sha256
debug2: host key algorithms: ssh-rsa,ssh-dss,ecdsa-sha2-nistp256,ssh-ed25519
debug2: ciphers ctos: chacha20-poly1305@openssh.com,aes256-gcm@openssh.com,aes128-gcm@openssh.com,aes256-ctr,aes192-ctr,aes128-ctr
debug2: ciphers stoc: chacha20-poly1305@openssh.com,aes256-gcm@openssh.com,aes128-gcm@openssh.com,aes256-ctr,aes192-ctr,aes128-ctr
debug2: MACs ctos: hmac-sha2-512-etm@openssh.com,hmac-sha2-256-etm@openssh.com,umac-128-etm@openssh.com,hmac-sha2-512,hmac-sha2-256,umac-128@openssh.com
debug2: MACs stoc: hmac-sha2-512-etm@openssh.com,hmac-sha2-256-etm@openssh.com,umac-128-etm@openssh.com,hmac-sha2-512,hmac-sha2-256,umac-128@openssh.com
debug2: compression ctos: none,zlib@openssh.com
debug2: compression stoc: none,zlib@openssh.com
debug2: languages ctos: 
debug2: languages stoc: 
debug2: first_kex_follows 0 
debug2: reserved 0 
debug1: kex: algorithm: curve25519-sha256@libssh.org
debug1: kex: host key algorithm: ecdsa-sha2-nistp256
debug1: kex: server->client cipher: chacha20-poly1305@openssh.com MAC: <implicit> compression: none
debug1: kex: client->server cipher: chacha20-poly1305@openssh.com MAC: <implicit> compression: none
debug3: send packet: type 30
debug1: expecting SSH2_MSG_KEX_ECDH_REPLY
debug3: receive packet: type 31
debug1: Server host key: ecdsa-sha2-nistp256 SHA256:zYBIo3+IoJW4Jm13gUDO8YKvBVzivJ1MC7+Huzdo6hM
debug3: hostkeys_foreach: reading file "/home/edsu/.ssh/known_hosts"
debug3: record_hostkey: found key type ECDSA in file /home/edsu/.ssh/known_hosts:1
debug3: load_hostkeys: loaded 1 keys from ws-web
debug3: hostkeys_foreach: reading file "/home/edsu/.ssh/known_hosts"
debug3: record_hostkey: found key type ECDSA in file /home/edsu/.ssh/known_hosts:2
debug3: load_hostkeys: loaded 1 keys from 172.16.6.146
debug1: Host 'ws-web' is known and matches the ECDSA host key.
debug1: Found key in /home/edsu/.ssh/known_hosts:1
debug3: send packet: type 21
debug2: set_newkeys: mode 1
debug1: rekey after 134217728 blocks
debug1: SSH2_MSG_NEWKEYS sent
debug1: expecting SSH2_MSG_NEWKEYS
debug3: receive packet: type 21
debug1: SSH2_MSG_NEWKEYS received
debug2: set_newkeys: mode 0
debug1: rekey after 134217728 blocks
debug2: key: /home/edsu/.ssh/id_rsa ((nil))
debug2: key: /home/edsu/.ssh/id_dsa ((nil))
debug2: key: /home/edsu/.ssh/id_ecdsa ((nil))
debug2: key: /home/edsu/.ssh/id_ed25519 ((nil))
debug3: send packet: type 5
debug3: receive packet: type 6
debug2: service_accept: ssh-userauth
debug1: SSH2_MSG_SERVICE_ACCEPT received
debug3: send packet: type 50
debug3: receive packet: type 51
debug1: Authentications that can continue: publickey
debug3: start over, passed a different list publickey
debug3: preferred gssapi-keyex,gssapi-with-mic,publickey,keyboard-interactive,password
debug3: authmethod_lookup publickey
debug3: remaining preferred: keyboard-interactive,password
debug3: authmethod_is_enabled publickey
debug1: Next authentication method: publickey
debug1: Trying private key: /home/edsu/.ssh/id_rsa
debug3: no such identity: /home/edsu/.ssh/id_rsa: No such file or directory
debug1: Trying private key: /home/edsu/.ssh/id_dsa
debug3: no such identity: /home/edsu/.ssh/id_dsa: No such file or directory
debug1: Trying private key: /home/edsu/.ssh/id_ecdsa
debug3: no such identity: /home/edsu/.ssh/id_ecdsa: No such file or directory
debug1: Trying private key: /home/edsu/.ssh/id_ed25519
debug3: no such identity: /home/edsu/.ssh/id_ed25519: No such file or directory
debug2: we did not send a packet, disable method
debug1: No more authentication methods to try.
Permission denied (publickey).
bd808 added a comment.Sep 10 2019, 11:55 PM
In T232486#5481574, @edsu wrote:

Thanks for getting wikistream back working again. I'm still unable to ssh in however.

edsu@bastion-eqiad1-01:~$ ssh -vvvv ws-web
...snip...
debug3: authmethod_is_enabled publickey
debug1: Next authentication method: publickey
debug1: Trying private key: /home/edsu/.ssh/id_rsa
debug3: no such identity: /home/edsu/.ssh/id_rsa: No such file or directory
debug1: Trying private key: /home/edsu/.ssh/id_dsa
debug3: no such identity: /home/edsu/.ssh/id_dsa: No such file or directory
debug1: Trying private key: /home/edsu/.ssh/id_ecdsa
debug3: no such identity: /home/edsu/.ssh/id_ecdsa: No such file or directory
debug1: Trying private key: /home/edsu/.ssh/id_ed25519
debug3: no such identity: /home/edsu/.ssh/id_ed25519: No such file or directory
debug2: we did not send a packet, disable method
debug1: No more authentication methods to try.
Permission denied (publickey).

It appears that you are doing an ssh from your personal computer to bastion.wmflabs.org, and then from the shell on bastion.wmflabs.org trying to ssh to ws-web.wikistream.eqiad.wmflabs. This would only work if you are using ssh agent forwarding so that the ssh keys on your personal computer are available to the ssh process on bastion.wmflabs.org or you have ssh private keys stored in $HOME/.ssh/ on bastion.wmflabs.org. Based on the -vvv output you gave I believe neither is the case. That's actually not horrible as neither practice would be ideal. Agent forwarding has a potential to expose your ssh agent to root users on the intermediate host or worse to anyone who finds a local root exploit on those same hosts. Storing ssh private keys on any Cloud VPS instance, whether it is a bastion or not, is not a great idea either for the same reason of potential exposure of the private key to anyone with authorized or unauthorized root access to the filesystem.

The "best practice" method of accessing instances via ssh is by using the bastions as a "jump host" rather than with an ssh-in-ssh session. https://wikitech.wikimedia.org/wiki/Help:Access#Accessing_instances_with_ProxyJump_ssh_option_(recommended) documents the easiest way to set this up if you have a modern openssh client. There are also instructions there for older openssh clients which implement the ProxyCommand directive but not the ProxyJump directive. If you are using PuTTY on windows, see https://wikitech.wikimedia.org/wiki/Help:Access_to_Cloud_VPS_instances_with_PuTTY_and_WinSCP#How_to_set_up_PuTTY_for_proxying_through_bastion.wmflabs.org_to_your_instance

edsu added a comment.Sep 11 2019, 2:51 PM

@bd808 thank you! I only was doing the ssh-within-ssh because it worked for me in the past. But I've got ProxyJump working fine now, and can access ws-web, so I appreciate you taking the time to explain this solution.

edsu closed this task as Resolved.Sep 11 2019, 2:51 PM

Thanks again for everyone's help.

edsu reopened this task as Open.Oct 12 2019, 2:30 PM

I'm sorry to have to reopen this. But I've been notified that wikistream.wmflabs.org is down again and I no longer seem to ssh with the the ProxyJump configuration? Here's the log of my ssh attempt:

kaizen:~ edsu$ ssh -vvv edsu@ws-web.wmflabs
OpenSSH_7.9p1, LibreSSL 2.7.3
debug1: Reading configuration data /Users/edsu/.ssh/config
debug1: /Users/edsu/.ssh/config line 1: Applying options for *.wmflabs
debug1: Reading configuration data /etc/ssh/ssh_config
debug1: /etc/ssh/ssh_config line 48: Applying options for *
debug1: Setting implicit ProxyCommand from ProxyJump: ssh -p 22 -vvv -W '[%h]:%p' primary.bastion.wmflabs.org
debug1: Executing proxy command: exec ssh -p 22 -vvv -W '[ws-web.wmflabs]:22' primary.bastion.wmflabs.org
debug1: identity file /Users/edsu/.ssh/id_rsa type 0
debug1: identity file /Users/edsu/.ssh/id_rsa-cert type -1
debug1: identity file /Users/edsu/.ssh/id_dsa type -1
debug1: identity file /Users/edsu/.ssh/id_dsa-cert type -1
debug1: identity file /Users/edsu/.ssh/id_ecdsa type -1
debug1: identity file /Users/edsu/.ssh/id_ecdsa-cert type -1
debug1: identity file /Users/edsu/.ssh/id_ed25519 type -1
debug1: identity file /Users/edsu/.ssh/id_ed25519-cert type -1
debug1: identity file /Users/edsu/.ssh/id_xmss type -1
debug1: identity file /Users/edsu/.ssh/id_xmss-cert type -1
debug1: Local version string SSH-2.0-OpenSSH_7.9
OpenSSH_7.9p1, LibreSSL 2.7.3
debug1: Reading configuration data /Users/edsu/.ssh/config
debug1: /Users/edsu/.ssh/config line 5: Applying options for primary.bastion.wmflabs.org
debug1: Reading configuration data /etc/ssh/ssh_config
debug1: /etc/ssh/ssh_config line 48: Applying options for *
debug1: Connecting to primary.bastion.wmflabs.org port 22.
debug1: Connection established.
debug1: identity file /Users/edsu/.ssh/id_rsa type 0
debug1: identity file /Users/edsu/.ssh/id_rsa-cert type -1
debug1: identity file /Users/edsu/.ssh/id_dsa type -1
debug1: identity file /Users/edsu/.ssh/id_dsa-cert type -1
debug1: identity file /Users/edsu/.ssh/id_ecdsa type -1
debug1: identity file /Users/edsu/.ssh/id_ecdsa-cert type -1
debug1: identity file /Users/edsu/.ssh/id_ed25519 type -1
debug1: identity file /Users/edsu/.ssh/id_ed25519-cert type -1
debug1: identity file /Users/edsu/.ssh/id_xmss type -1
debug1: identity file /Users/edsu/.ssh/id_xmss-cert type -1
debug1: Local version string SSH-2.0-OpenSSH_7.9
debug1: Remote protocol version 2.0, remote software version OpenSSH_7.4p1 Debian-10+deb9u6
debug1: match: OpenSSH_7.4p1 Debian-10+deb9u6 pat OpenSSH_7.0*,OpenSSH_7.1*,OpenSSH_7.2*,OpenSSH_7.3*,OpenSSH_7.4*,OpenSSH_7.5*,OpenSSH_7.6*,OpenSSH_7.7* compat 0x04000002
debug3: fd 5 is O_NONBLOCK
debug1: Authenticating to primary.bastion.wmflabs.org:22 as 'edsu'
debug3: hostkeys_foreach: reading file "/Users/edsu/.ssh/known_hosts"
debug3: record_hostkey: found key type ECDSA in file /Users/edsu/.ssh/known_hosts:94
debug3: load_hostkeys: loaded 1 keys from primary.bastion.wmflabs.org
debug3: order_hostkeyalgs: prefer hostkeyalgs: ecdsa-sha2-nistp256-cert-v01@openssh.com,ecdsa-sha2-nistp384-cert-v01@openssh.com,ecdsa-sha2-nistp521-cert-v01@openssh.com,ecdsa-sha2-nistp256,ecdsa-sha2-nistp384,ecdsa-sha2-nistp521
debug3: send packet: type 20
debug1: SSH2_MSG_KEXINIT sent
debug3: receive packet: type 20
debug1: SSH2_MSG_KEXINIT received
debug2: local client KEXINIT proposal
debug2: KEX algorithms: curve25519-sha256,curve25519-sha256@libssh.org,ecdh-sha2-nistp256,ecdh-sha2-nistp384,ecdh-sha2-nistp521,diffie-hellman-group-exchange-sha256,diffie-hellman-group16-sha512,diffie-hellman-group18-sha512,diffie-hellman-group14-sha256,diffie-hellman-group14-sha1,ext-info-c
debug2: host key algorithms: ecdsa-sha2-nistp256-cert-v01@openssh.com,ecdsa-sha2-nistp384-cert-v01@openssh.com,ecdsa-sha2-nistp521-cert-v01@openssh.com,ecdsa-sha2-nistp256,ecdsa-sha2-nistp384,ecdsa-sha2-nistp521,ssh-ed25519-cert-v01@openssh.com,rsa-sha2-512-cert-v01@openssh.com,rsa-sha2-256-cert-v01@openssh.com,ssh-rsa-cert-v01@openssh.com,ssh-ed25519,rsa-sha2-512,rsa-sha2-256,ssh-rsa
debug2: ciphers ctos: chacha20-poly1305@openssh.com,aes128-ctr,aes192-ctr,aes256-ctr,aes128-gcm@openssh.com,aes256-gcm@openssh.com
debug2: ciphers stoc: chacha20-poly1305@openssh.com,aes128-ctr,aes192-ctr,aes256-ctr,aes128-gcm@openssh.com,aes256-gcm@openssh.com
debug2: MACs ctos: umac-64-etm@openssh.com,umac-128-etm@openssh.com,hmac-sha2-256-etm@openssh.com,hmac-sha2-512-etm@openssh.com,hmac-sha1-etm@openssh.com,umac-64@openssh.com,umac-128@openssh.com,hmac-sha2-256,hmac-sha2-512,hmac-sha1
debug2: MACs stoc: umac-64-etm@openssh.com,umac-128-etm@openssh.com,hmac-sha2-256-etm@openssh.com,hmac-sha2-512-etm@openssh.com,hmac-sha1-etm@openssh.com,umac-64@openssh.com,umac-128@openssh.com,hmac-sha2-256,hmac-sha2-512,hmac-sha1
debug2: compression ctos: none,zlib@openssh.com,zlib
debug2: compression stoc: none,zlib@openssh.com,zlib
debug2: languages ctos: 
debug2: languages stoc: 
debug2: first_kex_follows 0 
debug2: reserved 0 
debug2: peer server KEXINIT proposal
debug2: KEX algorithms: curve25519-sha256,curve25519-sha256@libssh.org,ecdh-sha2-nistp256,ecdh-sha2-nistp384,ecdh-sha2-nistp521,diffie-hellman-group-exchange-sha256,diffie-hellman-group16-sha512,diffie-hellman-group18-sha512,diffie-hellman-group14-sha256,diffie-hellman-group14-sha1
debug2: host key algorithms: ssh-rsa,rsa-sha2-512,rsa-sha2-256,ecdsa-sha2-nistp256,ssh-ed25519
debug2: ciphers ctos: chacha20-poly1305@openssh.com,aes256-gcm@openssh.com,aes128-gcm@openssh.com,aes256-ctr,aes192-ctr,aes128-ctr
debug2: ciphers stoc: chacha20-poly1305@openssh.com,aes256-gcm@openssh.com,aes128-gcm@openssh.com,aes256-ctr,aes192-ctr,aes128-ctr
debug2: MACs ctos: umac-64-etm@openssh.com,umac-128-etm@openssh.com,hmac-sha2-256-etm@openssh.com,hmac-sha2-512-etm@openssh.com,hmac-sha1-etm@openssh.com,umac-64@openssh.com,umac-128@openssh.com,hmac-sha2-256,hmac-sha2-512,hmac-sha1
debug2: MACs stoc: umac-64-etm@openssh.com,umac-128-etm@openssh.com,hmac-sha2-256-etm@openssh.com,hmac-sha2-512-etm@openssh.com,hmac-sha1-etm@openssh.com,umac-64@openssh.com,umac-128@openssh.com,hmac-sha2-256,hmac-sha2-512,hmac-sha1
debug2: compression ctos: none,zlib@openssh.com
debug2: compression stoc: none,zlib@openssh.com
debug2: languages ctos: 
debug2: languages stoc: 
debug2: first_kex_follows 0 
debug2: reserved 0 
debug1: kex: algorithm: curve25519-sha256
debug1: kex: host key algorithm: ecdsa-sha2-nistp256
debug1: kex: server->client cipher: chacha20-poly1305@openssh.com MAC: <implicit> compression: none
debug1: kex: client->server cipher: chacha20-poly1305@openssh.com MAC: <implicit> compression: none
debug3: send packet: type 30
debug1: expecting SSH2_MSG_KEX_ECDH_REPLY
debug3: receive packet: type 31
debug1: Server host key: ecdsa-sha2-nistp256 SHA256:s+xuLo91PcVIFcFdxPQC7IXgJ2nYxaXcqa7bKE7/ufA
debug3: hostkeys_foreach: reading file "/Users/edsu/.ssh/known_hosts"
debug3: record_hostkey: found key type ECDSA in file /Users/edsu/.ssh/known_hosts:94
debug3: load_hostkeys: loaded 1 keys from primary.bastion.wmflabs.org
debug3: hostkeys_foreach: reading file "/Users/edsu/.ssh/known_hosts"
debug3: record_hostkey: found key type ECDSA in file /Users/edsu/.ssh/known_hosts:92
debug3: load_hostkeys: loaded 1 keys from 185.15.56.13
debug1: Host 'primary.bastion.wmflabs.org' is known and matches the ECDSA host key.
debug1: Found key in /Users/edsu/.ssh/known_hosts:94
debug3: send packet: type 21
debug2: set_newkeys: mode 1
debug1: rekey after 134217728 blocks
debug1: SSH2_MSG_NEWKEYS sent
debug1: expecting SSH2_MSG_NEWKEYS
debug3: receive packet: type 21
debug1: SSH2_MSG_NEWKEYS received
debug2: set_newkeys: mode 0
debug1: rekey after 134217728 blocks
debug1: Will attempt key: /Users/edsu/.ssh/id_rsa RSA SHA256:8wr5fCskfroOegit499fhgCADc/0otkzqy7rHgT+1FQ
debug1: Will attempt key: /Users/edsu/.ssh/id_dsa 
debug1: Will attempt key: /Users/edsu/.ssh/id_ecdsa 
debug1: Will attempt key: /Users/edsu/.ssh/id_ed25519 
debug1: Will attempt key: /Users/edsu/.ssh/id_xmss 
debug2: pubkey_prepare: done
debug3: send packet: type 5
debug3: receive packet: type 7
debug1: SSH2_MSG_EXT_INFO received
debug1: kex_input_ext_info: server-sig-algs=<ssh-ed25519,ssh-rsa,ssh-dss,ecdsa-sha2-nistp256,ecdsa-sha2-nistp384,ecdsa-sha2-nistp521>
debug3: receive packet: type 6
debug2: service_accept: ssh-userauth
debug1: SSH2_MSG_SERVICE_ACCEPT received
debug3: send packet: type 50
debug3: receive packet: type 51
debug1: Authentications that can continue: publickey
debug3: start over, passed a different list publickey
debug3: preferred publickey,keyboard-interactive,password
debug3: authmethod_lookup publickey
debug3: remaining preferred: keyboard-interactive,password
debug3: authmethod_is_enabled publickey
debug1: Next authentication method: publickey
debug1: Offering public key: /Users/edsu/.ssh/id_rsa RSA SHA256:8wr5fCskfroOegit499fhgCADc/0otkzqy7rHgT+1FQ
debug3: send packet: type 50
debug2: we sent a publickey packet, wait for reply
debug3: receive packet: type 60
debug1: Server accepts key: /Users/edsu/.ssh/id_rsa RSA SHA256:8wr5fCskfroOegit499fhgCADc/0otkzqy7rHgT+1FQ
debug3: sign_and_send_pubkey: RSA SHA256:8wr5fCskfroOegit499fhgCADc/0otkzqy7rHgT+1FQ
debug3: sign_and_send_pubkey: signing using ssh-rsa
debug3: send packet: type 50
debug3: receive packet: type 52
debug1: Authentication succeeded (publickey).
Authenticated to primary.bastion.wmflabs.org ([185.15.56.13]:22).
debug3: ssh_init_stdio_forwarding: ws-web.wmflabs:22
debug1: channel_connect_stdio_fwd ws-web.wmflabs:22
debug1: channel 0: new [stdio-forward]
debug2: fd 6 setting O_NONBLOCK
debug2: fd 7 setting O_NONBLOCK
debug1: getpeername failed: Bad file descriptor
debug3: send packet: type 90
debug2: fd 5 setting TCP_NODELAY
debug3: ssh_packet_set_tos: set IP_TOS 0x48
debug1: Requesting no-more-sessions@openssh.com
debug3: send packet: type 80
debug1: Entering interactive session.
debug1: pledge: network
debug3: receive packet: type 80
debug1: client_input_global_request: rtype hostkeys-00@openssh.com want_reply 0
debug3: receive packet: type 92
channel 0: open failed: administratively prohibited: open failed
debug2: channel_input_open_failure: channel 0: callback start
stdio forwarding failed
ssh_exchange_identification: Connection closed by remote host
Krenair subscribed.EditedOct 12 2019, 2:36 PM
In T232486#5569798, @edsu wrote:

I'm sorry to have to reopen this. But I've been notified that wikistream.wmflabs.org is down again and I no longer seem to ssh with the the ProxyJump configuration? Here's the log of my ssh attempt:

kaizen:~ edsu$ ssh -vvv edsu@ws-web.wmflabs

ws-web.wmflabs cannot work (without some extra SSH config anyway), you want ws-web.wikistream.eqiad.wmflabs.

edsu closed this task as Resolved.Oct 12 2019, 2:43 PM

Ahh, yes that works for me. Thank you for clarifying!

Content licensed under Creative Commons Attribution-ShareAlike (CC BY-SA) 4.0 unless otherwise noted; code licensed under GNU General Public License (GPL) 2.0 or later and other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL