Hi, today i tryed to ssh into puppet-paladox, but it failed with Connection closed by UNKNOWN port 65535 then i tryed with puppet-phabricator and the same problem happened.
Another user reported on irc they are having the same problem (but not sure if they have puppetmaster on there instance)
So this issue may be a recent change to puppetmaster or something else
Hitting the issue as well, no puppetmaster on my instance.
ssh configuration as per https://wikitech.wikimedia.org/wiki/Help:Access#Accessing_instances_with_ProxyCommand_ssh_option_(recommended)
Pasting full debug output below.
$ ssh -vvv pst.eqiad.wmflabs OpenSSH_7.6p1, LibreSSL 2.6.2 debug1: Reading configuration data /Users/focs/.ssh/config debug1: /Users/focs/.ssh/config line 1: Applying options for * debug1: /Users/focs/.ssh/config line 2: Deprecated option "useroaming" debug1: /Users/focs/.ssh/config line 4: Applying options for *.eqiad.wmflabs debug1: /Users/focs/.ssh/config line 7: Applying options for *.wmflabs debug1: /Users/focs/.ssh/config line 10: Applying options for * debug3: kex names ok: [curve25519-sha256@libssh.org,diffie-hellman-group-exchange-sha256] debug1: Reading configuration data /etc/ssh/ssh_config debug1: /etc/ssh/ssh_config line 48: Applying options for * debug1: Executing proxy command: exec ssh -a -W pst.eqiad.wmflabs:22 hjfocs@primary.bastion.wmflabs.org debug1: identity file /Users/focs/.ssh/id_rsa type 0 debug1: key_load_public: No such file or directory debug1: identity file /Users/focs/.ssh/id_rsa-cert type -1 debug1: key_load_public: No such file or directory debug1: identity file /Users/focs/.ssh/id_dsa type -1 debug1: key_load_public: No such file or directory debug1: identity file /Users/focs/.ssh/id_dsa-cert type -1 debug1: permanently_drop_suid: 501 debug1: key_load_public: No such file or directory debug1: identity file /Users/focs/.ssh/id_ecdsa type -1 debug1: key_load_public: No such file or directory debug1: identity file /Users/focs/.ssh/id_ecdsa-cert type -1 debug1: key_load_public: No such file or directory debug1: identity file /Users/focs/.ssh/id_ed25519 type -1 debug1: key_load_public: No such file or directory debug1: identity file /Users/focs/.ssh/id_ed25519-cert type -1 debug1: Local version string SSH-2.0-OpenSSH_7.6 debug1: Remote protocol version 2.0, remote software version OpenSSH_6.7p1 Debian-5+deb8u4 debug1: match: OpenSSH_6.7p1 Debian-5+deb8u4 pat OpenSSH* compat 0x04000000 debug2: fd 5 setting O_NONBLOCK debug2: fd 4 setting O_NONBLOCK debug1: Authenticating to pst.eqiad.wmflabs:22 as 'hjfocs' debug3: hostkeys_foreach: reading file "/Users/focs/.ssh/known_hosts" debug3: record_hostkey: found key type ECDSA in file /Users/focs/.ssh/known_hosts:5 debug3: load_hostkeys: loaded 1 keys from pst.eqiad.wmflabs debug3: order_hostkeyalgs: prefer hostkeyalgs: ecdsa-sha2-nistp256-cert-v01@openssh.com,ecdsa-sha2-nistp384-cert-v01@openssh.com,ecdsa-sha2-nistp521-cert-v01@openssh.com,ecdsa-sha2-nistp256,ecdsa-sha2-nistp384,ecdsa-sha2-nistp521 debug3: send packet: type 20 debug1: SSH2_MSG_KEXINIT sent debug3: receive packet: type 20 debug1: SSH2_MSG_KEXINIT received debug2: local client KEXINIT proposal debug2: KEX algorithms: curve25519-sha256@libssh.org,diffie-hellman-group-exchange-sha256,ext-info-c debug2: host key algorithms: ecdsa-sha2-nistp256-cert-v01@openssh.com,ecdsa-sha2-nistp384-cert-v01@openssh.com,ecdsa-sha2-nistp521-cert-v01@openssh.com,ecdsa-sha2-nistp256,ecdsa-sha2-nistp384,ecdsa-sha2-nistp521,ssh-ed25519-cert-v01@openssh.com,ssh-rsa-cert-v01@openssh.com,ssh-ed25519,rsa-sha2-512,rsa-sha2-256,ssh-rsa debug2: ciphers ctos: chacha20-poly1305@openssh.com,aes256-gcm@openssh.com,aes128-gcm@openssh.com,aes256-ctr,aes192-ctr,aes128-ctr debug2: ciphers stoc: chacha20-poly1305@openssh.com,aes256-gcm@openssh.com,aes128-gcm@openssh.com,aes256-ctr,aes192-ctr,aes128-ctr debug2: MACs ctos: umac-64-etm@openssh.com,umac-128-etm@openssh.com,hmac-sha2-256-etm@openssh.com,hmac-sha2-512-etm@openssh.com,hmac-sha1-etm@openssh.com,umac-64@openssh.com,umac-128@openssh.com,hmac-sha2-256,hmac-sha2-512,hmac-sha1 debug2: MACs stoc: umac-64-etm@openssh.com,umac-128-etm@openssh.com,hmac-sha2-256-etm@openssh.com,hmac-sha2-512-etm@openssh.com,hmac-sha1-etm@openssh.com,umac-64@openssh.com,umac-128@openssh.com,hmac-sha2-256,hmac-sha2-512,hmac-sha1 debug2: compression ctos: none,zlib@openssh.com,zlib debug2: compression stoc: none,zlib@openssh.com,zlib debug2: languages ctos: debug2: languages stoc: debug2: first_kex_follows 0 debug2: reserved 0 debug2: peer server KEXINIT proposal debug2: KEX algorithms: curve25519-sha256@libssh.org,diffie-hellman-group-exchange-sha256 debug2: host key algorithms: ssh-rsa,ssh-dss,ecdsa-sha2-nistp256,ssh-ed25519 debug2: ciphers ctos: chacha20-poly1305@openssh.com,aes256-gcm@openssh.com,aes128-gcm@openssh.com,aes256-ctr,aes192-ctr,aes128-ctr debug2: ciphers stoc: chacha20-poly1305@openssh.com,aes256-gcm@openssh.com,aes128-gcm@openssh.com,aes256-ctr,aes192-ctr,aes128-ctr debug2: MACs ctos: hmac-sha2-512-etm@openssh.com,hmac-sha2-256-etm@openssh.com,umac-128-etm@openssh.com,hmac-sha2-512,hmac-sha2-256,umac-128@openssh.com debug2: MACs stoc: hmac-sha2-512-etm@openssh.com,hmac-sha2-256-etm@openssh.com,umac-128-etm@openssh.com,hmac-sha2-512,hmac-sha2-256,umac-128@openssh.com debug2: compression ctos: none,zlib@openssh.com debug2: compression stoc: none,zlib@openssh.com debug2: languages ctos: debug2: languages stoc: debug2: first_kex_follows 0 debug2: reserved 0 debug1: kex: algorithm: curve25519-sha256@libssh.org debug1: kex: host key algorithm: ecdsa-sha2-nistp256 debug1: kex: server->client cipher: chacha20-poly1305@openssh.com MAC: <implicit> compression: none debug1: kex: client->server cipher: chacha20-poly1305@openssh.com MAC: <implicit> compression: none debug3: send packet: type 30 debug1: expecting SSH2_MSG_KEX_ECDH_REPLY debug3: receive packet: type 31 debug1: Server host key: ecdsa-sha2-nistp256 SHA256:d07rGf5HZpE8E2+LayLYFL8LF7CidBV9QQOM82ahJBk debug3: hostkeys_foreach: reading file "/Users/focs/.ssh/known_hosts" debug3: record_hostkey: found key type ECDSA in file /Users/focs/.ssh/known_hosts:5 debug3: load_hostkeys: loaded 1 keys from pst.eqiad.wmflabs debug1: Host 'pst.eqiad.wmflabs' is known and matches the ECDSA host key. debug1: Found key in /Users/focs/.ssh/known_hosts:5 debug3: send packet: type 21 debug2: set_newkeys: mode 1 debug1: rekey after 134217728 blocks debug1: SSH2_MSG_NEWKEYS sent debug1: expecting SSH2_MSG_NEWKEYS debug3: receive packet: type 21 debug1: SSH2_MSG_NEWKEYS received debug2: set_newkeys: mode 0 debug1: rekey after 134217728 blocks debug2: key: /Users/focs/.ssh/id_rsa (0x7f959b402190) debug2: key: /Users/focs/.ssh/id_dsa (0x0) debug2: key: /Users/focs/.ssh/id_ecdsa (0x0) debug2: key: /Users/focs/.ssh/id_ed25519 (0x0) debug3: send packet: type 5 debug3: receive packet: type 6 debug2: service_accept: ssh-userauth debug1: SSH2_MSG_SERVICE_ACCEPT received debug3: send packet: type 50 debug3: receive packet: type 51 debug1: Authentications that can continue: publickey debug3: start over, passed a different list publickey debug3: preferred publickey,keyboard-interactive,password debug3: authmethod_lookup publickey debug3: remaining preferred: keyboard-interactive,password debug3: authmethod_is_enabled publickey debug1: Next authentication method: publickey debug1: Offering public key: RSA SHA256:g/CnqlIKa6Qa0J3JQKBGRMlcjv15of/soJ4a0iOQmXg /Users/focs/.ssh/id_rsa debug3: send_pubkey_test debug3: send packet: type 50 debug2: we sent a publickey packet, wait for reply debug3: receive packet: type 60 debug1: Server accepts key: pkalg ssh-rsa blen 279 debug2: input_userauth_pk_ok: fp SHA256:g/CnqlIKa6Qa0J3JQKBGRMlcjv15of/soJ4a0iOQmXg debug3: sign_and_send_pubkey: RSA SHA256:g/CnqlIKa6Qa0J3JQKBGRMlcjv15of/soJ4a0iOQmXg debug3: send packet: type 50 Connection closed by UNKNOWN port 65535
This was caused (somehow) by https://gerrit.wikimedia.org/r/#/c/405671/ which has been reverted.
In Toolforge itself this seems to be effecting tools-puppmaster-01 and tools-clushmaster-01 at least. I don't understand why but the symptoms are the same.
re-resolving tentatively as I think this is related to a different thing that manifests in teh same way at T185574: toolsadmin removed rush from admin tool and can't find the user