Gerrit ssh host key fails on fedora39
Closed, DuplicatePublicBUG REPORT
Actions

Assigned To

None

Authored By

	Tarrow
	May 4 2024, 11:33 AM

Description

Steps to replicate the issue (include links if applicable):

Install fedora 39 (or perhaps earlier)
Add your gerrit ssh key to your ssh agent
run ssh -p 29418 <username>@gerrit.wikimedia.org

What happens?:
Bad server host key: Invalid key length

What should have happened instead?:
I should have seen the gerrit command interface

Software version (on Special:Version page; skip for WMF-hosted wikis like Wikipedia):
n/a

Other information (browser name/version, screenshots, etc.):
This seems to be related to the host key being too short for the crypto setting on this version of fedora (or maybe earlier ones too).

A "workaround" is to run sudo update-crypto-policies --set LEGACY and then it behaves as normal

Related Objects

Mentioned Here: T276486: gerrit's sshd is incompatible with RSA pubkeys + Fedora 33 clients (and future versions of OpenSSH proper)

Event Timeline

Tarrow created this task.May 4 2024, 11:33 AM

Restricted Application added a subscriber: Aklapper. · View Herald TranscriptMay 4 2024, 11:33 AM

brennen added projects: Release-Engineering-Team, collaboration-services.May 4 2024, 11:43 AM

Aklapper closed this task as a duplicate of T276486: gerrit's sshd is incompatible with RSA pubkeys + Fedora 33 clients (and future versions of OpenSSH proper).May 4 2024, 1:51 PM

I'm not informed enough to fully understand but I didn't *think* this is a dupe of T276486. I appear to experience this with my ed25519 key.

I think it would be worth double checking given that apparently is resolved but I can still reproduce this now.

I also wanted to leave a breadcrumb trail somehow to Bad server host key: Invalid key length because this didn't appear in a phabricator search when I looked.

Of course, if I'm wrong please reclose as dupe :)

The tail output of ssh -vvvv

debug1: expecting SSH2_MSG_KEX_ECDH_REPLY
debug3: receive packet: type 31
debug1: SSH2_MSG_KEX_ECDH_REPLY received
Bad server host key: Invalid key length

I can't tell whether that is the host key being too short or the kex algorithm, the ssh -vvv misses a few lines.

Here are the KEX algorithms being reported for me (Debian):

debug1: SSH2_MSG_KEXINIT received
debug2: local client KEXINIT proposal
debug2: KEX algorithms: sntrup761x25519-sha512@openssh.com,curve25519-sha256,curve25519-sha256@libssh.org,ecdh-sha2-nistp256,ecdh-sha2-nistp384,ecdh-sha2-nistp521,diffie-hellman-group-exchange-sha256,diffie-hellman-group16-sha512,diffie-hellman-group18-sha512,diffie-hellman-group14-sha256

And:

debug2: host key algorithms: ecdsa-sha2-nistp256,ecdsa-sha2-nistp384,ecdsa-sha2-nistp521

Our Gerrit runs 3.8 and the ssh service is powered by Apache SSHD 2.12.0.

And with ssh-keyscan -v -p 29418 gerrit.wikimedia.org I get:

debug1: kex: algorithm: curve25519-sha256
debug1: kex: host key algorithm: rsa-sha2-512
debug1: kex: server->client cipher: chacha20-poly1305@openssh.com MAC: <implicit> compression: none
debug1: kex: client->server cipher: chacha20-poly1305@openssh.com MAC: <implicit> compression: none

Note, the Fedora policy has some details at https://fedoraproject.org/wiki/Changes/CryptoPolicy#Scope , then it is unclear to me which part is not supported anymore and I suspect the wiki page to be outdated compared to what Fedora 39 provides.

Or maybe that is because the ssh-rsa host key is 1024 bits which is T240266.

Scrolled up further:

debug2: local client KEXINIT proposal
debug2: KEX algorithms: curve25519-sha256,curve25519-sha256@libssh.org,ecdh-sha2-nistp256,ecdh-sha2-nistp384,ecdh-sha2-nistp521,diffie-hellman-group-exchange-sha256,diffie-hellman-group14-sha256,diffie-hellman-group16-sha512,diffie-hellman-group18-sha512,ext-info-c,kex-strict-c-v00@openssh.com
debug2: host key algorithms: rsa-sha2-512-cert-v01@openssh.com,rsa-sha2-256-cert-v01@openssh.com,rsa-sha2-512,rsa-sha2-256,ssh-ed25519-cert-v01@openssh.com,ecdsa-sha2-nistp256-cert-v01@openssh.com,ecdsa-sha2-nistp384-cert-v01@openssh.com,ecdsa-sha2-nistp521-cert-v01@openssh.com,sk-ssh-ed25519-cert-v01@openssh.com,sk-ecdsa-sha2-nistp256-cert-v01@openssh.com,ssh-ed25519,ecdsa-sha2-nistp256,ecdsa-sha2-nistp384,ecdsa-sha2-nistp521,sk-ssh-ed25519@openssh.com,sk-ecdsa-sha2-nistp256@openssh.com
debug2: ciphers ctos: aes256-gcm@openssh.com,chacha20-poly1305@openssh.com,aes256-ctr,aes128-gcm@openssh.com,aes128-ctr
debug2: ciphers stoc: aes256-gcm@openssh.com,chacha20-poly1305@openssh.com,aes256-ctr,aes128-gcm@openssh.com,aes128-ctr
debug2: MACs ctos: hmac-sha2-256-etm@openssh.com,hmac-sha1-etm@openssh.com,umac-128-etm@openssh.com,hmac-sha2-512-etm@openssh.com,hmac-sha2-256,hmac-sha1,umac-128@openssh.com,hmac-sha2-512
debug2: MACs stoc: hmac-sha2-256-etm@openssh.com,hmac-sha1-etm@openssh.com,umac-128-etm@openssh.com,hmac-sha2-512-etm@openssh.com,hmac-sha2-256,hmac-sha1,umac-128@openssh.com,hmac-sha2-512
debug2: compression ctos: none,zlib@openssh.com,zlib
debug2: compression stoc: none,zlib@openssh.com,zlib
debug2: languages ctos:
debug2: languages stoc:
debug2: first_kex_follows 0
debug2: reserved 0
debug2: peer server KEXINIT proposal
debug2: KEX algorithms: curve25519-sha256,curve25519-sha256@libssh.org,curve448-sha512,ecdh-sha2-nistp521,ecdh-sha2-nistp384,ecdh-sha2-nistp256,diffie-hellman-group-exchange-sha256,diffie-hellman-group18-sha512,diffie-hellman-group17-sha512,diffie-hellman-group16-sha512,diffie-hellman-group15-sha512,diffie-hellman-group14-sha256,ext-info-s,kex-strict-s-v00@openssh.com
debug2: host key algorithms: rsa-sha2-512,rsa-sha2-256,ssh-rsa
debug2: ciphers ctos: chacha20-poly1305@openssh.com,aes128-ctr,aes192-ctr,aes256-ctr,aes128-gcm@openssh.com,aes256-gcm@openssh.com,aes128-cbc,aes192-cbc,aes256-cbc
debug2: ciphers stoc: chacha20-poly1305@openssh.com,aes128-ctr,aes192-ctr,aes256-ctr,aes128-gcm@openssh.com,aes256-gcm@openssh.com,aes128-cbc,aes192-cbc,aes256-cbc
debug2: MACs ctos: hmac-sha2-256-etm@openssh.com,hmac-sha2-512-etm@openssh.com,hmac-sha1-etm@openssh.com,hmac-sha2-256,hmac-sha2-512,hmac-sha1
debug2: MACs stoc: hmac-sha2-256-etm@openssh.com,hmac-sha2-512-etm@openssh.com,hmac-sha1-etm@openssh.com,hmac-sha2-256,hmac-sha2-512,hmac-sha1
debug2: compression ctos: none
debug2: compression stoc: none
debug2: languages ctos:
debug2: languages stoc:
debug2: first_kex_follows 0
debug2: reserved 0
debug3: kex_choose_conf: will use strict KEX ordering
debug1: kex: algorithm: curve25519-sha256
debug1: kex: host key algorithm: rsa-sha2-512
debug1: kex: server->client cipher: aes256-gcm@openssh.com MAC: <implicit> compression: none
debug1: kex: client->server cipher: aes256-gcm@openssh.com MAC: <implicit> compression: none
debug1: kex: curve25519-sha256 need=32 dh_need=32
debug1: kex: curve25519-sha256 need=32 dh_need=32
debug3: send packet: type 30
debug1: expecting SSH2_MSG_KEX_ECDH_REPLY
debug3: receive packet: type 31
debug1: SSH2_MSG_KEX_ECDH_REPLY received
Bad server host key: Invalid key length

let me know if I can help more but I'm pretty sure it is indeed a dupe of T240266 that's now impacting users with fedora's default settings

quoting from a serverfault.com question: "If your getting the "Invalid key length" error, the problem isn't your Ciphers (that may be it's own problem, but if you're getting a key, SSH has agreed to a Cipher)"

so from there, looking at the line "debug2: host key algorithms: rsa-sha2-512,rsa-sha2-256,ssh-rsa" it agreed on using RSA for the host key

and since openssh 7.6 there is "Refuse RSA keys <1024 bits in length".

so seems to me like is actually T240266

@Tarrow You could try if it works with ssh -o RequiredRSASize=1024 just to debug or until we can fix this.

@Tarrow please reopen if the fix provided by @Dzahn above doesn't work, otherwise we'll continue in {T240266}.

In T364217#9781787, @Dzahn wrote:

@Tarrow You could try if it works with ssh -o RequiredRSASize=1024 just to debug or until we can fix this.

I can confirm this works for me as another more fine grained workaround than sudo update-crypto-policies --set LEGACY.

For other people who encounter this I have ended up with a setup like this and it seems fairly satisfactory to me

~/.ssh/config

Host gerrit.wikimedia.org
 User <user>
 IdentityFile <gerrit_key>
 RequiredRSASize 1024

Let's continue in the other ticket

Thanks for confirming that @Tarrow Good to know we have a workaround.

Gerrit ssh host key fails on fedora39Closed, DuplicatePublicBUG REPORTActions

Description

Related Objects

Event Timeline

Gerrit ssh host key fails on fedora39
Closed, DuplicatePublicBUG REPORT
Actions