Disable insecure rsa-ssh public key signature algorithm
Open, MediumPublic
Actions

Assigned To

None

Authored By

	bking
	Sep 22 2022, 2:02 PM

Description

Per the OpenSSH developers, "It is now possible[1] to perform chosen-prefix attacks against the SHA-1 hash algorithm for less than USD$50K. For this reason, we will be disabling the "ssh-rsa" public key signature algorithm that depends on SHA-1 by default in a near-future release."

Note that it is not necessary to disable RSA host key verification entirely, as "...RSA SHA-2 signature algorithms rsa-sha2-256/512. These algorithms have the advantage of using the same key type as "ssh-rsa" but use the safe SHA-2 hash algorithms."

We don't use SSH CAs, at the moment but it's a good idea to note the following in case we ever do: "Certificates are at special risk to the aforementioned SHA1 collision vulnerability as an attacker has effectively unlimited time in which to craft a collision that yields them a valid certificate, far more than the relatively brief LoginGraceTime window that they have to forge a host key signature."

Creating this ticket to disable the rsa-ssh public key signature algorithm in our sshd_config.

you can test whether this is working with ssh client:

ssh -oHostKeyAlgorithms=ssh-rsa

If this algorithm is disabled, you should get this response:

Unable to negotiate with UNKNOWN port 65535: no matching host key type found. Their offer: rsa-sha2-512,rsa-sha2-256,ecdsa-sha2-nistp256,ssh-ed25519

Details

	Subject	Repo	Branch	Lines +/-
	sshd_config: disable ssh-rsa public key signature algorithm	operations/puppet	production	+2 -0

Customize query in gerrit

Event Timeline

bking created this task.Sep 22 2022, 2:02 PM

Restricted Application added a subscriber: Aklapper. · View Herald TranscriptSep 22 2022, 2:02 PM

bking updated the task description. (Show Details)Sep 22 2022, 2:14 PM

bking added a subscriber: jbond.

Change 834340 had a related patch set uploaded (by Bking; author: Bking):

[operations/puppet@production] sshd_config: disable ssh-rsa public key signature algorithm

https://gerrit.wikimedia.org/r/834340

gerritbot added a project: Patch-For-Review.Sep 22 2022, 2:29 PM

you can test whether this is working with ssh client:

ssh -oHostKeyAlgorithms=ssh-rsa

FWIW, when I try this I get a fairly nasty error:

$ ssh -oHostKeyAlgorithms=ssh-rsa deployment.eqiad.wmnet
@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
@    WARNING: REMOTE HOST IDENTIFICATION HAS CHANGED!     @
@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
IT IS POSSIBLE THAT SOMEONE IS DOING SOMETHING NASTY!
Someone could be eavesdropping on you right now (man-in-the-middle attack)!
It is also possible that a host key has just been changed.
The fingerprint for the RSA key sent by the remote host is
SHA256:0KD5Wof0On3oINcLMV+uCl/U9BzrI38275wc9wtuQP0.
Please contact your system administrator.
Add correct host key in /home/luwe/.ssh/known_hosts to get rid of this message.
Offending ED25519 key in /home/luwe/.ssh/known_hosts:69
  remove with:
  ssh-keygen -f "/home/luwe/.ssh/known_hosts" -R "deployment.eqiad.wmnet"
Host key for deployment.eqiad.wmnet has changed and you have requested strict checking.
Host key verification failed.

That’s under OpenSSH_8.9p1 Ubuntu-3, OpenSSL 3.0.2 15 Mar 2022, using a YubiKey (specifically a Yubico.com Yubikey 4/5 OTP+U2F+CCID according to lsusb).

The new fingerprint shown there matches the RSA fingerprint listed on Wikitech, so it’s probably not a MITM attack, but I’m still curious why this error would happen. When I SSH into the same host without any options, it seems to use an ed25519 key instead:

debug1: Server host key: ssh-ed25519 SHA256:h8QexqDisZib47cPqdZ3np/GcrSuqSS/2I/wli8GCPQ

That’s the second Server host key printed – the first one printed, ecdsa-sha2-nistp256 SHA256:t8jNhMIQlEMrU/SBxVTd1FvVAN9TsYfqbN5SHgZAuno, is the same whether I run SSH with or without the option, and matches bast3005. But if I try to SSH into bast3005 directly, then I get the same error, with ssh-rsa SHA256:P0LdEvrlb7GOJN5CJSi3Hkw1b8jRk6RDZaVIf9+48OE as the unknown host key fingerprint. (Which again matches Wikitech.) I don’t know why I can proxy through the bastion but not connect to it properly… is OpenSSH ignoring the -oHostKeyAlgorithms for the ProxyJump host?

bking removed a project: Discovery-Search.Sep 26 2022, 3:12 PM

Not ignoring @Lucas_Werkmeister_WMDE 's last question, but I did untag Discovery-Search as it is not directly related to Search. I will respond to the last comment as time permits (or if anyone in Infra Foundations wants to respond, feel free).

(FWIW, I don’t mean to block this – I don’t even know if you were looking for feedback or not ^^ just thought I’d mention it.)

That’s under OpenSSH_8.9p1 Ubuntu-3, OpenSSL 3.0.2 15 Mar 2022, using a YubiKey (specifically a Yubico.com Yubikey 4/5 OTP+U2F+CCID according to lsusb).

The new fingerprint shown there matches the RSA fingerprint listed on Wikitech, so it’s probably not a MITM attack, but I’m still curious why this error would happen. When I SSH into the same host without any options, it seems to use an ed25519 key instead:

@Lucas_Werkmeister_WMDE I belive thjis is expected and is down to the keys you already have in your authrosied_keys files. When you try to login to bast3005 you are forcing the key for ssh-rsa but you dont have the ssh-rsa key locally.

debug1: Server host key: ssh-ed25519 SHA256:h8QexqDisZib47cPqdZ3np/GcrSuqSS/2I/wli8GCPQ

This suggests you have both ed25519 and ecdsa keys in yuor authorised keys files and openssh is preferring the ed25519 one however to double check which keys is actully been used look for a message similar to the following
debug1: Host 'bast3005.wikimedia.org' is known and matches the ECDSA host key.

I don’t know why I can proxy through the bastion but not connect to it properly… is OpenSSH ignoring the -oHostKeyAlgorithms for the ProxyJump host?

From looking at the output the option passed with -oHostKeyAlgorithms=ssh-rsa only applies to the host yuor are actually trying to connect to and not the proxy host that you use. i.e. if you do ssh -v -oHostKeyAlgorithms=ssh-rsa lvs3007.esams.wmnet openssh will use the default options (edcsa in my case) for the bastion host key but will ask for ssh-rsa for the lvs host. If things work for, in this case the lvs3007 host it likely means that you already have a ssh-rsa entry for lvs3007 in your known_hosts file the h

jbond added a subscriber: MoritzMuehlenhoff.Sep 27 2022, 9:06 AM

Hang on, I think I misunderstood the testing command. If I want to test whether I would still be able to connect to a host that has ssh-rsa disabled server-side (when the host hasn’t actually disabled it yet), I should be using -oHostKeyAlgorithms=-ssh-rsa, with a minus sign in front of ssh-rsa to disable that algorithm. The command in the task description is for someone to test whether a host actually has ssh-rsa disabled server-side, which (IIUC) we haven’t done yet.

ssh -oHostKeyAlgorithms=-ssh-rsa mwdebug1001.eqiad.wmnet works fine on my end, so it sounds like my setup at least will not be broken by disabling rsa-ssh server-side. Sorry for the noise.

In T318345#8263863, @Lucas_Werkmeister_WMDE wrote:

Hang on, I think I misunderstood the testing command. If I want to test whether I would still be able to connect to a host that has ssh-rsa disabled server-side (when the host hasn’t actually disabled it yet), I should be using -oHostKeyAlgorithms=-ssh-rsa, with a minus sign in front of ssh-rsa to disable that algorithm. The command in the task description is for someone to test whether a host actually has ssh-rsa disabled server-side, which (IIUC) we haven’t done yet.

exactly :)

ssh -oHostKeyAlgorithms=-ssh-rsa mwdebug1001.eqiad.wmnet works fine on my end, so it sounds like my setup at least will not be broken by disabling rsa-ssh server-side.

Great thanks for the confirmation

This will most likely cause some issues with at least some cloud vps/toolforge users so we probably want to announce this in advance on cloud-announce@.

fnegri edited projects, added cloud-services-team; removed cloud-services-team (Kanban).Jan 18 2023, 6:37 PM

fnegri moved this task from Kanban to Inbox on the cloud-services-team board.

Change 834340 abandoned by Bking:

[operations/puppet@production] sshd_config: disable ssh-rsa public key signature algorithm

Reason:

Cleaning out gerrit queue; IF is aware of this issue and would be best to address it.

https://gerrit.wikimedia.org/r/834340

Maintenance_bot removed a project: Patch-For-Review.Sep 8 2023, 1:32 PM

joanna_borun triaged this task as Medium priority.Jan 22 2024, 4:10 PM

jhathaway subscribed.Jan 22 2024, 4:10 PM

MoritzMuehlenhoff added a project: User-MoritzMuehlenhoff.Jan 22 2024, 4:16 PM

Disable insecure rsa-ssh public key signature algorithmOpen, MediumPublicActions

Description

Details

Event Timeline

Disable insecure rsa-ssh public key signature algorithm
Open, MediumPublic
Actions