My bad:

• Zuul connects to Gerrit with the user jenkins-bot as pointed out by QChris.
• not a valid EC private key file was due to my example pointing to the public key id_rsa.pub instead of the private one.

I have updated the example at T240266#6289890 and it is working now.

I finally looked at the blocker task and the Paramiko version bundled in our Zuul manages to connect to bastion.wmflabs.org with ecdsa-sha2-nistp256. T171165#6364439 and marked it resolved.

So I guess we can schedule the update afterall.

@Dzahn: Hi, the Due Date set for this open task is more than a month ago.
Can you please either update or reset the Due Date (by clicking Edit Task), or set the status of this task to resolved in case this task is done? Thanks!

@Aklapper It was not done on due date per request from @hashar

updated to Dec 31 2020 for now. @hashar I will be off for 2 weeks but let's get this done late November, early December if possible and you have no more concerns.

Is this rejected?

In T240266#6778343, @Dzahn wrote:

Is this rejected?

I hope not, I think this is still a good thing to do. Still determining how to schedule it (this is why I moved it to the Release-Engineering-Team INBOX so that it ends up in triage).

In discussing in our triage meeting the security risk from ssh host keys is small, i.e., you can impersonate a host you can possibly reveal the contents of a particular ssh session but there shouldn't be anything actually secret that can be revealed. Does this seem accurate to other folks on the ticket?

Referring back to the security-team who created this based on "Report originally from external pentest".

This had already been scheduled in the past and the only reason it was stopped again was T240266#6289890 and T171165 which has been resolved since.

Also T240266#6364444. But if the risk is considered acceptable then I'm happy to close it as rejected. Let's just do one or the other please.

rolled back my own edit on https://wikitech.wikimedia.org/wiki/Help:SSH_Fingerprints/gerrit.wikimedia.org:29418 because that had already been updated to new keys and then we reverted back in July.

this confused users, a good catch, as reported by addshore

T326204: Gerrit's host SSH key is incompatible with Fedora 33 crypto settings is a public duplicate of this.

Some notes the keys are read and written by Apache Mina Sshd SimpleGeneratorHostKeyProvider and reading
https://github.com/apache/mina-sshd/blob/c17c507bf35759abfbffab1d76c96aa6a3410282/sshd-common/src/main/java/org/apache/sshd/server/keyprovider/SimpleGeneratorHostKeyProvider.java the class documentation has:

A simple implementation of an {@link AbstractGeneratorHostKeyProvider} that writes and reads host keys using the
OpenSSH file format. Legacy keys written by earlier implementations used Java serialization. De-serializing is
restricted to a small number of classes known to exist in serialized {@link KeyPair}s.

So tentatively the key could be switched to use the OpenSSH file format when it is currently using the legacy Java serialization format.

Looking at Gerrit HostKeyProvider, it check the existence of any of the following files under /var/lib/gerrit2/review_site/etc:

ssh_host_rsa_key
ssh_host_ecdsa_key  # 256bits
ssh_host_ecdsa_384_key
ssh_host_ecdsa_521_key
ssh_host_ed25519_key

If none of those resulted in a key, it checks for ssh_host_key and read it as a serialized key.

If both types exists, Gerrit aborts with Multiple host keys exist and we thus CAN NOT add ecdsa key along side the existing ssh_host_key. The later, currently in use, MUST be unserialized and written in OpenSSH format as ssh_host_rsa_key before we can add more types.

Also digging into Gerrit java/com/google/gerrit/pgm/init/InitSshd.java, it generates the ssh keys using ssh-keygen but would not generate missing ones from the set it is supporting (the generating code doesn't run when ssh_host_rsa_key exists).

Is T370837: Remove RSA certificates and use only ECDSA certificates going to influence (or even fix) this?

@Aklapper I think it would be unrelated. This ticket here is about a key used by the gerrit ssh server which is directly exposed to the Internet (so not behind traffic caching servers). I would be surprised if that shows up in the webrequest_sampled_live table that is used in the other ticket.

Also the part that the traffic ticket talks about cost for issueing the certs makes me almost certain it's just about certs used for https in the CDN.

Though not entirely sure what the traffic team means by "removing support" exactly.

I think this needs to be prioritized. Occasional reports like:

17:33 <+wm-bb> <nemoralis> ❯ ssh -p 29418 nmw03@gerrit.wikimedia.org
17:33 <+wm-bb> <nemoralis> Bad server host key: Invalid key length

17:34 <+wm-bb> <nemoralis> I can't clone anything from Gerrit 🤔

currently our version is:

debug1: Remote protocol version 2.0, remote software version GerritCodeReview_3.10.6 (APACHE-SSHD-2.12.0)

Ed25519 Support: Gerrit has natively supported Ed25519 keys since version 2.14.

RSA Deprecation: While Gerrit still supports ssh-rsa (RSA with SHA-1), many modern clients (like OpenSSH 8.8+) disable it by default because it is weak. Moving to a larger RSA key size or Ed25519 avoids these compatibility issues.

FIPS Note: If your server is running in FIPS-compliant mode, Ed25519 might be disabled, and you should use ECDSA or high-bit RSA instead.

In T240266#11783791, @Dzahn wrote:

I think this needs to be prioritized. Occasional reports like:

17:33 <+wm-bb> <nemoralis> ❯ ssh -p 29418 nmw03@gerrit.wikimedia.org
17:33 <+wm-bb> <nemoralis> Bad server host key: Invalid key length

17:34 <+wm-bb> <nemoralis> I can't clone anything from Gerrit 🤔

Looks like recent versions of openssh are starting to reject 1024 RSA modulus keys. Seems like we ought to prioritize this.

If https://config-master.wikimedia.org/known_hosts is the canonical known_hosts for prod, this would be a good chance to ensure everything is using the canonical prod known_hosts.

(CC @hashar

Here is the command to verify what key type is used and it still confirms the 1024 bit:

ssh-keygen -lf <(ssh-keyscan -p 29418 gerrit.wikimedia.org 2>/dev/null)
1024 SHA256:j7HQoQ6fIuEgDHjONjI2CZ+2Iwxqgo2Ur5LbPqBgxOU [gerrit.wikimedia.org]:29418 (RSA)

The key in the file system should be this:

[gerrit1003:/srv/gerrit/site_path/review_site/etc] $ sudo file ssh_host_key 
ssh_host_key: Java serialization data, version 5

It is in the Java binary format where private and public key are combined into one binary file. So can't just compare finger print or checksum easily.

In puppet this is installed via:

    if $ssh_host_key != undef {
        file { "${gerrit_site}/etc/ssh_host_key":
            ensure    => present,
            # Java binary key format
            content   => wmflib::secret("gerrit/${ssh_host_key}", true),

...
Hiera:    profile::gerrit::ssh_host_key: 'ssh_host_key'

The wmflib::secret part means it is actually located at [puppetserver1001:/srv/git/private/modules/secret/secrets/gerrit.

Here we already created new host keys a couple years ago:

4.0K -rw-r--r--  1 gitpuppet gitpuppet  545 Jun 20  2023 ssh_host_ecdsa_256_key
4.0K -rw-r--r--  1 gitpuppet gitpuppet  655 Jun 20  2023 ssh_host_ecdsa_384_key
4.0K -rw-r--r--  1 gitpuppet gitpuppet  781 Jun 20  2023 ssh_host_ecdsa_521_key
4.0K -rw-r--r--  1 gitpuppet gitpuppet  444 Jun 20  2023 ssh_host_ed25519_key
4.0K -rw-r--r--  1 gitpuppet gitpuppet 1.2K Jun 20  2023 ssh_host_key
4.0K -rw-r--r--  1 gitpuppet gitpuppet 1.9K Jun 20  2023 ssh_host_rsa_key

But they are not in the Java binary format they need to be in:

ssh_host_ecdsa_256_key: OpenSSH private key
ssh_host_ecdsa_384_key: OpenSSH private key
ssh_host_ecdsa_521_key: OpenSSH private key
ssh_host_ed25519_key:   OpenSSH private key
ssh_host_key:           Java serialization data, version 5
ssh_host_rsa_key:       OpenSSH private key

BUT:

"Since you are running Gerrit 3.10.6, you should move away from the legacy binary ssh_host_key entirely and replace it with standard OpenSSH-formatted host keys. Modern Gerrit versions prioritize algorithm-specific files (RSA, Ed25519, ECDSA) and only use the Java-serialized binary for backward compatibility."

So we should be able to use the OpenSSH keys we already have. If we had to use a Java binary format we would have to stop gerrit and let "gerrit init" create it.. but per above.. we can avoid that.

Patch uploaded that simply replaces "ssh_host_key" with "ssh_host_key_ed25519" which already existed.

This needs to be communicated to users and the fingerprint needs to be stored somewhere.

Then we need to test it in a maintenance window.

https://gerrit.wikimedia.org/r/c/operations/puppet/+/1282395

The public key of the ed25519 key:

ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIFp+VIoTbE8Js9fwRCUy9KnSAewDQa2f6Dwi77R7IqS7 gerrit-code-review@gerrit1001.wikimedia.org

The fingerprint of the ed25519 key:

256 SHA256:njCmWMsshq3MqQxyIFO36UNwCwzTamXERqylF1XJhd8 gerrit-code-review@gerrit1001.wikimedia.org (ED25519)

In T240266#11879421, @thcipriani wrote:

If https://config-master.wikimedia.org/known_hosts is the canonical known_hosts for prod, this would be a good chance to ensure everything is using the canonical prod known_hosts.

--> T340947

@hashar Should we create a MOTD banner on Gerrit to tell users about a new SSH host key fingerprint? I had some trouble uploading one. Did plugins under operations/software/gerrit start to use submodules or so?

In T240266#11887308 @Dzahn wrote:

Since you are running Gerrit 3.10.6, you should move away from the legacy binary ssh_host_key entirely and replace it with standard OpenSSH-formatted host keys. Modern Gerrit versions prioritize algorithm-specific files (RSA, Ed25519, ECDSA) and only use the Java-serialized binary for backward compatibility.

Where does that quote comes from? Is that from an exchange with upstream? It matches what I had described last time T240266#10006329 and remains a blocker.

The current ssh-rsa host key is in ssh_host_key which is Java serialized. As soon as we add an OpenSSH key, that file will be ignored and that would cause anything using ssh to fail. Beside human users, there are a lot of automatization hitting Gerrit over SSH and all of them would end up breaking if we get rid of the key.

My idea is in my previous comment is to unserialize the Java serialized key, put it in ssh_host_rsa_key and remove ssh_host_key. We could then add support for other key and progressively migrate downstream user to the new key format then ultimately remove the legacy RSA key. I haven't looked at how to unserialize the binary though, it is certainly doable by writing some Java script / using jshell.

I used this program:

import java.io.*;
import java.security.*;
import java.util.Base64;

public class Convert {
  public static void main(String[] a) throws Exception {
    Object obj;
    try (ObjectInputStream ois = new ObjectInputStream(new FileInputStream(a[0]))) {
      obj = ois.readObject();
    }
    if (!(obj instanceof KeyPair)) {
      System.err.println("Not a KeyPair; got: " + obj.getClass().getName());
      System.exit(1);
    }
    PrivateKey pk = ((KeyPair) obj).getPrivate();
    String b64 = Base64.getMimeEncoder(64, new byte[]{'\n'})
                       .encodeToString(pk.getEncoded());   // PKCS#8 DER
    System.out.println("-----BEGIN PRIVATE KEY-----");
    System.out.println(b64);
    System.out.println("-----END PRIVATE KEY-----");
    System.err.println("Key algorithm: " + pk.getAlgorithm());
  }
}

plus ssh-keygen -p -N '' and ssh-keygen -y to convert /srv/gerrit/site_path/review_site/etc/ssh_host_key to openssh format files on gerrit2003.wikimedia.org:

root@gerrit2003:~# pwd
/root
root@gerrit2003:~# ls -l ssh_host*
-rw------- 1 root root 1020 Jul  8 21:04 ssh_host_rsa_key
-rw-r--r-- 1 root root  213 Jul  8 21:05 ssh_host_rsa_key.pub

:) thanks for working on this!

I took the unserialized keys you generated and added them to the private puppet repo.

Since we already had multiple keys there, including a 2048 bit ssh_host_rsa_key from 2023 (when we last worked on this I think),
this existing old but unserialized key is now called ssh_host_rsa_1024_key / ssh_host_rsa_1024_key.pub.

Puppet code installs the existing key from there with this:

# Java binary key format
content   => wmflib::secret("gerrit/${ssh_host_key}", true),

So the 1024 bit key in openssh format can be accessed as wmflib::secret("gerrit/ssh_host_rsa_1024_key").

Also see my commit message on old change https://gerrit.wikimedia.org/r/c/operations/puppet/+/1282395

I will amend or abandon though accordingly to add support for multiple keys instead of just replacing it.

To simplify things and clean up in the private repo I deleted the ecdsa keys that we created in June 2023 but did not end up using.

We should switch to ed25519 or keep using RSA.

delete mode 100644 modules/secret/secrets/gerrit/ssh_host_ecdsa_256_key
delete mode 100644 modules/secret/secrets/gerrit/ssh_host_ecdsa_384_key
delete mode 100644 modules/secret/secrets/gerrit/ssh_host_ecdsa_521_key

Just in case: backup is in /root/gerrit_host_keys on puppetserver1001 for a moment

Also deleted unused ssh_host_rsa_key which also was created in June 2023, has 2048 bits, and was meant to replace the 1024 existing host key but is in OpenSSH format to avoid confusion.

Ahmon made this patch to also support multiple keys for the known_hosts file.

https://gerrit.wikimedia.org/r/c/operations/puppet/+/1310197

I checked the wmf-laptop package pulls from the config-master host, deployed Ahmon's change and confirm there are now multiple lines for gerrit here:

https://config-master.wikimedia.org/known_hosts

• scheduled for tomorrow to do gerrit restart after https://gerrit.wikimedia.org/r/c/operations/software/gerrit/+/1313233 has been merged
• added to https://wikitech.wikimedia.org/wiki/Help:SSH_Fingerprints/gerrit.wikimedia.org:29418

fingerprint:

256 SHA256:njCmWMsshq3MqQxyIFO36UNwCwzTamXERqylF1XJhd8 ssh server at gerrit.wikimedia.org (ED25519)

public key:

ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIFp+VIoTbE8Js9fwRCUy9KnSAewDQa2f6Dwi77R7IqS7 ssh server at gerrit.wikimedia.org

• adding to Deployment calendar

• announcing on Wikitech-l

This has happened.

Gerrit is now sending the new ed25519 key. (not both though)

There were some issues with zuul (lacking ed25519 support) but they have been fixed by Dancy by upgrading the paramiko library used by old zuul. :)

follow-up fix https://gerrit.wikimedia.org/r/c/operations/puppet/+/1314098 was needed

now gerrit is serving both keys, old (RSA, but not in the Java format anymore on disk) and new (ed25519)

ssh-keyscan -p 29418 gerrit2003.wikimedia.org 2>&1 | grep -v APACHE

[gerrit2003.wikimedia.org]:29418 ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAAAgQCF8pwFLehzCXhbF1jfHWtd9d1LFq2NirplEBQYs7AOrGwQ/6ZZI0gvZFYiEiaw1o+F1CMfoHdny1VfWOJF3mJ1y9QMKAacc8/Z3tG39jBKRQCuxmYLO1SWymv7/Uvx9WQlkNRoTdTTa9OJFy6UqvLQEXKYaokfMIUHZ+oVFf1CgQ==

[gerrit2003.wikimedia.org]:29418 ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIFp+VIoTbE8Js9fwRCUy9KnSAewDQa2f6Dwi77R7IqS7

basically resolved. The last step is to remove the old key on August 4th as announced.

A volunteer has asked about making this ticket public now that it's resolved. @Dzahn can we do that?

I would like to do that on August 4th, when the old key is fully removed.

Until then, can we help the volunteer with any specific questions?

The announcement email kind of directs users here. They are naturally curious when they click on a link and its all *restricted*

deployed https://gerrit.wikimedia.org/r/c/operations/puppet/+/1320990

restarted gerrit using the Cookbook sre.gerrit.restart-gerrit

the RSA host key is now gone:

ssh-keyscan -p 29418 gerrit.wikimedia.org 2>&1 | grep -v APACHE
[gerrit.wikimedia.org]:29418 ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIFp+VIoTbE8Js9fwRCUy9KnSAewDQa2f6Dwi77R7IqS7

In T240266#12152251, @Bawolff wrote:

The announcement email kind of directs users here. They are naturally curious when they click on a link and its all *restricted*

fair enough! yea. well, it's public now.

@Dzahn Congratulations!

Change #1314995 merged by jenkins-bot:

[operations/software/gerrit@deploy/wmf/stable-3.10] disable the banner about the SSH host key change

https://gerrit.wikimedia.org/r/1314995