Page MenuHomePhabricator

Wikimedia Gerrit doesn't work if OpenSSH version is higher than 7.0
Closed, ResolvedPublic

Description

$ git pull
Unable to negotiate with 208.80.154.81: no matching key exchange method found. Their offer: diffie-hellman-group1-sha1
fatal: Could not read from remote repository.

diffie-hellman-group1-sha1 was deprecated since openssh 7.0, and wikimedia gerrit doesn't support diffie-hellman-group14-sha1 which is still supported by current version of openssh.

More information and a possible solution in the server-side: https://code.google.com/p/gerrit/issues/detail?id=3517

woarkaround

A workaround client-side solution is adding below lines into your .ssh/config file, but it is not recommended since it was deprecated and less secure than new one.

Host gerrit.wikimedia.org
    KexAlgorithms +diffie-hellman-group1-sha1

Affects:

  • OS X (El Capitan?)
  • Arch Linux
  • Gentoo
  • Any Windows systems which have git 2.5.1 or newer.
  • Any un*x systems which have openssh 7.0 or newer.

Event Timeline

devunt raised the priority of this task from to High.
devunt updated the task description. (Show Details)
devunt added a project: Gerrit.
devunt added a subscriber: devunt.

Based on IRC discussions, this seems to only effect users of the latest OSX, correct? Any other users effected?

Debian unstable is still at 6.9 (and latest Debian stable, Jessie, is at 6.7).

Most probably related:

The Jenkins master could not ssh to the labs slaves due to the algo changes (T100509) and spurted out:

fatal: no matching mac found: client hmac-sha1-96,hmac-sha1,hmac-md5-96,hmac-md5 server hmac-sha2-512,hmac-sha2-256 [preauth]
error: Could not load host key: /etc/ssh/ssh_host_ed25519_key

Jenkins has a ssh java implementation built-in (trilead-ssh2) which does not have the new algos (T103351).

@QChris found the same problem for Gerrit which failed to replicate its git repo over SSH: https://gerrit.wikimedia.org/r/#/c/213216/

The Gerrit master runs on ytterbium.wikimedia.org which is a Precise box. I have no idea whether Gerrit comes with a built-in SSH or depends on some system .deb package. Either case it is outdated.

Meanwhile the workaround is to tweak the ssh_config for the gerrit.wikimedia.org to force it to known algos.

In short diffie-hellman-group14-sha1 is supported by JSch since 0.1.52 and apparently our Gerrit has 0.1.50.


From our gerrit.git in branch wikimedia-cur-deployed:

lib/BUCK list jsch which is used by JGit afaik:

maven_jar(
  name = 'jsch',
  id = 'com.jcraft:jsch:0.1.50',
  sha1 = 'fae4a0b1f2a96cb8f58f38da2650814c991cea01',
  license = 'jsch',
)

And lib/mina/BUCK for mina (network lib) and apache sshd:

maven_jar(
  name = 'core',
  id = 'org.apache.mina:mina-core:2.0.7',
  sha1 = 'c878e2aa82de748474a624ec3933e4604e446dec',
  license = 'Apache2.0',
  exclude = EXCLUDE,
)

maven_jar(
  name = 'sshd',
  id = 'org.apache.sshd:sshd-core:0.9.0-2-g90093c5',
  sha1 = 'cb518a4c35d9f34c4be3b076faa5b5b8f5b9b0ea',
  license = 'Apache2.0',
  deps = [':core'],
  exclude = EXCLUDE,
  repository = GERRIT,
)

For what is worth http://www.jcraft.com/jsch/ChangeLog JSch Changelog lists algorithms support being introduced/removed:

Changes since version 0.1.52:
...
- change: Logjam: use ecdh-sha2-nistp* if available,
          ecdh-sha2-nistp256,ecdh-sha2-nistp384,ecdh-sha2-nistp521,
          diffie-hellman-group14-sha1,
          diffie-hellman-group-exchange-sha256,
          diffie-hellman-group-exchange-sha1,diffie-hellman-group1-sha1
- change: Logjam: diffie-hellman-group-exchange-sha256 and
          diffie-hellman-group-exchange-sha1 will use 2048-bit key on
          Java8's SunJCE, thanks to JDK-6521495 and JDK-7044060.
...
Changes since version 0.1.51:
...
- feature: added the support for SSH ECC defined in RFC5656,
           ecdsa-sha2-nistp256,
           ecdsa-sha2-nistp384,
           ecdsa-sha2-nistp521,
           ecdh-sha2-nistp256,
           ecdh-sha2-nistp384,
           ecdh-sha2-nistp521
           This functionality requires Java7 or later.
- feature: added the support for server host keys in
           ecdsa-sha2-nistp256,
           ecdsa-sha2-nistp384,
           ecdsa-sha2-nistp521
- feature: generating key-pairs in
           ecdh-sha2-nistp256,
           ecdh-sha2-nistp384,
           ecdh-sha2-nistp521
- change: aes192-ctr, aes256-ctr and
          diffie-hellman-group-exchange-sha256 have been enabled
          by the default.
- change: key exchange methods, ecdh-sha2-nistp256,
          ecdh-sha2-nistp384 and ecdh-sha2-nistp521 have been enabled
          by the default.
- change: the support for host keys in ecdsa-sha2-nistp256,
          ecdsa-sha2-nistp384 and ecdsa-sha2-nistp521 have been enabled
          by the default.
...

Gerrit 2.11 (T70271: Upgrade gerrit to 2.12) comes with jsch 0.1.51.

The bump to jsch 0.1.52 hasn't made it to any Gerrit stable branch yet (i.e. it is only in master).

$ git show 11746e1e483b9542d98d856e8a11805873eb6275
commit 11746e1e483b9542d98d856e8a11805873eb6275
Author: David Ostrovsky <david@ostrovsky.org>
Date:   Sun May 10 20:13:29 2015 +0200

    Bump Jsch to 1.52
    
    According to the change log the new version fixed some bugs and added
    some new features: [1].
    
    [1] http://www.jcraft.com/jsch/ChangeLog
    
    Change-Id: Iff40eec5da1906c85c4325bb2cd14f996d849d6d

diff --git a/lib/BUCK b/lib/BUCK
index 98d5739..1230727 100644
--- a/lib/BUCK
+++ b/lib/BUCK
@@ -78,8 +78,8 @@ maven_jar(
 
 maven_jar(
   name = 'jsch',
-  id = 'com.jcraft:jsch:0.1.51',
-  sha1 = '6ceee2696b07cc320d0e1aaea82c7b40768aca0f',
+  id = 'com.jcraft:jsch:0.1.52',
+  sha1 = '4988681cd939d45f88cb56a0806ad4064e621d1c',
   license = 'jsch',
 )

So unless we can rebuild our Gerrit against jsch 0.1.52, I guess this task is blocked upon upgrading to a yet unreleased version of Gerrit.

hashar lowered the priority of this task from High to Medium.Sep 10 2015, 10:08 AM

I get this problem and have to use the work around on Windows 10

Does it really have to be a Gerrit upgrade and can't be a change to a puppetized config file as we would do with regular sshd? I mean, a version upgrade would be good, but as in "do we have to rebuild it for each change to ssh settings".

re: this suggested solution to delete/replace BouncyCastle libs, i checked on the server and don't see those files. /usr/share/java does not have them and i can't find a bcprov file elsewhere either so far.

Looks like git 2.5.1 start to use OpenSSH 7.0.
Just update my git to lastest in Win10 and XP, same result before I add KexAlgorithms to .ssh/config

re: this suggested solution to delete/replace BouncyCastle libs, i checked on the server and don't see those files. /usr/share/java does not have them and i can't find a bcprov file elsewhere either so far.

Apparently, you still need bcpkix-jdk15on-151.jar and bcprov-jdk15on-151.jar

@demon: I know I know, but I'd appreciate your take on this :)

Based on IRC discussions, this seems to only effect users of the latest OSX, correct? Any other users effected?

I am on Gentoo and affected too.

git 2.x on windows too all windows that support the new git update are affected.

I have no idea whether Gerrit comes with a built-in SSH or depends on some system .deb package.

All SSH parts are built-in, some aspects can be tuned a bit by adding BouncyCastle jars.
(So as you said further down, it once again comes down to updating Gerrit)

An upgrade of gerrit will benefit not just this bug but several others. We just need to pick whch one 2.8.6.6 or 2.11.3.

User named RandomDSdevel also complained about this on #wikimedia-dev.

@greg: I'm still on OS X v10.10.5 'Yosemite,' and I'm affected by this problem as well. I don't know whether it's due either to something that's changed server-side or the fact that I just recently did a brew install -vd --build-from-source openssh --with-ldns, which updated me from the built-in version of OpenSSH ('OpenSSH_6.2p2, OSSLShim 0.9.8r 8 Dec 2011') to 'OpenSSH_7.1p1, OpenSSL 1.0.2d 9 Jul 2015,' but trying to run git pull -v on my copy of the MediaWiki repository gives me the expected error output:

Unable to negotiate with 2620::861:3:208:80:154:81: no matching key exchange method found. Their offer: diffie-hellman-group1-sha1
fatal: Could not read from remote repository.

Please make sure you have the correct access rights
and the repository exists.

@revi: Ack, ya 'ninja'd me! I've got more details for you guys, though.

(So as you said further down, it once again comes down to updating Gerrit)

@demon is out until monday, I'll have him look at this first thing.

greg raised the priority of this task from Medium to High.Sep 11 2015, 6:17 PM
I don't know whether it's due either to something that's changed server-side or the fact that I just recently did a `brew install -vd --build-from-source openssh --with-ldns`,

@RandomDSdevel. it's the latter, the update that got you to openssh 7. That disabled some insecure key exchange methods and the gerrit server has not been changed but is outdated, so now there is no method that they can both agree on.

as a _temporary_ workaround you can configure your client to still use the old method as before:

Host gerrit.wikimedia.org
    KexAlgorithms +diffie-hellman-group1-sha1

@Dzahn: Got 'ya, will do. I'll be keeping an eye on this task, though, so I can remove this hack when things are updated on the server end.

Does it really have to be a Gerrit upgrade and can't be a change to a puppetized config file as we would do with regular sshd? I mean, a version upgrade would be good, but as in "do we have to rebuild it for each change to ssh settings".

Sadly this isn't configurable by Gerrit. Yet another side effect of not using the system sshd... Nothing we can do about that.

In short diffie-hellman-group14-sha1 is supported by JSch since 0.1.52 and apparently our Gerrit has 0.1.50.

[snip a lot of useful information]

So unless we can rebuild our Gerrit against jsch 0.1.52, I guess this task is blocked upon upgrading to a yet unreleased version of Gerrit.

So, running master wouldn't be fun or a good idea, lol.

We can try building 2.8.x against the newer jsch. Could work. Could also do T65847 while we're there. If that doesn't work, I guess 2.11 would have to be the next route...

2.11 fixes so much bugs and includes own built in editor would make work load so much easier if we upgrade to that one. And as we are going to upgrade to sort this issue we might as well go to a stable and newer version with updated security.

Change 237753 had a related patch set uploaded (by QChris):
Make gerrit offer newer key exchange algorithms for new sshs

https://gerrit.wikimedia.org/r/237753

I verified that even for our old gerrit, adding BouncyCastle is sufficient to make OpenSSH 7 happy again.
Hence, we have found yet another way to postpone updating gerrit :-((

(The above change is not yet in mergable state, but should show how to make gerrit offer better key exchange algos. Please chime in if you know how to deploy jars at WMF these days.)

We can try building [...] against the newer jsch.

I'd not attempt that.
Mina SSHD caused issues for us more than once. Upstream gerrit had to roll back Mina SSHD updates more than once because of unexpected issues. I would not mess with Mina SSHD or Jsch behind gerrit's back. That's bound to backfire.

Yeah, sounds like doing something like that could get ugly!

I verified that even for our old gerrit, adding BouncyCastle is sufficient to make OpenSSH 7 happy again.

1.44? If my notes at T65847#699055 are correct, gerrit 2.8.5+ requires SSHD 0.11.0 which requires bouncycastle 1.49 if installed.

Hence, we have found yet another way to postpone updating gerrit :-((

Heh.

(The above change is not yet in mergable state, but should show how to make gerrit offer better key exchange algos. Please chime in if you know how to deploy jars at WMF these days.)

T107083#1486202 claims the libbcprov-java apt package is installed on logstash1001.

@Paladox: Other arguments (=unrelated to this very task, like random features in 2.11) for upgrading to Gerrit 2.11 should go to T70271 instead.

1.44? If my notes at T65847#699055 are correct, gerrit 2.8.5+ requires SSHD 0.11.0 which requires bouncycastle 1.49 if installed.

Our gerrit is not at 2.8.5+. Our gerrit is older. It is only 2.8.1-ish* :-)
Hence, we want bouncycastle in version 1.44.

*) Gerrit exposes it's version number for example on most pages in the lower right-hand corner. It's version is currently 2.8.1-4-ga1048ce.

I know. I was wondering if installing BouncyCastle 1.44 will prevent the upgrade to gerrit 2.8.5+.

Change 237918 had a related patch set uploaded (by QChris):
Add jar for BouncyCastle 1.44 from Debian wheezy

https://gerrit.wikimedia.org/r/237918

I was wondering if installing BouncyCastle 1.44 will prevent the upgrade to gerrit 2.8.5+.

It will not prevent the upgrade to 2.8.5+.
It only means that when upgrading Gerrit to 2.8.5+, we will have to upgrade BouncyCastle too.

QChris: It is trivial to move it from Debian to our repo: https://wikitech.wikimedia.org/wiki/APT_repository#Updating_the_repository

@jcrespo I like how you use the term “trivial” here :-)
The page you linked starts with saying to add things to the svn repo that got killed a few months back :-D
Also running reprepro commands on brewster is something that is not exactly "trivial", when you're not Ops and cannot log in to brewster to start with.

I guess the right way would be to create a new operations/debs/libbcprov repo in our Gerrit. Import the wheezy debianization there. And roll from there. But that's way too much overhead when considering that the .deb isn't a perfect fit in first place.

So I went forward and uploaded a change to bring the jar to ytterbium similar to Gerrit plugins jars get there. Although bcprov does not implement Gerrit's plugin interfaces, in the end bcprov is pluggable functionality for Gerrit. So it's not a too bad fit. And it would spare us the whole packaging overhead. Let's see what @demon thinks about it.

Yes, I was offering my help here to make things "the right way", and I would take the "supposed overhead", but it seems that is not helpful. I am sorry :-(

Yes, I was offering my help here to make things "the right way", and I would take the "supposed overhead", but it seems that is not helpful. I am sorry :-(

@jcrespo Sorry, I misunderstood your message. If you have time to do the proper packaging of libbcprov, that'd be great!

So if you have time to do the proper packaging of libbcprov, please do. Then I'll upload a change to gerrit to pick up the jar from there.

Change 237918 merged by Chad:
Add jar for BouncyCastle 1.44 from Debian wheezy

https://gerrit.wikimedia.org/r/237918

Does this mean we doint need the workaround now since the patch was merged.

@Paladox Does it now work for you without the workaround?

@JanZerebecki no it still says

Unable to negotiate with 208.80.154.81: no matching key exchange method found. Their offer: diffie-hellman-group1-sha1
fatal: Could not read from remote repository.

Please make sure you have the correct access rights
and the repository exists.

It's not yet expected to work without the workaround.

There are two changes linked to this task. One got merged. The other is still pending.
Once both are merged, deployed, and everything, then it should work without the workaround.

Change 237753 merged by Dzahn:
Make gerrit offer newer key exchange algorithms for new sshs

https://gerrit.wikimedia.org/r/237753

When will we be able to start using gerrit normaly without needing to do a workaround. So that I can test if it works.

@Paladox it was a restart to fix this issue. try it now :)

This comment was removed by Paladox.
Dzahn claimed this task.