The trielad-ssh2 version is not the Debian package:

$ apt-cache search trilead
libjenkins-trilead-ssh2-java - Trilead SSH2 implementation for Java (Jenkins variant)
libjenkins-trilead-ssh2-java-doc - Documentation for libjenkins-trilead-ssh2-java
libtrilead-putty-extension-java - PuTTY key support for Trilead SSH2 library
libtrilead-putty-extension-java-doc - Documentation for libtrilead-putty-extension-java
libtrilead-ssh2-java - Java SSH libarary
$ dpkg -l '*trilead*'
No packages found matching *trilead*.

Some versions of it are embedded:

lsof -n -p 14468|grep -i trilead                                                                                                                                                                                    
java  jenkins  mem    /var/lib/jenkins/plugins/subversion/WEB-INF/lib/trilead-ssh2-1.0.0-build217.jar
java  jenkins  mem    /var/lib/jenkins/plugins/subversion/WEB-INF/lib/jsch.agentproxy.svnkit-trilead-ssh2-0.0.7.jar
java  jenkins  mem    /var/lib/jenkins/plugins/subversion/WEB-INF/lib/trilead-ssh2-build217-jenkins-7.jar
java  jenkins  mem    /run/jenkins/war/WEB-INF/lib/trilead-ssh2-build217-jenkins-8.jar
java  jenkins  mem    /run/jenkins/war/WEB-INF/lib/trilead-putty-extension-1.2.jar
java  jenkins   35r   /run/jenkins/war/WEB-INF/lib/trilead-putty-extension-1.2.jar
java  jenkins   84r   /run/jenkins/war/WEB-INF/lib/trilead-ssh2-build217-jenkins-8.jar
java  jenkins  378r   /var/lib/jenkins/plugins/subversion/WEB-INF/lib/trilead-ssh2-build217-jenkins-7.jar
java  jenkins  383r   /var/lib/jenkins/plugins/subversion/WEB-INF/lib/jsch.agentproxy.svnkit-trilead-ssh2-0.0.7.jar
java  jenkins  388r   /var/lib/jenkins/plugins/subversion/WEB-INF/lib/trilead-ssh2-1.0.0-build217.jar

It seems the Jenkins project has forked trilead-ssh2 :( We can dismiss the /subversion/ ones so that leave us with the jenkins core dependencies:

trilead-putty-extension-1.2.jar
trilead-ssh2-build217-jenkins-8.jar

So Jenkins fork/port is hosted at https://github.com/jenkinsci/trilead-ssh2

Upstream is http://svn.svnkit.com/repos/3rdparty/com.trilead.ssh2/tags/ and the Jenkins fork uses 1.0.0-build217.

I looked at the next build http://svn.svnkit.com/repos/3rdparty/com.trilead.ssh2/tags/1.0.0-build219/ and it does not support any hmac-sha2*.

Same goes for KEX: diffie-hellman-group-exchange-sha1 is supported but not the sha256 flavor.

Gotta report that to Upstream Jenkins :)

In T103351#1387872, @hashar wrote:

<snip>

Gotta report that to Upstream Jenkins :)

Completely forgot to report. I have filled a dupe, the existing issue is JENKINS-33021.

I've created this https://github.com/jenkinsci/ssh-slaves-plugin/issues/39 upstream.

Also upstream jenkins updated ssh from 0.8 to 0.14.0 in jenkins 2.37, see pull https://github.com/jenkinsci/jenkins/pull/2662

This https://issues.apache.org/jira/secure/ReleaseNote.jspa?projectId=12310849&version=12324784 supports the HMAC SHA-2

Feel free to raise it on the upstream bug tracker: https://issues.jenkins-ci.org/browse/JENKINS-33021

@hashar someone added support for this in trilead-ssh2 here https://github.com/jenkinsci/trilead-ssh2/pull/7 and one user wrote that it worked for them.

Nice!!! I replied there referencing various tickets in Jenkins Jira. Asked for a way to rebuild Jenkins with that new version of trilead and then I guess we can give it a try / test it out :}

@hashar hi, here's how to build it.

First this requires maven 3.1+ and as jessie only has 3.0 we need to use jessie-backports which has 3.3.x (Not sure if this applies to building jenkins 1.x)

First clone jenkins. Checkout related branch not sure if theres a 1.x.

Then clone the trilead-ssh2 repo from the user that created the patch since we want to then checkout the commit id.

then cd into trilead-ssh2

then do mvn install

then that will create something like build217-jenkins-9-SNAPSHOT somewhere in .m2/repository/*

We will then want to cd into jenkins and then edit the pom files in core and cli find trilead-ssh2 and updated it to build217-jenkins-9-SNAPSHOT

then we run mvn -Plight-test install

if you want some debug information incase it fails, then do mvn -Plight-test install -X

to do this offline do the following mvn -Plight-test install -o -X

@hashar Looks like it worked.

But in the /etc/ssh/sshd_config file it looks like strict is enabled and the hmac sha1 keys are not enabled.

This may be because i run the the ssh command on jenkins slave. IE it does ssh -i /var/lib/jenkins/.ssh/id_rsa test-jenkins@jenkins-slave-01 java -jar /slave.jar

@hashar may be related https://issues.jenkins-ci.org/browse/JENKINS-26495

"Have you installed the unrestricted policy into your Jenkins JVM? " I wonder what this would do exactly

"Java Cryptography Extension (JCE) installed " sounds interesting for this ticket

I tried installing jenkins crypto extension and that dosen't seem to get it working either.

I got this working by only commenting out KexAlgorithms.

It's using the newer macs though :)

@hashar ^^ :) :) :)

[01/10/17 21:59:52] [SSH] Opening SSH connection to 10.68.20.204:22.
[01/10/17 21:59:53] [SSH] Authentication successful.
[01/10/17 21:59:53] [SSH] The remote users environment is:
BASH=/bin/bash
BASHOPTS=cmdhist:complete_fullquote:extquote:force_fignore:hostcomplete:interactive_comments:progcomp:promptvars:sourcepath
BASH_ALIASES=()
BASH_ARGC=()
BASH_ARGV=()
BASH_CMDS=()
BASH_EXECUTION_STRING=set
BASH_LINENO=()
BASH_SOURCE=()
BASH_VERSINFO=([0]="4" [1]="3" [2]="30" [3]="1" [4]="release" [5]="x86_64-pc-linux-gnu")
BASH_VERSION='4.3.30(1)-release'
DIRSTACK=()
EUID=15041
GROUPS=()
HOME=/home/test-jenkins
HOSTNAME=gerrit-test
HOSTTYPE=x86_64
IFS=$' \t\n'
LANG=en_US.UTF-8
LOGNAME=test-jenkins
MACHTYPE=x86_64-pc-linux-gnu
MAIL=/var/mail/test-jenkins
OPTERR=1
OPTIND=1
OSTYPE=linux-gnu
PATH=/usr/local/bin:/usr/bin:/bin:/usr/games
PIPESTATUS=([0]="0")
PPID=9376
PS4='+ '
PWD=/home/test-jenkins
SHELL=/bin/bash
SHELLOPTS=braceexpand:hashall:interactive-comments
SHLVL=1
SSH_CLIENT='10.68.20.204 51222 22'
SSH_CONNECTION='10.68.20.204 51222 10.68.20.204 22'
TERM=dumb
UID=15041
USER=test-jenkins
_=']'
[01/10/17 21:59:53] [SSH] Checking java version of java
[01/10/17 21:59:53] [SSH] java -version returned 1.8.0_111.
[01/10/17 21:59:53] [SSH] Starting sftp client.
[01/10/17 21:59:53] [SSH] Copying latest slave.jar...
hudson.util.IOException2: Could not copy slave.jar into '/var/lib/jenkins' on slave
at hudson.plugins.sshslaves.SSHLauncher.copySlaveJar(SSHLauncher.java:1057)
at hudson.plugins.sshslaves.SSHLauncher.access$300(SSHLauncher.java:139)
at hudson.plugins.sshslaves.SSHLauncher$2.call(SSHLauncher.java:726)
at hudson.plugins.sshslaves.SSHLauncher$2.call(SSHLauncher.java:709)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
Caused by: hudson.util.IOException2: Could not copy slave.jar to '/var/lib/jenkins/slave.jar' on slave
at hudson.plugins.sshslaves.SSHLauncher.copySlaveJar(SSHLauncher.java:1052)
... 7 more
Caused by: com.trilead.ssh2.SFTPException: Permission denied (SSH_FX_PERMISSION_DENIED: The user does not have sufficient permissions to perform the operation.)
at com.trilead.ssh2.SFTPv3Client.openFile(SFTPv3Client.java:1201)
at com.trilead.ssh2.SFTPv3Client.createFile(SFTPv3Client.java:1074)
at com.trilead.ssh2.SFTPv3Client.createFile(SFTPv3Client.java:1055)
at hudson.plugins.sshslaves.SFTPClient.writeToFile(SFTPClient.java:93)
at hudson.plugins.sshslaves.SSHLauncher.copySlaveJar(SSHLauncher.java:1042)
... 7 more
[01/10/17 21:59:53] Launch failed - cleaning up connection
[01/10/17 21:59:53] [SSH] Connection closed.

Yes it fails at slaves.jar since i have slaves in / but i just wanted it to correctly ssh in.

In T103351#2932103, @Paladox wrote:

I got this working by only commenting out KexAlgorithms.

Ok, but i'm afraid that is just the same state as it was since before @hashar said "I have reenable the MAC/KEX on beta cluster but then...".

Can we see which KexAlgorithms client and server agree on once you commented that line, please.

Found it, we need to change https://github.com/jenkinsci/trilead-ssh2/blob/master/src/com/trilead/ssh2/transport/KexManager.java#L322 also.

Yea, so needs change in upstream client, but that is unmaintained and really old. Paladox said there are already multiple bugs open with jenkins for using this ancient client.

In T103351#2932698, @Paladox wrote:

Found it, we need to change https://github.com/jenkinsci/trilead-ssh2/blob/master/src/com/trilead/ssh2/transport/KexManager.java#L322 also.

Yep, so as Hashar says we need diffie-hellman-group-exchange-sha256.

I think that means we need to replace "diffie-hellman-group14-sha1" with:

diffie-hellman-group14-sha256@ssh.com

or

diffie-hellman-group15-sha256@ssh.com

and build the trilead war

It seems to have kind of worked.

I am now stuck at [01/11/17 01:36:38] [SSH] Opening SSH connection to 10.68.20.204:22.

IE thats the only line shown. So progress :)

@hashar hi would you know what we should do for supporting sha256 in https://github.com/jenkinsci/trilead-ssh2/blob/master/src/com/trilead/ssh2/transport/KexManager.java#L322 ttps://github.com/jenkinsci/trilead-ssh2/blob/master/src/com/trilead/ssh2/transport/KexManager.java

please?

@hashar Yay over night it stayed at that line. So now we just need to figure out how to support sha256 in that file :)

it seems it stayed like that because i added something to the file that i forgot to remove (was a test)

I guess we can stay we the support for hmac 256 but will need to disable the kex algo.

all we need to do is enable diffie-hellman-group-exchange-sha1 in the kex algo. So we apply the patch, then we can enable harden mac's again :)

In T103351#2933133, @Paladox wrote:

all we need to do is enable diffie-hellman-group-exchange-sha1 in the kex algo

No, i think we need diffie-hellman-group-exchange-sha256 , sha256 instead of sha1.

Oh it also needs sha256 in Mac too. Tested without the patch and disabled kex algo dosent work. Tested with patch + disabled kex algo and it works.

@hashar i found someone forked trilead-ssh2 and updated it to support newer mac's + kex see https://github.com/connectbot/sshlib/ :)

@hashar and @Dzahn woo hoo it worked.

[01/11/17 21:45:34] [SSH] Opening SSH connection to jenkins-slave-01:22.
[01/11/17 21:45:35] [SSH] Authentication successful.

it's using sha256 with sha1 disabled on sshd server :)

more info

[01/11/17 21:49:01] [SSH] Opening SSH connection to jenkins-slave-01:22.
[01/11/17 21:49:02] [SSH] Authentication successful.
[01/11/17 21:49:02] [SSH] The remote users environment is:
BASH=/bin/bash
BASHOPTS=cmdhist:complete_fullquote:extquote:force_fignore:hostcomplete:interactive_comments:progcomp:promptvars:sourcepath
BASH_ALIASES=()
BASH_ARGC=()
BASH_ARGV=()
BASH_CMDS=()
BASH_EXECUTION_STRING=set
BASH_LINENO=()
BASH_SOURCE=()
BASH_VERSINFO=([0]="4" [1]="3" [2]="30" [3]="1" [4]="release" [5]="x86_64-pc-linux-gnu")
BASH_VERSION='4.3.30(1)-release'
DIRSTACK=()
EUID=15041
GROUPS=()
HOME=/home/test-jenkins
HOSTNAME=jenkins-slave-01
HOSTTYPE=x86_64
IFS=$' \t\n'
LANG=en_US.UTF-8
LOGNAME=test-jenkins
MACHTYPE=x86_64-pc-linux-gnu
MAIL=/var/mail/test-jenkins
OPTERR=1
OPTIND=1
OSTYPE=linux-gnu
PATH=/usr/local/bin:/usr/bin:/bin:/usr/games
PIPESTATUS=([0]="0")
PPID=7606
PS4='+ '
PWD=/home/test-jenkins
SHELL=/bin/bash
SHELLOPTS=braceexpand:hashall:interactive-comments
SHLVL=1
SSH_CLIENT='10.68.20.204 59748 22'
SSH_CONNECTION='10.68.20.204 59748 10.68.22.0 22'
TERM=dumb
UID=15041
USER=test-jenkins
_=']'
[01/11/17 21:49:02] [SSH] Checking java version of java
[01/11/17 21:49:02] [SSH] java -version returned 1.7.0_111.
[01/11/17 21:49:02] [SSH] Starting sftp client.
[01/11/17 21:49:02] [SSH] Copying latest slave.jar...
hudson.util.IOException2: Could not copy slave.jar into '/var/lib/jenkins' on slave
at hudson.plugins.sshslaves.SSHLauncher.copySlaveJar(SSHLauncher.java:1057)
at hudson.plugins.sshslaves.SSHLauncher.access$300(SSHLauncher.java:139)
at hudson.plugins.sshslaves.SSHLauncher$2.call(SSHLauncher.java:726)
at hudson.plugins.sshslaves.SSHLauncher$2.call(SSHLauncher.java:709)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
Caused by: hudson.util.IOException2: Could not copy slave.jar to '/var/lib/jenkins/slave.jar' on slave
at hudson.plugins.sshslaves.SSHLauncher.copySlaveJar(SSHLauncher.java:1052)
... 7 more
Caused by: com.trilead.ssh2.SFTPException: Failure (SSH_FX_FAILURE: An error occurred, but no specific error code exists to describe the failure.)
at com.trilead.ssh2.SFTPv3Client.write(SFTPv3Client.java:1361)
at hudson.plugins.sshslaves.SFTPClient$1.write(SFTPClient.java:102)
at java.io.OutputStream.write(OutputStream.java:75)
at hudson.plugins.sshslaves.SSHLauncher.copySlaveJar(SSHLauncher.java:1044)
... 7 more
[01/11/17 21:49:02] Launch failed - cleaning up connection
[01/11/17 21:49:02] [SSH] Connection closed.

ignore slave.jar, i have it in a different place. just wanted to test ssh. yipeee

@hashar and @Dzahn here's source code i used https://github.com/paladox/trilead-ssh2/commit/74268eb68774cb1ae59648e40cd0994c214d9d41

Created a pull request at https://github.com/jenkinsci/trilead-ssh2/pull/8

In T103351#2934316, @Paladox wrote:

@hashar and @Dzahn woo hoo it worked.
it's using sha256 with sha1 disabled on sshd server :)

Very nice Paladox, thanks!

@hashar and @Dzahn i created another pull here https://github.com/jenkinsci/trilead-ssh2/pull/9 (I figured out which parts were needed to support this newer hmac and kex algo and made a commit that is smaller and only adds that support) :) should make upstream reviewer easier and less susceptible to being rejected for being to big or adding more dependancy's on other repo's :)

@hashar could you go the the jenkins issue that describes this mac (ssh) problem please and link to the pull request please? as i doint have a jenkins accounts.

@hashar and @Dzahn If we doint want to wait for upstream to merge it. We can always build jenkins our self by doing the following.

Git cloning jenkins then checkout the 1.x tag.

Then git clone trilead-ssh2 rep from jenkins

Then Apply this https://github.com/jenkinsci/trilead-ssh2/pull/9 patch onto the trilead-ssh2 repo.

Then run mvn -Plight-test -Dmaven.test.skip=true install

Then cd into jenkins and vi core/pom*

Then find trilead-ssh2 and replace the version 8 with 9-SNAPSHOT.

Then do he same for cli/pom*

Then run mvn -Plight-test -Dmaven.test.skip=true install

I tested this with jenkins 2.x but this should work with 1.x.

the war will be found in war/target/jenkins.war

Then you replace the war in /use/share/jenkins.war with the war here war/target/jenkins.war.

Make sure you have stopped jenkins before replacing the war there first.

Using these steps i managed to build it on jenkins 1.651.3 succesfully and it was quick too :)

@hashar even better see this https://github.com/jenkinsci/trilead-ssh2/pull/13 and https://github.com/jenkinsci/trilead-ssh2/pull/12

eddsa support + ed25519 support too :)

Upstream will merge https://github.com/jenkinsci/trilead-ssh2/pull/14 that in less then 24 hours if there is no reviews. Once it is merged this task can be added to one of the tasks that states upgrade to jerkins 2.x

A new release may happen next week :)

I think the new maintainer of trilead-ssh2 for jenkins will try and see if we can upgrade trilead-ssh2 in the ssh slaves plugin. If not we will have to upgrade jenkins.

@hashar Great news. The change has been merged now :). He is planning to see weather he can upgrade trilead-ssh2 in the ssh slaves plugin.

This will require a jenkins update.

This will be done in https://github.com/mc1arke/trilead-api-plugin which is not ready for release and has yet to be updated to the new version released. All plugins will need to be updated to use this new plugin.

This has landed upstream and will be in 2.58+

Thank you @Paladox !

Your welcome :). Anyways trilead is being removed from the core too. And hopefully the new ssh plugin will be released soon.

Upgraded Jenkins to 2.73.1 however the SSH plugin reports upon connection:

WARNING: [hudson.plugins.sshslaves.verifiers.TrileadVersionSupportManager getTrileadSupport]
Could not create Trilead support class. Using legacy Trilead features

So I guess the SSH plugin still uses some old Trilead version that lacks the latest SSH algos.

In T103351#3669098, @hashar wrote:

Upgraded Jenkins to 2.73.1 however the SSH plugin reports upon connection:

WARNING: [hudson.plugins.sshslaves.verifiers.TrileadVersionSupportManager getTrileadSupport]
Could not create Trilead support class. Using legacy Trilead features

So I guess the SSH plugin still uses some old Trilead version that lacks the latest SSH algos.

It shouldn't be using an old trilead version. It's using the latest https://github.com/jenkinsci/jenkins/blob/jenkins-2.73.1/core/pom.xml#L118

Did you also upgrade the ssh plugins?

@hashar https://issues.jenkins-ci.org/browse/JENKINS-44893 we could be using an older cipher.

One user found removing java 7 fixed the issue "remove jdk dir with old java 7"

Change 383120 had a related patch set uploaded (by Hashar; owner: Hashar):
[operations/puppet@production] Jenkins now supports our MAC/KEX algorithms

https://gerrit.wikimedia.org/r/383120

I have applied https://gerrit.wikimedia.org/r/383120 on the CI puppet master. sshd_config got:

+KexAlgorithms curve25519-sha256@libssh.org,diffie-hellman-group-exchange-sha256
+MACs hmac-sha2-512-etm@openssh.com,hmac-sha2-256-etm@openssh.com,umac-128-etm@openssh.com,hmac-sha2-512,hmac-sha2-256,umac-128@openssh.com

I disconnected / reconnected a few agents after having applied puppet and they connect just fine :]

Change 383122 had a related patch set uploaded (by Hashar; owner: Hashar):
[operations/puppet@production] Jenkins now supports our MAC/KEXY algorithms [prod]

https://gerrit.wikimedia.org/r/383122

Change 383120 merged by Muehlenhoff:
[operations/puppet@production] Jenkins now supports our MAC/KEX algorithms [labs]

https://gerrit.wikimedia.org/r/383120

Your welcome :). Also m1clark implemented the support. I helped test the fix and found the problem and he then adjusted his fix that worked :).

Change 383122 merged by Muehlenhoff:
[operations/puppet@production] Jenkins now supports our MAC/KEXY algorithms [prod]

https://gerrit.wikimedia.org/r/383122