⚓ T100509 Jenkins master / client ssh connection fails due to missing ssh algorithm

Subject	Repo	Branch	Lines +/-
deployment-prep/integration: stop downgrading sshd MAC and KEX	operations/puppet	production	+0 -9
Reenable sshd MAC/KEX hardening for Jenkins and Beta	operations/puppet	production	+0 -11
Turn off sshd MAC/KEX hardening for Jenkins and Beta	operations/puppet	production	+11 -0

Status	Assigned	Task
Resolved	hashar	T100509 Jenkins master / client ssh connection fails due to missing ssh algorithm
Invalid	hashar	T100517 Jenkins jar should ship with a more recent jsch java lib version to support hardened algorithm
Resolved	hashar	T103342 Backport libjsch-java to Precise
Resolved	Paladox	T103351 Jenkins trilead-ssh2 doesn't support our MAC/KEX algorithms
Resolved	hashar	T144106 Upgrade Jenkins from 1.x to latest 2.x
Declined	None	T144105 Upgrade Jenkins to 1.651.3
Resolved	MoritzMuehlenhoff	T157429 Upload Jenkins LTS v2.46.2 to jessie-wikimedia/third-party
Resolved	hashar	T168644 Upgrade jenkins to 2.73.1 (new lts release)
Resolved	hashar	T162828 Upgrade jenkins server and jenkins slaves to java 8

hashar created this task.May 27 2015, 1:45 PM

hashar raised the priority of this task from to Needs Triage.

hashar updated the task description. (Show Details)

hashar added projects: acl*sre-team, Continuous-Integration-Infrastructure.

hashar added subscribers: hashar, MoritzMuehlenhoff.

Restricted Application added a subscriber: Aklapper. · View Herald TranscriptMay 27 2015, 1:45 PM

Being investigated with @MoritzMuehlenhoff

QChris sent a change for Gerrit which is related.

https://gerrit.wikimedia.org/r/#/c/213216/ Turn off sshd MAC and KEX hardening for gerrit replication targets

So seems java doesn't support some algorithms. That got turned off for gallium / lanthanum which explain why they pass. Need to do the same for all of CI :-}

Applied on hiera page https://wikitech.wikimedia.org/wiki/Hiera:Integration

"ssh::server::disable_nist_kex": false
"ssh::server::explicit_macs": false

Puppet then change /etc/ssh/sshd_config

-KexAlgorithms curve25519-sha256@libssh.org,diffie-hellman-group-exchange-sha256
-MACs hmac-sha2-512-etm@openssh.com,hmac-sha2-256-etm@openssh.com,umac-128-etm@openssh.com,hmac-sha2-512,hmac-sha2-256,umac-128@openssh.com

That solves the problem. Sending it to operations/puppet.

Gerrit had the same issue with T99990

Change 214055 had a related patch set uploaded (by Hashar):
Turn off sshd MAC and KEX hardening for Jenkins slaves

https://gerrit.wikimedia.org/r/214055

gerritbot added a project: Patch-For-Review.May 27 2015, 2:03 PM

Issue is fixed by cherry picked the puppet patch https://gerrit.wikimedia.org/r/214055 . Pending ops review to refine the patch if needed.

hashar mentioned this in T100517: Jenkins jar should ship with a more recent jsch java lib version to support hardened algorithm.May 27 2015, 2:37 PM

I have filled follow up tasks (T100517 and T100518). Puppet patch is already deployed on integration and beta cluster puppet masters.

All Jenkins slaves are now attached to the Jenkins master. The issue is solved.

Change 214055 merged by Muehlenhoff:
Turn off sshd MAC/KEX hardening for Jenkins and Beta

https://gerrit.wikimedia.org/r/214055

hashar mentioned this in rOPUP0f0c1dbcc506: Turn off sshd MAC/KEX hardening for Jenkins and Beta.May 27 2015, 2:54 PM

hashar mentioned this in T103342: Backport libjsch-java to Precise.Jun 22 2015, 12:39 PM

hashar closed subtask T100517: Jenkins jar should ship with a more recent jsch java lib version to support hardened algorithm as Invalid.

Change 219828 had a related patch set uploaded (by Hashar):
Reenable sshd MAC/KEX hardening for Jenkins and Beta

https://gerrit.wikimedia.org/r/219828

hashar closed subtask T103342: Backport libjsch-java to Precise as Resolved.Jun 22 2015, 2:56 PM

hashar added a subtask: T103351: Jenkins trilead-ssh2 doesn't support our MAC/KEX algorithms.Jun 22 2015, 3:01 PM

Change 219828 abandoned by Hashar:
Reenable sshd MAC/KEX hardening for Jenkins and Beta

Reason:
Broken for now because of java trilead-ssh2 T103351

https://gerrit.wikimedia.org/r/219828

hashar mentioned this in T112025: Wikimedia Gerrit doesn't work if OpenSSH version is higher than 7.0.Sep 10 2015, 7:53 AM

greg added a project: Essential-Work.Sep 21 2015, 8:53 PM

Restricted Application added a subscriber: Matanya. · View Herald TranscriptSep 21 2015, 8:53 PM

@hashar do you think this will change on contint1001 with openjdk-7-jdk:amd64 7u111-2.6.7-1~deb8u1 ?

Change 318248 had a related patch set uploaded (by Dzahn):
deployment-prep/integration: stop downgrading sshd MAC and KEX

https://gerrit.wikimedia.org/r/318248

Change 318248 abandoned by Hashar:
deployment-prep/integration: stop downgrading sshd MAC and KEX

Reason:
The issue is in Jenkins itself, not Java :D T100509

https://gerrit.wikimedia.org/r/318248

gerritbot mentioned this in T95757: Phase out gallium.wikimedia.org.Oct 27 2016, 6:10 AM

@Dzahn the issue is in Jenkins itself which uses an old / apparently unmaintained ssh library: trilead-ssh2 T103351. So we need to keep the rule around.