Page MenuHomePhabricator

Sort out DHE for Forward Secrecy w/ older clients
Closed, ResolvedPublic

Description

In a nutshell, and just speaking about the clients represented in ssllabs.com tests (there are probably other similar corner cases with old/unpopular platforms), there are 3 cases where we don't currently offer Forward Secrecy, but we could in theory by enabling a single DHE ciphersuite option and generating a dhparam file for nginx: Android 2.x, Java6, and OpenSSL 0.9.8.

Android 2.x is obviously the biggest reason, but there are probably (sadly) a fair amount of tools out there using older Java and OpenSSL that connect to us as well. In all 3 cases, the ciphersuite change for them would be to switch from TLS_RSA_WITH_AES_128_CBC_SHA (0x2f) to TLS_DHE_RSA_WITH_AES_128_CBC_SHA (0x33).

The primary issue here is that while we'd prefer to generate a decently-sized dhparam with, say, a prime size of 2048 or greater, anything larger than 1024 will break Java6 clients (and again, probably other corner cases we don't commonly think about), because it lacks support for larger dhparam. We can only specify a single dhparam to use for this, so it limits the other cases like Android to 1024 as well.

The open question in my mind is whether FS at the "dhparam 1024" level notably weakens crypto versus the straight RSA-2048 we're using with these clients today, enough that it's not worth the (strong) benefit of Forward Secrecy. There are two decent StackExchange questions on this: http://security.stackexchange.com/questions/42812/1024-bit-dhe-vs-2048-bit-rsa and http://security.stackexchange.com/questions/35471/is-there-any-particular-reason-to-use-diffie-hellman-over-rsa-for-key-exchange/35521 . Based on who's saying what in those threads, I'm inclined to go forward with the dhparam-1024 option here, adding only DHE-RSA-AES128-SHA to our ciphersuite list (just after our primary set of ECDHE FS-enabled ciphers, and before all of the non-FS options).

Since the issues in those StackExchange links are complex, I thought I'd ask here for any further opinions first.

Event Timeline

BBlack raised the priority of this task from to Needs Triage.
BBlack updated the task description. (Show Details)
BBlack added projects: acl*sre-team, Traffic.
BBlack added subscribers: BBlack, faidon, JanZerebecki and 2 others.

A notable counter-opinion: ssllabs.com caps us down from "A+" to "B" for allowing DHE-1024 as they consider it "weak".

Obviously our other two options are "Status Quo: leave Android 2x. and old Java/OpenSSL without FS", or "DHE-2048+, and accept Java6 breakage (and possibly other unknown/ancient/strange clients)".

More info on Java6:

20% of JVM marketshare and declining: https://plumbr.eu/blog/java/java-version-statistics-2015-edition
Oracle's Java Roadmap says public release updates/bugfixes for Java6 ended in Feb2013, and Java7 in Apr2015, but also notes that they plan to offer commercial support pretty much indefinitely for them: http://www.oracle.com/technetwork/java/eol-135779.html
Obviously, there's also many other vendors/distros involved here, especially IBM and Redhat.

I wonder if we're using any Java6 stuff in-house?

Redhat seems to have patched around the Java6/1024 problem late last year: https://rhn.redhat.com/errata/RHSA-2014-1634.html .

The picture I'm getting is vendors other than Oracle/Sun offer updates for this, it's just a question of whether people have bothered updating in the past several months.

Also, there are Java-specific workarounds we could point people at as well (to disable DHE and fall back to RSA on the client side):
http://blog.nashcom.de/nashcomblog.nsf/dx/dha-with-more-than-1024-key-size-and-java-still-works.htm
http://stackoverflow.com/questions/14253039/is-there-a-workaround-for-java-lang-runtimeexception-could-not-generate-dh-key

I wonder if we're using any Java6 stuff in-house?

I checked that as part of the leap second preparations (the Java bugfix for timer handling wasn't backported to Java 6): There are a handful of systems using openjdk-6 (some of the opendj instances e.g), but it's mostly gone. Jenkins was switched last week with T103491 and some gerrit cleanup is missing in T103668). The important Java code (hadoop, elastic) all run Java 7.

The situation for Java is a bit complicated:
As you mentioned the freely available Java support from Oracle ended in 2013 for 6 and last month for 7. After that period security updates and support are only available on a commercial basis from Oracle. Since large parts of Red Hat's business model are nowadays based on Java such short support time frames are critical to Red Hat, so they are maintaining a custom version of Java for a much longer period. This fork is called Icedtea and shipped in all Linux distros (for weird reasons Icedtea is branded OpenJDK in Debian).
Oracle still sends Icedtea their security patch sets from the closed branches in order to not let the Java ecosystem deviate too much, so the difference between the Icedtea and the Oracle releases is usually small (e.g. icedtea has e.g. pulseaudio supportand icedtea supports archs not supported by Oracle).

The patch to accept DH > 1024 is a Red Hat patch which they shipped initially through https://rhn.redhat.com/errata/RHSA-2014-1634.html in October 2014 (and the related Icedtea release). So every who updated Java on Linux in the last eight months has the patch deployed (all the previous Java security update rounds had high impact vulnerabilitiies/sandboxing breakouts, so they'll be deployed widely.

It's not clear whether Oracle supports DH > 1024 in the closed source line of Java 6, but everyone who runs Java 6 on Mac OS or Windows, has a copy which is unsupported since more than two years (and full of severe security problems).

Summarising, the possible breakage seems acceptable to me (after all the current Java is two major releases ahead these days).

Change 222023 had a related patch set uploaded (by BBlack):
tlsproxy: enable DHE-2048 FS for Android 2.x, etc.

https://gerrit.wikimedia.org/r/222023

^ That's assuming we're all ok with possibly breaking outdated and/or Oracle Java. I think I am.

BBlack triaged this task as Medium priority.Jul 3 2015, 2:28 AM
BBlack moved this task from Backlog to Traffic team actively servicing on the Traffic board.
BBlack set Security to None.

Now that we're logging negotiated ciphersuites, that makes it easier to clear up issues like these. I've been checking varnish logs on this, and it doesn't look like we have any significant traffic from Java6 clients. I did some logging matching for any Java6 client that currently fetches over SSL successfully, but uses a non-ECDHE cipher choice. The only hit that really comes up with any regularity at all is like this one:

75 RxURL        c /static/1.26wmf12/extensions/TimedMediaHandler/MwEmbedModules/EmbedPlayer/binPlayers/cortado/cortado-ovt-stripped-0.6.0.jar
75 RxHeader     c X-Forwarded-Proto: https
75 RxHeader     c X-Connection-Properties: SPDY=0; SSR=0; SSL=TLSv1; C=AES128-SHA;
75 RxHeader     c User-Agent: Mozilla/4.0 (Windows 7 6.1) Java/1.6.0_24
75 TxStatus     c 200

These requests are extremely rare, though (on the order of "somewhere well under 00.00%, takes minutes to catch just one"), and even then I can't be sure whether it would succeed at 2048-bit or not. It's entirely possible that while this client prefers AES128-SHA to ECDHE now, that it has the necessary patch to switch to 2048-bit DHE as well.

Based on the logging checks, I'm inclined to say we can move forward on this pretty quickly and easily. Should still be announced before making the switch to reduce confusion, perhaps to wikitech-l?

just fyi: this is the java applet for playing ogg on older ie browsers. the best part is it is not working and being replaced by video.js (iirc). @brion to confirm.

my two cents- very safe to move forward.

Yeah after talking it over with @faidon, it seems like the right general approach here is either to announce with big lead-time to wikitech if we think there's significant impact to worry about and react to, or not at all (in the general case, for future changes like this).

For this one, I'm going to unblock my -1 on the pending change for it and ping -ops channel again, and then go for it later today, I think.

Change 222023 merged by BBlack:
tlsproxy: enable DHE-2048 FS for Android 2.x, etc.

https://gerrit.wikimedia.org/r/222023

BBlack claimed this task.
BBlack moved this task from Traffic team actively servicing to Done on the Traffic board.