Page MenuHomePhabricator

Removing support for AES128-SHA TLS cipher
Closed, ResolvedPublic

Description

This is a slightly lesser priority than the other non-forward-secret cipher we're removing in T147199 , but we'd still like to remove this as soon as reasonably possible so that we can reach 100% forward-secrecy and remove most of the motivation for anyone attacking our private keys.

Current informal analysis indicates that the overwhelming majority of the 0.25% (and declining) of our requests which use this cipher are not from outdated user agents, but rather due to outdated and/or mis-configured outbound corporate TLS proxies which actively downgrade the connection security of modern clients behind them. Assuming this analysis holds up, we can probably try to run a CN campaign targeting these users in hopes of reducing that percentage further before we eliminate the cipher (whereas in the 3DES case, the bulk of the affected UAs are too old for CN to work at all).

The first step here is we need to run better analysis on the UAs choosing this cipher, try to confirm percentage which are due to corporate TLS proxies vs truly outdated UAs, identify any major outdated UAs that might warrant modifying this plan.

Assuming the bulk are in fact bad corporate TLS proxies and there aren't significant oudated UAs to worry about, we'll probably set a date a few months out for final cutoff and begin running a CN campaign to warn affected users and encourage them to talk to their IT department about fixing the issue.

Event Timeline

I've re-done some of the sampled/informal AES128-SHA analysis from before, since it hasn't been done in about a year, and the past results were never recorded in detail. This is informational, to remind ourselves of the scope/impact in the future, whenever we get around to planning this one. Note below that the "BlueCoat" label is a stand-in for any kind of TLS-downgrading proxy, BlueCoat just happens to be the brand name of one of the most popular ones. We can identify these requests by a few key attributes:

  • Sometimes the UA string is literally ProxySG Appliance
  • By the origin IPs belonging to BlueCoat's cloud proxy service
  • Because the browser's UA string is far more modern and shouldn't be using ancient crypto like AES128-SHA, leaving a downgrading proxy as the only explanation
  • The combination of the legacy AES128-SHA cipher with the TLSv1.2 protocol choice. Legitimate ancient UAs do not implement TLSv1.2, whereas these proxies do implement it, but intentionally chose weak/ancient ciphers.

Contrasting these is the "Legit" subset - these appear to be genuine ancient/outdated user agents faithfully negotiating TLSv1.0+AES128-SHA as their best option with us.

On to the stats:

  • Avg reqs over past 30d: 0.220% (~170/sec)
  • BlueCoat / Legit split: 79% / 21%
  • Legit reqs over past 30d: 0.046% (~36/sec)
  • Top 20 UA strings among Legit set:
435  Mozilla/5.0 (PLAYSTATION 3 4.81) AppleWebKit/531.22.8 (KHTML, like Gecko)
400  Perl MediaWiki::Bot/5.006002 (https://metacpan.org/MediaWiki::Bot; [[User:Malarz pl]])
346  Nokia6280/2.0 (03.60) Profile/MIDP-2.0 Configuration/CLDC-1.1
172  Mozilla/5.0 (compatible; MJ12bot/v1.4.7; http://mj12bot.com/)
112  Nokia7610/2.0 (5.0509.0) SymbianOS/7.0s Series60/2.1 Profile/MIDP-2.0 Configuration/CLDC-1.0
 61  Opera/9.80 (S60; SymbOS; Opera Mobi/SYB-1204232254; U; ru) Presto/2.10.254 Version/12.00
 58  Mozilla/5.0 (DTV) AppleWebKit/531.2+ (KHTML, like Gecko) Espial/6.1.16 AQUOSBrowser/2.0 (US01DTV;V;0001;0001)
 58  Mozilla/5.0 (compatible; MSIE 9.0; Windows NT 6.0; Trident/5.0)
 57  Mozilla/5.0 (Windows NT 6.1; WOW64; Trident/7.0; rv:11.0) like Gecko
 56  ZTE-Z432/1.0.3 NetFront/4.2 QTV5.1 Profile/MIDP-2.1 Configuration/CLDC-1.1
 56  Mozilla/5.0 (Windows NT 6.1; WOW64; rv:38.0) Gecko/20100101 Firefox/38.0
 51  OpenOffice/4.1.1
 51  NokiaC2-01/5.0 (11.10) Profile/MIDP-2.1 Configuration/CLDC-1.1 Mozilla/5.0 AppleWebKit/420+ (KHTML, like Gecko) Safari/420+
 49  Mozilla/5.0 (SAMSUNG; SAMSUNG-GT-S5250/S525WXEKJ1; U; Bada/1.0; ru-ru) AppleWebKit/533.1 (KHTML, like Gecko) Dolfin/2.0 Mobile WQVGA SMM-MMS/1.2.0 NexPlayer/3.0 profile/MIDP-2.1 configuration/CLDC-1.1 OPN-B
 49  Mozilla/5.0 (SAMSUNG; SAMSUNG-GT-S5250/S5250XEKJ3; U; Bada/1.0; ru-ru) AppleWebKit/533.1 (KHTML, like Gecko) Dolfin/2.0 Mobile WQVGA SMM-MMS/1.2.0 NexPlayer/3.0 profile/MIDP-2.1 configuration/CLDC-1.1 OPN-B
 46  NokiaC3-00/5.0 (04.60) Profile/MIDP-2.1 Configuration/CLDC-1.1 Mozilla/5.0 AppleWebKit/420+ (KHTML, like Gecko) Safari/420+
 44  Mozilla/5.0 (Symbian/3; Series60/5.3 NokiaC7-00/111.030.0609; Profile/MIDP-2.1 Configuration/CLDC-1.1 ) AppleWebKit/533.4 (KHTML, like Gecko) NokiaBrowser/7.4.2.6 Mobile Safari/533.4 3gpp-gba
 41  DoCoMo/2.0 N01G(c500;TB;W24H16)
 40  Mozilla/4.0 (compatible; MSIE 7.0; Windows NT 6.0; GTB7.5; SLCC1; .NET CLR 2.0.50727; Media Center PC 5.0; eSobiSubscriber 2.0.4.16; .NET CLR 3.5.30729; .NET CLR 3.0.30618; .NET4
 35  SAMSUNG-GT-S5610K/S5610KJPMH1 NetFront/4.1 Profile/MIDP-2.0 Configuration/CLDC-1.1

Digging into these top UA strings a bit:

  • The top UA is the Playstation3. I don't expect it's common for someone to own a PS3 as their *only* means of access to Wikipedia and/or the Internet; I tend to think these users have other more up-to-date access mechanisms they can revert to (and probably use more often).
  • MediaWiki::Bot and MJI2bot are the second and fourth most popular UAs in this list, and they're obviously bots running on outdated software/infrastructure rather than real human UAs, and should be relatively-painless to fix.
  • After that we have mostly a long tail of outdated phones, including a bunch of Nokia Symbian stuff, various Java/MIDP-based browsers, possibly some somewhat-newer phones that ship vendor-specific and horribly crappy browser software (I'm looking at you, Samsung...).

My net taken on things boils down to:

  • Even if we started this deprecation today, it would proceed acceptably over a ~3 month window (similar to the 3DES one we're going through now)
  • The legit cases are already less than half the traffic level we started the 3DES deprecations at, so we have some precedent for how smoothly that can go (although we haven't finished it yet, as of this writing).
  • The BlueCoat-like cases we don't have to worry much about, relatively-speaking. These are mostly corporate traffic. We're not cutting off individuals with outdated phones here. When we start doing warnings to the UAs, these users can complain upstream to their IT departments or whatever, and hopefully get them fixed. Worst case they go use their personal or home devices and accept they can't browse us at work. The bottom line is these devices are actively downgrading connections from relatively-secure agents, and we don't want to do anything to support or encourage that. Putting pressure on them is a good thing. This is mostly the vendor's fault (BlueCoat) for shipping insecure-by-default configurations, and our best guess as to why is to save CPU costs on underpowered evil TLS interceptor middleboxes.
  • Realistically, I don't see us starting a 3-month countdown on this until somewhere in the early half of 2018. There's no definite planned dates yet, in any case. By then the stats will be even lower (they're still slowly but consistently dropping off naturally).

Another note-to-self for the future: https://gerrit.wikimedia.org/r/#/c/301817/ is where we removed the fairly-similar AES128-SHA256 and AES128-GCM-SHA256, throwing all such clients into the AES128-SHA bucket discussed here. It probably doesn't make sense to re-split these as we head towards deprecation, but it's a possibility worth considering if we want to split up the impacts over time a bit.

Since the last stats update ~6 months ago above, the overall percentage for AES128-SHA has continued its decline, from ~0.220% to ~0.0846% . We'll be looking to plan and start an abbreviated deprecation cycle on this during the upcoming quarter.

Change 449747 had a related patch set uploaded (by BBlack; owner: BBlack):
[operations/puppet@production] ssl_ciphersuite: forward-secret only

https://gerrit.wikimedia.org/r/449747

Change 449747 merged by Vgutierrez:
[operations/puppet@production] ssl_ciphersuite: forward-secret only

https://gerrit.wikimedia.org/r/449747