Removing support for DES-CBC3-SHA TLS cipher (drops IE8-on-XP support)
Closed, ResolvedPublic

Description

This is one of our two last remaining non-forward-secret ciphers, which we'd like to eliminate as soon as reasonably possible. It's also the subject of the SWEET32 birthday attack due to being a 64-bit (block size) cipher, which we've mostly-mitigated in other ways for now (shortened session key lifetimes).

The only statistically-significant browser which relies on this cipher to communicate with us is IE8-on-XP, which is long-unsupported (over 2 years now) and horribly insecure, so any motivation we can give users to get off of this browser is a net win for everyone involved.

Currently this cipher amounts to ~0.17% of all requests to our sites (and has been slowly declining for some time), and we've been running a very limited campaign for a while now which redirects a very small percentage of those requests (filtered down to just /wiki/ pageviews on the desktop sites, and only 1% odds) to the information at https://wikitech.wikimedia.org/wiki/HTTPS:_Browser_Recommendations . Because of technical limitations, we can't scale up that redirect much without having a better mechanism in place. IE8-on-XP is too old for CentralNotice JS to function correctly, so CN isn't really an option for campaigning here.

Users which cannot move off of the underlying Windows XP operating system can install the latest Firefox easily and use that to connect to us with much more secure cipher choices, so there is a fairly painless path forward for these users.

The plan of action here is this:

  • - Coordinate with the Community team to ensure they're aware of everything here ahead of any user complaints. This probably isn't the kind of situation where pre-announcements on community talk pages and/or mailing lists help much, as the target users aren't likely to be readers there, but it's still better to be prepared with answers.
  • - Prepare a Varnish synthetic page output based on our standard error page templates, which gives a very quick note about connection security and offers a link to the explanatory https://wikitech.wikimedia.org/wiki/HTTPS:_Browser_Recommendations .
  • - Code to show this to a random X% of pageviews from affected clients (already implemented for redirect, needs switch to synthetic output above).
  • - Once the above is ready, we'll set the final timeline in place: a 2 month period over which we'll ramp the percentage up from a small value (say, 5% of affected pageviews) to 100% of affected pageviews, and a further month during which we'll still allow connections from affected clients, but 100% of their pageviews will go to the information page.
  • - After the 3 month window is complete, remove support for this cipher entirely.
There are a very large number of changes, so older changes are hidden. Show Older Changes
Restricted Application removed a project: Patch-For-Review. · View Herald TranscriptOct 3 2016, 2:52 PM
BBlack moved this task from Triage to TLS on the Traffic board.Oct 3 2016, 2:55 PM
BBlack updated the task description. (Show Details)Oct 3 2016, 3:01 PM
BBlack updated the task description. (Show Details)Oct 3 2016, 3:46 PM
ema added a subscriber: ema.Oct 5 2016, 3:43 PM
Legoktm renamed this task from Removing support for DES-CBC3-SHA TLS cipher to Removing support for DES-CBC3-SHA TLS cipher (drops IE8-on-XP support).Oct 12 2016, 9:23 PM
JW added a subscriber: JW.Oct 27 2016, 4:42 AM
BBlack updated the task description. (Show Details)Oct 31 2016, 11:12 PM

After some IRC discussion, it seemed better to host the content pages on metawiki. It will look more-official, and it will also be easier to develop and maintain the content. Description updated to reflect that.

We've been stalled out on this because ETOOBUSY, which continues to push off the start of the timeline as well. At best we're looking at the 3-month window starting ~Dec1 at this point, and that's if we put some priority back on this.

We've been stalling on this a bit too long now. I'd like to start kicking off this process and getting in touch with Community as well. I've kinda backtracked on the idea of a redirect to meta-wiki for the initial notice to the user. I think for the initial page replacement, we should stick with a synthetic output directly from Varnish itself. That output can in turn contain a link to our existing wikitech page, or a similar page to that one on meta-wiki that's more-detailed. I've stolen from our existing Varnish errorpage.html and proposed some HTML for this here: P5175 (I wish pastes could be viewed raw with content-type!). Obviously wording and layout can be worked on a bit (and actual dates inserted for the real thing), but I like it having our standardized error theme and logo.

Text looks good to me, but two points:

  • "(upgrade or use Firefox!)" is somethat confusing since people might think an updated IE would be available or fix this. Let rather say "(upgrade your operating system or use Firefox!)"
  • Also https://wikitech.wikimedia.org/wiki/HTTPS:_Browser_Recommendations mentions a "market share" of 0.2%, while the proposed text says "less than 0.5%"?
BBlack added a comment.Apr 3 2017, 2:26 PM

Depending on the context I've been flipping between whether we're talking about just 3DES or both of the non-FS ciphers, sorry. In current weekly stats, 3DES is around 0.125% and AES128-SHA is around 0.225% for total of ~0.35% non-FS ( https://grafana.wikimedia.org/dashboard/db/tls-ciphers ). Probably the bolded redirect notice with the 0.2% number should be removed from the wikitech page, and the synthetic varnish error shown here should use "less than 0.2%" during the 3DES campaign. Once 3DES is disabled we can re-assess how we approach AES128-SHA and fix up various things appropriately.

BBlack updated the task description. (Show Details)Apr 17 2017, 2:55 PM

Change 348477 had a related patch set uploaded (by BBlack):
[operations/puppet@production] use synthetic warning for 1% of 3DES pageviews

https://gerrit.wikimedia.org/r/348477

BBlack updated the task description. (Show Details)Apr 17 2017, 3:58 PM

Change 348477 merged by BBlack:
[operations/puppet@production] use synthetic warning for 1% of 3DES pageviews

https://gerrit.wikimedia.org/r/348477

BBlack claimed this task.Apr 17 2017, 6:47 PM

I've deployed the change above, which gets all of the basics on track for how we want to operate the real campaign. We'll obviously make appropriate minor wording changes once we have dates set and as percentages increase.

Next step is I need to get in touch with the Community team about all of this before we move any further.

The synthetic warning output can be manually viewed at /test-sec-warning on any domain that maps to the text cluster, e.g.: https://en.wikipedia.org/test-sec-warning

Elitre added a subscriber: Elitre.Apr 21 2017, 10:35 AM
Johan added a subscriber: Johan.Aug 3 2017, 4:41 AM

Since this task was created in 2015, Firefox has stopped supporting Windows XP. They might need to install Firefox 52 (as opposed to current 54). Most folks who are still using Internet Explorer 8 on XP will probably find it fairly difficult to find and install an old version of a browser.

BBlack added a comment.Aug 3 2017, 2:07 PM

Cross-ticket updates: There's a separate sub-ticket for the Communications side of this change at T163251, and a timeline has been laid out there in T163251#3478043 . The TL;DR of the timeline is we'll ramp from 5% to ~29% blocked over the period of Aug 17 -> Oct 12, then 100% blocked on Oct 17, then protocol-disabled on Nov 17.

Other relevant notes:

  • The "blocked" requests will replace a pageview with a message about the upcoming end of support dates and a link to help with browser upgrading.
  • The block percentage is randomly per-request (we can't really rely on working cookies to do more advanced behaviors), and the ramping to ~1/3 before the jump to 100% is that there's little point using higher percentages for the ramp, as past ~1/3 it would be very frustrating to reload past the block anyways.
  • When we reach protocol-disabled on Nov 17, they won't be able to connect to get the friendlier error message at all.
  • 3DES currently clocks in at ~0.117% of our request statistics, and the bulk of those requests come from IE8-on-XP. For users, we're trying to keep the messaging simpler for the vast-majority case and talking about IE8-XP instead of 3DES, but on a technical level this is about the cipher in use, not the browser, and there will be other tiny-minority browsers/devices caught up in this as well, all of which fall within that overall ~0.117%.
Bawolff added a subscriber: Bawolff.Aug 3 2017, 5:59 PM

perhaps in the error page, the "use Firefox!" should be directly linked to the firefox 52 esr download page. The easier for users to find the link, and the less clicks the user has to go through, the more likely they will actually do it.

perhaps in the error page, the "use Firefox!" should be directly linked to the firefox 52 esr download page. The easier for users to find the link, and the less clicks the user has to go through, the more likely they will actually do it.

+1 to that. The next issue of Tech/News (draft, but will be frozen for translators in ~20 hours) points directly to the ESR page.
However, note that Firefox ESR 52 is ending support in Q2 of 2018, IIUC from the FAQ
But there are no decent alternatives for WinXP, at all.

perhaps in the error page, the "use Firefox!" should be directly linked to the firefox 52 esr download page. The easier for users to find the link, and the less clicks the user has to go through, the more likely they will actually do it.

+1 to that. The next issue of Tech/News (draft, but will be frozen for translators in ~20 hours) points directly to the ESR page.
However, note that Firefox ESR 52 is ending support in Q2 of 2018, IIUC from the FAQ
But there are no decent alternatives for WinXP, at all.

Midori maybe? Thats starting to get rather obscure though. There is a list at http://www.msfn.org/board/topic/176386-list-of-web-browsers-working-with-xp-2017/ but they mostly sound sketch.

Even while FF 52 is still supported by Mozilla, it's unlikely that Mozilla's security efforts can actually prevent all the possible exploits that breach the underlying WinXP.

From the perspective of the users' own local security concerns: the "end of support" for FF52 just means updates stop coming from Mozilla and it gets harder to initially install (have to go dig for backdated release links), but doesn't necessarily denote any big downgrade in the effective security of the user (which is basically zero, so long as they stick to XP).

From our perspective there's a couple things to note about the FF52 end date:

  1. The eventual end-date for FF52 is and has been a factor on our end: it's important we make our transition beforehand, or it may get even harder to offer any legitimate-looking, immediate, and easy "install Firefox" link.
  2. FF52 as it exists today already supports the strongest standardized TLSv1.2 cipher available today (ECDHE-ECDSA-CHACHA20-POLY1305 + X25519). So even if they get no further sec updates from Mozilla, we're unlikely to have a reason to disable them for "bad crypto" reasons anytime in the foreseeable future. Apple's latest (Safari 10) is further behind the curve on multiple fronts than FF52-for-XP, and way more popular.
MaxSem added a subscriber: MaxSem.Aug 4 2017, 6:30 PM

Users which cannot move off of the underlying Windows XP operating system can install the latest Firefox easily

For some value of "easily" which disregards corporate IT policies :-(

If a corporation is insane enough to still run XP and force their users to run IE, we can only hope that yet another site they can't use will be the final straw forcing them to do something.

TheDJ added a subscriber: TheDJ.Aug 8 2017, 12:25 AM

@Pigsonthewing if such companies are out there, then us cutting them off might finally be an indication to them how incredibly outdated they are and that it is time to get with the program... I personally count on some library computers without access to Wikipedia pretty soon and that's a good thing.

Nirmos added a subscriber: Nirmos.Aug 8 2017, 11:44 AM

What about IE7 on Windows XP and Windows Vista? If this also no longer has access to Wikipedia, then we should next do IE8 and IE9 on Windows Vista and Windows 7, followed by IE10 on Windows 7 and Windows 8 (not 8.1).

If a corporation is insane enough to still run XP and force their users to run IE, we can only hope that yet another site they can't use will be the final straw forcing them to do something.

Yes, Our mission is indeed to try to force corporate IT upgrades, and not to make knowledge freely available.

Nevertheless, my point stands. The claim I quoted is clearly false.

Both IE7 and IE8 for XP are what's being cut off in this transition, with IE8 being the newest IE that's even available for XP, AFAIK. However, we're not doing this with the express intent of deprecating older browser tech; it's just the natural fallout of raising our minimum security level for network connections. There's no further firm plans yet on Operations' end of things regarding deprecating specific browser versions, but we will most likely run through similar cipher/protocol deprecations in the future which may take out older browsers along the way like this one did.

I can do a sort of informal brain dump of what little we know about future IE+Windows problems:

Vista: supports, AFAIK, IE7 - IE9. Vista as well as all IE versions that run on it are out-of-support from Microsoft, so we'd certainly *like* people to abandon these ASAP, but there's no driving need yet, and I haven't dug into whether stats support this being an easy transition yet. IE7/Vista will continue to work through the current transition, as it supports much better ciphers than IE7 or 8 on XP did. However, IE7/Vista only supports TLS protocol version 1.0 (no 1.1 or 1.2 support). Therefore, support for IE7/Vista will likely fall apart everywhere on the Internet by about mid-2018 as TLSv1.0 gets dropped everywhere (see more on that below). I believe even IE9/Vista will fall to the same TLSv1.0 transition period, as I don't think any version of IE before IE11 supported TLSv1.1+, regardless of OS version. There may or may not be options like FF or Chrome for Vista users at that time, I haven't looked into that part yet.

Win7-10: are all still supported by MS at the OS level, and fare much better on these fronts than XP and Vista did. I believe Win7 security patches stop in early 2020, so there's still lots of time for it to fade out a little more gracefully, too. However, there are still lots of users (esp Win7/8 users) using IE versions 9-10 on these platforms, and they'll all need to upgrade to IE11 (or Edge, FF, Chrome, etc) by about the same mid-2018 timeframe as above, for the same basic reasons (to gain TLSv1.1+ support). IE11 on all of Win7-10 supports fairly modern crypto (TLSv1.2, ECDHE w/ P-256, AES-GCM for an AEAD cipher, etc), so there probably won't be any pressing need to drop support for them for a very long time, barring fresh new attacks on what is today the second-best standardized crypto suite available to anyone (the best being ChaPoly+X25519 that modern FF and Chrome support. Both that and what IE11/Win7 supports are strong enough that they continue to be used as primitives in the upcoming TLSv1.3 protocol as well).

In general, the mid-2018 transition where the Internet kills TLSv1.0 is going to be a little rough on older UAs. Aside from everything mentioned above in the Microsoft realm (all IE<11, and implicitly all Vista), we'll also be losing all Android versions prior to v4.4 (some even higher than that, with some vendors' customized builds that lagged behind on security...) and all Safari/iOS versions prior to v7. Unlike some of the other recent transitions, we'll probably follow the trend of the rest of the Internet on this one instead of trying to help drive the effort ourselves. It will be driven by commercial sites processing payment transactions that must drop TLSv1.0 by mid-2018 to keep their PCI compliance (and thus our Fundraising sites as well, but not necessarily the wikis).

If a corporation is insane enough to still run XP and force their users to run IE, we can only hope that yet another site they can't use will be the final straw forcing them to do something.

Yes, Our mission is indeed to try to force corporate IT upgrades, and not to make knowledge freely available.

Nevertheless, my point stands. The claim I quoted is clearly false.

It's not our mission to force corporate IT upgrades, but it is clearly on them rather than us to keep up with the most basic of upgrades. An IT department is supposedly staffed by professionals who have taken on the duty of maintaining systems for end-users. If anything, they should be held to higher standards than individual end-users at home. If the end user at home can install Firefox, so can an IT department. If the IT department in question has completely abandoned its users, that's an entirely separate problem that organization must face for a variety of reasons unrelated to this, and it's not our responsibility to take into account any other organizations' internal failures when considering our policies and actions.

Our mission is to make knowledge freely available to the world. It is also implicitly our mission to do so without irresponsibly subjecting our users to surveillance and/or censorship of their reading habits by state and/or commercial actors. It is on these principles that we advance the minimum security requirements we enforce. We're not asking anyone to be bleeding edge here. We're asking them to stop using platforms that are horribly insecure and outdated. In-between what we're dropping support for today and the bleeding edge of the most-current software, there exists several generations of slightly-less-horribly-insecure software which we continue to support which they can choose to upgrade to (or could have years ago).

Change 384578 had a related patch set uploaded (by BBlack; owner: BBlack):
[operations/puppet@production] ssl_ciphersuite: dump 3DES on 2017-11-17

https://gerrit.wikimedia.org/r/384578

Sad to see this claim made in 'Tech News' today:

If you use Internet Explorer 8 on Windows XP you can install Firefox 52 ESR instead

.

Johan added a comment.Oct 16 2017, 7:15 PM

@Pigsonthewing Yes, Tech News deals in simplification, for a number of reasons (non-native speakers without access to a translation to their language, non-technical users, ease of translation etc etc). This comes at the cost of precision, in this case "you" instead of "you or someone with administrator access". The newsletter makes similar simplified claims most weeks, I'd say.

Pigsonthewing added a comment.EditedOct 16 2017, 9:02 PM

simplification... at the cost of precision

"you" instead of "you or someone with administrator access".

That's an unnecessarily over-verbose example (and still false in some cases); all that is needed is, say, "you may be able to install" instead of "you can install". Alternatively, "try installing", which is one character shorter.

Johan added a comment.Oct 19 2017, 4:22 PM

Very well. "You may be able to install" is actually fairly complicated to parse (three verbs, not obvious to non-fluent speakers how they work together), but it might have been phrased better. This, however, is not on anyone you've discussed this with earlier in the thread, as Tech News items are typically phrased by me and not the developers. I suggest further discussion about how items are phrased in Tech News take place either on m:Talk:Tech/News/2017/42 or on m:User talk:Johan (WMF).

Change 384578 merged by BBlack:
[operations/puppet@production] ssl_ciphersuite: dump 3DES on 2017-11-17

https://gerrit.wikimedia.org/r/384578

BBlack closed this task as Resolved.Nov 17 2017, 3:05 PM

No 3DES connections or saved sessions left on the public cache endpoints \o/

Dzahn awarded a token.Nov 17 2017, 3:07 PM
Izno added a subscriber: Izno.EditedDec 28 2017, 1:22 PM

Just for documentation's sake, this ended AutoWikiBrowser on XP (ref request for help at en.wp). Users will either need to upgrade their Windows operating system or attempt to run it on Linux or Mac to use AWB on Wikimedia wikis.

BBlack updated the task description. (Show Details)May 4 2018, 6:05 PM

Change 440087 had a related patch set uploaded (by Ema; owner: Ema):
[operations/puppet@production] vcl: remove 3DES deprecation warning

https://gerrit.wikimedia.org/r/440087

Change 440087 abandoned by Ema:
vcl: remove 3DES deprecation warning

Reason:
There's no rush to merge this, let's wait for https://gerrit.wikimedia.org/r/#/c/operations/puppet/ /440114/ instead

https://gerrit.wikimedia.org/r/440087