We've mitigated breach for csrf tokens using variable tokens. Iirc, when breach was first announced, ops had said turning off gzip wasn't an option.

Do we know when gzipping was disabled?

Actually now that I'm looking more into it, it seems like it might be a Firefox-only issue.

So far I've failed to reproduce the issue in cURL using the same request headers shown in Firefox developer tools:

$ curl 'https://en.wikipedia.org/wiki/Barack_Obama' -H 'Host: en.wikipedia.org' -H 'User-Agent: Mozilla/5.0 (Windows NT 6.0; rv:40.0) Gecko/20100101 Firefox/40.0' -H 'Accept: text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8' -H 'Accept-Language: en-US,en;q=0.5' -H 'Accept-Encoding: gzip, deflate' -H 'Cookie: GeoIP=FR:Montpellier:43.6109:3.8772:v4; WMF-Last-Access=08-Jun-2015; uls-previous-languages=%5B%22en%22%5D; mediaWiki.user.sessionId=76e0f7a134b1788f' -H 'Connection: keep-alive' -H 'If-Modified-Since: Mon, 08 Jun 2015 11:44:00 GMT' -H 'Cache-Control: max-age=0' --head
HTTP/1.1 200 OK
Server: nginx/1.6.2
Date: Mon, 08 Jun 2015 14:47:35 GMT
Content-Type: text/html; charset=UTF-8
Content-Length: 194933
Connection: keep-alive
X-Content-Type-Options: nosniff
X-Powered-By: HHVM/3.6.1
Content-language: en
X-UA-Compatible: IE=Edge
Vary: Accept-Encoding,Cookie
Content-Encoding: gzip
Last-Modified: Mon, 08 Jun 2015 14:11:24 GMT
X-Varnish: 4043454383, 1734306895 1733988526, 2745213930 2734259812
Via: 1.1 varnish, 1.1 varnish, 1.1 varnish
Accept-Ranges: bytes
Age: 2165
X-Cache: cp1054 miss (0), cp3003 hit (11), cp3030 frontend hit (32)
Cache-Control: private, s-maxage=0, max-age=0, must-revalidate
X-Analytics: page_id=534366;ns=0;https=1;WMF-Last-Access=08-Jun-2015

Content-Encoding: gzip is correctly returned in that case.

According to the Firefox dev tools, this is what's returned:

Age: 3462
Cache-Control: private, s-maxage=0, max-age=0, must-revalidate
Content-Language: en
Content-Type: text/html; charset=UTF-8
Date: Mon, 08 Jun 2015 12:41:48 GMT
Last-Modified: Mon, 08 Jun 2015 11:44:00 GMT
Server: nginx/1.6.2
Vary: Accept-Encoding,Cookie
Via: 1.1 varnish, 1.1 varnish, 1.1 varnish
X-Cache: cp1054 hit (1), cp3003 hit (2), cp3030 frontend hit (35)
X-Firefox-Spdy: 3.1
x-analytics: page_id=534366;ns=0;https=1;WMF-Last-Access=08-Jun-2015
x-content-type-options: nosniff
x-powered-by: HHVM/3.6.1
x-ua-compatible: IE=Edge
x-varnish: 4027695602 4027683338, 1723669700 1723628161, 2700574771 2680389388

One thing that catches my attention in the differences is X-Firefox-Spdy

Sure enough, if I turn off "network.http.spdy.enabled.v3-1" in about:config, the issue goes away. That's a default, though, so all users of recent Firefox would be affected.

Now, as far as I can see, cURL doesn't support SPDY at the moment, meaning I can't reproduce the request from the command line. But it's probably safe to assume at this point that Firefox is unlikely to lie and that when SPDY is enabled, we serve the content uncompressed.

Looking at Chrome's net-internals, Chrome is probably unaffected because unlike Firefox it uses HTTP2 already.

In T101435#1345802, @Gilles wrote:

Looking at Chrome's net-internals, Chrome is probably unaffected because unlike Firefox it uses HTTP2 already.

What's the HTTP2 connection here? Our servers don't yet support HTTP2 regardless.

I think this might contain the answer: https://github.com/jsdelivr/jsdelivr/issues/993#issuecomment-47695878 To be verified. If that turns out to be true, then only Firefox would be affected, not all user agents sending SPDY requests.

Yeah I've been doing some googling as well. The bottom line is that FF doesn't bother sending Accept-Encoding: gzip over SPDY, but we should be gzipping anyways because that's implicit for SPDY. There's some nginx workaround stuff here: http://trac.nginx.org/nginx/ticket/542 . Not directly applicable as our use-case is a little different, but I think we can hack up a solution based on that.

Confirmed by other sources: http://trac.nginx.org/nginx/ticket/542

Everyone fixed this on the server side by serving gzipped content to SPDY connections regardless of Accept-Encoding headers, because the SPDY spec states that SPDY clients must support gzip encoding.

The extra complication here is we only use nginx as a TLS terminator; our actual gzipping is further down the stack in Varnish and the app layer. I'm going to play around with our test server and see if I can find a simple way to get this unbroken.

In T101435#1345855, @BBlack wrote:

In T101435#1345802, @Gilles wrote:

Looking at Chrome's net-internals, Chrome is probably unaffected because unlike Firefox it uses HTTP2 already.

What's the HTTP2 connection here? Our servers don't yet support HTTP2 regardless.

Well, Chrome's network internals state that it's using an HTTP2 connection to en.wikipedia.org. Maybe it's a naming thing and it's actually SPDY? You can see that stuff for yourself by looking at chrome://net-internals/#events and HTTP2_SESSION events

In T101435#1345880, @BBlack wrote:

I'm going to play around with our test server and see if I can find a simple way to get this unbroken.

Thanks!

Yeah I think it's just a naming thing. If you look under chrome://net-internals/#spdy you can see connections to our domains still show "Protocol Negotiated" as "spdy/3.1", whereas connections to google's servers show "h2".

Change 216724 had a related patch set uploaded (by BBlack):
nginx ssl proxies: Force AE:gzip header for all SPDY requests

https://gerrit.wikimedia.org/r/216724

@Gilles - a manual version of the above patch is currently running on https://pinkunicorn.wikimedia.org (ignore the broken images in the page it shows, that's unrelated, it's just a protocol/stack-testing instance of the outer layers). Can you confirm this works?

In T101435#1346004, @BBlack wrote:

@Gilles - a manual version of the above patch is currently running on https://pinkunicorn.wikimedia.org (ignore the broken images in the page it shows, that's unrelated, it's just a protocol/stack-testing instance of the outer layers). Can you confirm this works?

It doesn't seem to work. You can use Firefox Developer Edition and the built-in developer tools to test this. It conveniently shows "Transferred" and actual "Size" next to each other, which lets you see at a glance if it was gzipped, without having to dig into the response headers.

Oops, sorry, just before I asked you to test it, I had commented out part of the solution to double-check something. It's restarted with the full patch in place now :)

In T101435#1346031, @BBlack wrote:

Oops, sorry, just before I asked you to test it, I had commented out part of the solution to double-check something. It's restarted with the full patch in place now :)

Fix confirmed on your test host.

Change 216724 merged by BBlack:
nginx ssl proxies: Force AE:gzip header for all SPDY requests

https://gerrit.wikimedia.org/r/216724

It's deploying to production now, but on the usual slow schedule of puppet and eventual nginx reloads rather than forced. Will update here once they should all be updated (~1.5 hrs or so).

Thanks!