Page MenuHomePhabricator

Investigate TCP Fast Open for tlsproxy
Closed, ResolvedPublic

Description

TCP Fast Open has the potential to further reduce latency for at least some TLS connections. With TLS there's also no idempotency concerns like there are for using it with plain HTTP, as the fastopen-sent data would just be the TLS ClientHello. Our current cache cluster kernels and nginx builds support it, requiring sysctl adjustments and an nginx config change. Various concerns/work that need addressing before trying to really enable it:

  1. Is client-side adoption percentage high enough to matter yet? Right now probably fairly low, but growing. Linux+Chrom(e|ium) can do it by default on newer kernels, and Android 5 can do it. El Capitan Macs (Safari? Chrome?) may start trying it in the fall.
  1. How do we appropriately tune the limit for the outstanding fastopen queue per socket? (in nginx terms, this is the N in the fastopen=N parameter). Too high has DoS implications both for us and as a potential reflector. Too low and we may miss legit TFO due to our high SYN concurrency.
  1. Is LVS a factor here for TFO compatibility, and does it (or should it) pay attention to the cookies?
  1. We'd need to synchronize the TFO cookie key across machines in a cluster and rotate them periodically, or rely on client source IP hashing and only periodically regenerate locally per-machine. This is somewhat similar to the issues with RFC5077 secrets but considerably less security-critical; leaking it opens us to easier DoS, but does not affect TLS security of legitimate client traffic. We'd need to generate a random key and distribute it securely and periodically, etc. Everything about this is dependent on answering the LVS question above too.

Related Objects

StatusAssignedTask
Resolvedema
OpenBBlack
ResolvedBBlack
ResolvedDzahn
Resolvedezachte
ResolvedBBlack
ResolvedBBlack
ResolvedBBlack
ResolvedBBlack
ResolvedBBlack
ResolvedBBlack
ResolvedBBlack
ResolvedBBlack
ResolvedPchelolo
ResolvedPchelolo
Declinedvalhallasw
ResolvedWhatamidoing-WMF
DuplicateNone
OpenVgutierrez
ResolvedBBlack
ResolvedNone
ResolvedNone
DuplicateNone
ResolvedKrenair
ResolvedBBlack
ResolvedMarcoAurelio
ResolvedKrenair
Resolvedscfc
ResolvedVgutierrez
ResolvedVgutierrez
ResolvedVgutierrez
OpenVgutierrez
ResolvedVgutierrez
DeclinedNone
Resolved AlexMonk-WMF
Resolvedfaidon

Event Timeline

BBlack created this task.Aug 12 2015, 2:08 PM
BBlack raised the priority of this task from to Normal.
BBlack updated the task description. (Show Details)
BBlack added a project: Traffic.
BBlack added a subscriber: BBlack.
Restricted Application added a project: acl*sre-team. · View Herald TranscriptAug 12 2015, 2:08 PM
Restricted Application added a subscriber: Aklapper. · View Herald Transcript
ema added a subscriber: ema.Feb 26 2016, 12:08 PM

As of when the caches switch to Linux 4.4.2 kernels (coming soon), they'll have the updates for the real (not experimental) TCP Fast Open IANA option code from RFC 7413 ( kernel commit ). Given that we're still hashing traffic for HTTPS and I don't see us changing that anytime soon, we could probably go ahead and rely on that for now and just do machine local (re-)generation of cookies, too, and later upgrade to cluster-synchronized cookies when if we make a mechanism available for RFC5077 as well.

Change 295331 had a related patch set uploaded (by Ema):
tlsproxy: enable client/server TFO support in the kernel

https://gerrit.wikimedia.org/r/295331

ema added a comment.Jun 21 2016, 11:05 AM

So, here are a few findings so far.

tshark can be used to detect SYN packets with a TFO cookie request:

tshark -f 'tcp[tcpflags] & tcp-syn != 0' -Y 'tcp.options.experimental==fe:04:f9:89'

The output is something like this (note the last field, TFO=R):

88274  20.988901 92.110.255.XXX -> 91.198.174.192 TCP 78 49292→443 [SYN] Seq=0 Win=27200 Len=0 MSS=1360 SACK_PERM=1 TSval=56874279 TSecr=0 WS=128 TFO=R
  1. Is LVS a factor here for TFO compatibility, and does it (or should it) pay attention to the cookies?

It looks like LVS does not interfere with TFO compatibility. I've enabled TFO in chrome and confirmed that the cookie request reached cp3xx properly.

  1. Is client-side adoption percentage high enough to matter yet?

TFO support is available on chrome on linux but disabled by default, at least on the version I've tested (47.0.2526.106). It can be enabled under chrome://flags/.

I've monitored the SYN packets received by an esams cache node for a little bit (~5 minutes) and my SYN was the only one with TFO. So that might give an initial, partial answer to 1): client-side adoption is very low at the moment, but will grow as soon as clients start enabling the option by default.

Apple-wise, iOS 9 and OS X 10.11 support TFO: https://www.nanog.org/sites/default/files/Paasch_Network_Support.pdf. Also see http://devstreaming.apple.com/videos/wwdc/2015/719ui2k57m/719/719_your_app_and_next_generation_networks.pdf (starting from page 79).

On Windows, edge supports TCP Fast Open since Windows 10 Preview build 14352 (enabled by default apparently): https://developer.microsoft.com/en-us/microsoft-edge/platform/changelog/desktop/14352/

Change 295331 merged by Ema:
tlsproxy: enable client/server TFO support in the kernel

https://gerrit.wikimedia.org/r/295331

ema added a comment.Jun 22 2016, 3:23 PM

A few more things tools-wise.

TFO support has been added to curl in version 7.49.0: --tcp-fastopen. Unfortunately Debian sid ships an older version of curl, I've opened a bug for that.

httping supports TFO (-F) and it looks like a cool program!

Examples of simple TFO-enabled clients and servers can be found here even though they don't build properly. Pull request opened.

netstat -s | grep -i fastopen returns some TFO-related stats, but it would be nice to have access to more information! Which cookies are we storing, for which IPs? Any additional info we cached? For example, we sent a SYN with cookie request and got no SYN/ACK back. Perhaps a middlebox swallowed it? Negative responses must be cached so it would be interesting to somehow inspect the current status of cached TFO information.

ema added a comment.EditedJun 23 2016, 1:49 PM

The initial portion of the 3WHS can be used to check whether a remote TCP server supports TFO. For example, with scapy:

from scapy.all import sr1, IP, TCP

dst = "en.wikipedia.org"
dport = 443

res = sr1(IP(dst=dst)/TCP(dport=dport,flags="S",options=[('TFO', '')]), verbose=False)
print 'TFO' in dict(res[1].options)

Note that also in this case the version of scapy in Debian sid does not include TFO support. Grab it from github.

Also important: to try reproducing the whole 3WHS and establish a connection it's necessary to avoid sending RSTs due to unsolicited SYN/ACK with something like:

iptables -A OUTPUT -p tcp --tcp-flags RST RST -j DROP

The reason for the resets is that we're crafting a SYN packet with scapy that the kernel is not really aware of. It will get confused upon reception of a SYN/ACK part of an handshake it didn't initiate and reply with an RST. Again, this is not necessary when probing for TFO support with the python snippet above, but good to keep in mind for more advanced tests.

Change 295783 had a related patch set uploaded (by Ema):
tlsproxy: enable TCP Fast Open

https://gerrit.wikimedia.org/r/295783

Change 295783 merged by Ema:
tlsproxy: enable TCP Fast Open

https://gerrit.wikimedia.org/r/295783

Change 295810 had a related patch set uploaded (by Ema):
tlsproxy: only enable TFO on default_server

https://gerrit.wikimedia.org/r/295810

Change 295810 merged by Ema:
tlsproxy: only enable TFO on default_server

https://gerrit.wikimedia.org/r/295810

Mentioned in SAL [2016-06-24T09:59:28Z] <ema> nginx rolling restart to enable TFO on all tlsproxies (T108827)

ema added a comment.Jun 24 2016, 10:29 AM

We're currently running with TCP Fast Open enabled on all tlsproxies, limiting the number of concurrent pending TFO requests to 150 to mitigate the risk of Resource Exhaustion Attacks by SYN Flood with Valid Cookies.
See the relevant commits.

netstat -s | grep TCPFastOpen returns a bunch of TFO-related metrics. TCPFastOpenPassive is the number of successful inbound TFO connections, useful to get an idea of client support. Also, we need to keep an eye on TCPFastOpenListenOverflow, which gets incremented when the above mentioned limit of concurrent pending TFO requests is exceeded.

Change 295900 had a related patch set uploaded (by Ema):
diamond TCP collector: add TFO-related metrics

https://gerrit.wikimedia.org/r/295900

Change 295925 had a related patch set uploaded (by Ema):
tlsproxy: document safe/unsafe TFO usage

https://gerrit.wikimedia.org/r/295925

Change 295900 merged by Ema:
diamond TCP collector: add TFO-related metrics

https://gerrit.wikimedia.org/r/295900

Change 295925 merged by Ema:
tlsproxy: document safe/unsafe TFO usage

https://gerrit.wikimedia.org/r/295925

Change 297418 had a related patch set uploaded (by Ema):
lvs: switch port 80 loadbalancing to sh for cache_ services

https://gerrit.wikimedia.org/r/297418

ema added a subscriber: faidon.Jul 5 2016, 3:34 PM

Change 297418 had a related patch set uploaded (by Ema):
lvs: switch port 80 loadbalancing to sh for cache_ services
https://gerrit.wikimedia.org/r/297418

Either this ^ or we find a way to share tcp_fastopen_key among servers. Clients do change their IP address so sharing the key would have its advantages, but it introduces complexity. @BBlack, @faidon: comments?

faidon added a comment.Jul 5 2016, 4:25 PM

So, first of all, switching port 80 to sh isn't a bad option, given that 443 uses sh.

That said, what is the use case for using TCP Fast Open for port 80? I would guess that every client that supports TFO support HSTS, and every client that supports HSTS will only hit port 80 ~once in its lifetime (so thus the TFO cookies would not be used ever).

Are the TFO cookies shared across different ports and if so, is the use case you're thinking of the speed up of the initial port 80 -> port 443 redirect? Isn't that a relatively rare event, especially for new-enough clients that have TFO support (and thus likely HSTS/HSTS preload as well)?

ema added a comment.Jul 6 2016, 10:09 AM

Are the TFO cookies shared across different ports and if so, is the use case you're thinking of the speed up of the initial port 80 -> port 443 redirect?

Yes.

Isn't that a relatively rare event, especially for new-enough clients that have TFO support (and thus likely HSTS/HSTS preload as well)?

Absolutely. I wasn't thinking of HSTS. Thanks!

ema claimed this task.Jul 8 2016, 9:51 AM
ema moved this task from Triage to In Progress on the Traffic board.
ema set Security to None.

I'd like to share keys in the long run, but I think sh for port 80 is the right move for now. It will also clear up confusion on our TFO success/fail stats in general. Key sharing for this is very similar to key sharing for TLS session tickets - they'll both need a secure memory-only rotation mechanism for small key data across the global clusters. We can probably tackle both (later) with some shared coding/infrastructure...

Another thing just occurred to me though - until we switch port 80 to nginx or patch our varnish, we don't have TFO support on port 80 regardless, as varnish doesn't set it on the socket. It may or may not help much to make the sh switch until then.

ema added a comment.Jul 11 2016, 4:22 PM

Another thing just occurred to me though - until we switch port 80 to nginx or patch our varnish, we don't have TFO support on port 80 regardless, as varnish doesn't set it on the socket. It may or may not help much to make the sh switch until then.

TFO support has been added to varnish 4.1.3, but yeah for the clusters running 3.x we would either have to patch varnish or switch port 80 to nginx.

I don't particularly object into either moving port 80 to sh or to nginx, but I don't think that TFO on port 80 will make any kind of performance impact — at least as I understand it so far.

@faidon - so far we've seen TFO stats showing more TFO failures than successes, so we're looking for reasons why TFO so commonly fails when attempted. That port 80 isn't sh is a logical explanation for how clients are getting TFO failures if they're switching between 80 and 443 even briefly for some unknown reason.

Change 297418 merged by Ema:
lvs: switch port 80 loadbalancing to sh for cache_ services

https://gerrit.wikimedia.org/r/297418

Mentioned in SAL [2016-08-11T17:48:59Z] <ema> switched LVS schedulers for text, upload, maps and misc port 80 to source hash scheduling T108827

ema closed this task as Resolved.Aug 15 2016, 9:43 AM

The high rate of failed incoming TFO connections in esams seems to have stopped since we switched the load balancers to sh for port 80 too.

TFO has been enabled for a while now without causing any particular issue. Closing this ticket.