Optimize prod's resource domains for SPDY/HTTP2
Closed, ResolvedPublic

Description

Now that SPDY has been enabled across our projects (T35890), it's time to start thinking about various optimizations we could do to take more advantage of it.

Right now, connecting to production means estabilishing a lot of different connections with different domains & IPs, which means RTTs wasted to make a TCP & TLS handshake and not taking advantage of SPDY's multiplexing.

It should be noted that:

  • SPDY has a feature called "IP pooling", in which UAs opportunistically multiplex over the same session requests to different domains that a) resolve to the same IP and b) are valid according to the TLS/X.509 certificate presented. The latter is not the case for projects vs. wikimedia.org and would need new certificates, with a non-trivial cost attached to them.
  • Completely undoing domain sharding will have an impact for non-SPDY clients, which would need to be factored in and taken into account as a tradeoff.

So far we have:

  1. bits (T95448) — we should probably fold bits into text. This can happen at least in the edge caching layer (which we'd probably want to do anyway). bits is also sort of a hack MediaWiki-wise with Varnish rewriting URLs, so we could possibly undo all of it with the caveat of undoing an optimization for non-SPDY UAs.
  2. upload — problematic for multiple reasons: at least security separation for third-party data and possibly Wikipedia Zero IP-based traffic separation. upload is also much heavier in traffic than text and the distinction is now an implicit load-balancing technique. A potential merge would present scalability problems with at least LVS & Varnish.
  3. login.wikimedia.org (for checkLoggedIn), meta.wikimedia.org (CentralNotice, gadgets) — theoretically good candidates for SPDY IP pooling, with the caveat of the certificate troubles mentioned above. Those two should already use the same SPDY connection now, but in quick tests I've done they do not appear to do so, although I see a common connection between login/commons. This needs further debugging.
  4. www.mediawiki.org, commons.wikimedia.org, <random projects> (gadgets) — already a performance issue, not much we can do about them; hopefully Gadgets 2.0 can.
faidon created this task.Apr 2 2015, 8:38 PM
faidon updated the task description. (Show Details)
faidon raised the priority of this task from to Needs Triage.
faidon added subscribers: faidon, ori, BBlack.
Restricted Application added a subscriber: Aklapper. · View Herald TranscriptApr 2 2015, 8:38 PM

I've been looking forward to folding bits back into text for a while now anyways. On many levels, it's an appropriate move at this point in time regardless of SPDY, IMHO. It reduces reduces varnish config and mgmt complexity (it's the only prod 1layer cluster layout), reduces cluster count, makes better use of current CPU/mem hardware (if we buy a few SSDs and fold bits machines back into other clusters), etc. It's also probably the easiest of all of these cases to solve for SPDY.

Also, I tend to agree that upload should be left alone for now as there are too many other considerations there. If we ever get down to just two connections for upload and everything else, then maybe we can revisit it at that time and decide whether it's worth messing with it. For now there's other lower-hanging fruit to go after.

Andrew triaged this task as Normal priority.Apr 6 2015, 4:30 PM
Andrew set Security to None.
GWicke added a subscriber: GWicke.Apr 8 2015, 3:47 PM
mark added a subscriber: mark.Apr 8 2015, 3:47 PM
Krinkle moved this task from Tag to Doing on the Performance board.May 18 2015, 5:54 PM
Krinkle moved this task from Inbox to Doing on the Performance-Team board.
Krinkle updated the task description. (Show Details)Jul 10 2015, 11:10 AM
Restricted Application added a subscriber: Matanya. · View Herald TranscriptJul 10 2015, 11:10 AM
Peter added a subscriber: Peter.Aug 19 2015, 6:49 PM

Any pending tasks here or is this resolved?

Well this basically got solved along the way while doing other things. We've flipped back to using a unified cert that covers all the projects (including *.wp.o + *.wm.o), and so we get coalesce for all of the IPs that map to text-lb already, which includes bits now.

We're missing SPDY coalesce for upload.wm.o for images ref'd in projects' page outputs, but that's a trickier problem and we're not ready to move on any solution there. Arguably (a) it can wait and it's not that critical and (b) solving it too early hurts perf for non-SPDY clients, too. I think we can go ahead and close this ticket in favor of a possible new one about eventually looking at the specific upload problem.

I should note that mobile probably still has some coalescing to gain, but it's not the driver for those solutions anyways, which will get addressed as part of: T109286

Krinkle moved this task from Doing to Tag on the Performance board.Aug 29 2015, 12:35 AM
Krinkle moved this task from Doing to Next-up on the Performance-Team board.Sep 4 2015, 2:01 AM
Dzahn closed this task as Resolved.Oct 21 2015, 6:04 AM
Dzahn claimed this task.
Dzahn added a subscriber: Dzahn.

Well this basically got solved along the way while doing other things. ... I think we can go ahead and close this ticket in favor of a possible new one about eventually looking at the specific upload problem.

Dzahn added a comment.EditedOct 21 2015, 6:08 AM

new one about eventually looking at the specific upload problem.

T116132

BBlack moved this task from Triage to Done on the Traffic board.Nov 30 2015, 6:02 PM