Page MenuHomePhabricator

SPDY vs HTTP/1.1 on slow connections, what's the impact of start rendering?
Closed, ResolvedPublic

Description

In T124874 we tested different ways of making mobile pages faster on 2G and we wanted to have baseline how fast the page load so we tested also on HTTP/1. And guess what happened? According to WebPageTest HTTP/1 started to render the Obama page much faster than SPDY! We tested the page using a throttled connection (2GFast) and the outcome was that the rendered blocking CSS was delivered much later on SPDY.

Using SPDY it looked like this:

And HTTP/1 like this:

Could something be wrong in WebPageTest? When we tested we used the --use-spdy=false on Chrome to get HTTP/1 connections. We also tested using Firefox using SPDY and could see that we got a late renderer there too. Using IE 11 (IE 11 on WPT only supports HTTP/1.1) we could see that it was much faster. It really looks like SPDY is slower on slower connections. But still, there could be maybe some overall problem with SPDY on WPT.

Lets test it locally.

Part 1 - SPDY

I'm using Chrome Canary (because the network throttler is more configurable there, however setting the same values as WebPageTests 2GFast didn't work for me, I never got a first paint). I start Canary, open Developer Tools and choose Regular 2G (that is way faster than on WebPageTest but lets use it for now). I make sure I disable the cache and set the device mode to mobile and then in the network panel, I choose to show the protocol (to make sure it uses SPDY/1.1). And then access the Obama page https://en.m.wikipedia.org/wiki/Barack_Obama:

The same moment I press enter I also start the stopwatch on my phone (yep I'm old school). When the page starts to render, I stop the watch. And then check the first paint from Chrome:

console.log((window.chrome.loadTimes().firstPaintTime * 1000) - (window.chrome.loadTimes().startLoadTime*1000));

I also do the same thing throttling the connection as GPRS (yep that is really slow I know but I want to see what happens on a really slow connection).

Part 2 - HTTP 1.1

I use the exact same setup except that I close down Chrome Canary and open it using the command line with the switch to disable SPDY. On my local Canary is located /Applications/Google Chrome Canary.app/Contents/MacOS and I start it like this:

$ ./Google\ Chrome\ Canary --use-spdy=off

I follow the same procedure and verify the protocol in the dev tools panel, yep it is 1.1.

The result

Testing it locally gave me this result.

ProtocolConnection typeThe stopwatchChrome first paint
SPDYRegular 2G6.0 s5.2 s
SPDYGPRS27 s26.4 s
HTTP/1.1Regular 2G3.6 s2.9 s
HTTP/1.1GPRS16.4 s15.9 s

With the current setup we have HTTP/1.1 starts render the page faster when we have a slow connections. How can we make SPDY faster?

Questions

Things we should find out:

  • Is the way I test wrong in any way? Should we try something else?
  • Could we tweak SPDY so that the CSS is delivered earlier? Is there an easy win out there?
  • Will this also exists when we switch to HTTP/2?
  • Does the timings in any way reflects the numbers we fetch with the Navigation Timing metrics from real users (a.k.a is this a real problem)?

Event Timeline

Peter created this task.Jan 29 2016, 1:34 PM
Peter claimed this task.
Peter raised the priority of this task from to Needs Triage.
Peter updated the task description. (Show Details)
Peter added a project: Performance-Team.
Peter added subscribers: Peter, Krinkle, ori.
Restricted Application added subscribers: StudiesWorld, Aklapper. · View Herald TranscriptJan 29 2016, 1:34 PM
Peter added a subscriber: Gilles.Feb 1 2016, 7:36 PM
ori reassigned this task from Peter to Gilles.Feb 1 2016, 7:43 PM
ori triaged this task as High priority.
ori moved this task from Inbox to Doing on the Performance-Team board.
ori set Security to None.
Gilles added a comment.EditedFeb 1 2016, 9:38 PM

Instead of relying on Chrome's built-in traffic shaping, I've decided to use Network Link Conditioner on EDGE setting. I ran a first "warm up" request, then ran 5 actual run requests using "empty cache and hard reload":

First paint as reported by Chrome with SPDY turned off: 2.6

First paint as reported by Chrome with SPDY turned on: 8.1

I've tried reducing this to a smaller test case, by having an html documents that only points to the css and js in the head, but that behaves the same with and without SPDY. It might have to do with the browser's asset prioritizing code, but unless we can reduce this issue to a test case, it might be difficult to figure out if we can do anything about it.

Testing an empty article (https://en.wikipedia.org/wiki/User:GDubuc_(WMF)/sandbox/empty) under the same conditions as before. With SPDY turned off: 2.6 With SPDY turned on: 2.6

Same test on the mobile site (https://en.m.wikipedia.org/wiki/User:GDubuc_(WMF)/sandbox/empty) under the same conditions as before. With SPDY turned off: 2.2 With SPDY turned on: 2.1

Testing an article with a large image (https://en.m.wikipedia.org/wiki/User:GDubuc_(WMF)/sandbox/obama) under the same conditions as before. With SPDY turned off: 2.2 With SPDY turned on: 2.1

Testing a decently sized real article (https://en.m.wikipedia.org/wiki/Octopus) under the same conditions as before. With SPDY turned off: 2.8 With SPDY turned on: 2.8

So far this seems to point to something the Obama article sports, which the Octopus article doesn't have. But at a glance, the CSS and JS requests look the same between those two articles. The only difference I see is that the Obama article as twice as many images, which also weigh about twice the total size.

@Peter, can you reproduce the issue on the Octopus enwiki article? I can't.

Peter added a comment.Feb 2 2016, 6:20 PM

@Gilles yep I get almost the same numbers on the Octopus article, there's no measurable difference between SPDY & HTTP/1. It seems it only happens with articles with a lot of images. That's cool then that it will be fixed by lazy loading them. Or inline the CSS. Or push the CSS (if that ever will be available).

Interesting that SPDY/HTTP2 not always will be the fastest alternative and that some of the old fixes (inlining) still will be the best way in some cases.

I guess than that we can have missed this regression when we did the switch since the loadEventEnd happen so late for articles with many images on slow connections. But the thing is, the loadEventEnd happen so late on HTTP/1 also so that we didn't measure it before either :) Just good to know that in some cases the CSS is downloaded slower for SPDY.

Well, let's not jump to conclusions. The Octopus article has a lot of images too. It's half as many as the Obama article, but it's still a lot, and the Octopus article is completely unaffected. I'd like to find and test other articles with as many pictures as the Obama one to confirm that theory.

https://en.m.wikipedia.org/wiki/John_F._Kennedy looks like a good first candidate. Obama has 87 image requests totaling 1.1MB. JFK 79 image requests totaling 1.3MB.

The JFK article does seem to have the same issue as the Obama one. Which would indeed indicate a threshold for this to start happening. We have to keep in mind that this is Chrome-specific, though, and might be worth filing an issue with Chromium as it is surprising that the resource prioritization seems to behave differently beyond a certain amount of images.

I'll find other examples and file a bug report to see what the Chromium devs think.

Peter added a comment.Feb 3 2016, 1:27 PM

I tested in WebPageTest and could see that it is faster in IE 11 (uses HTTP/1) than Firefox (SPDY). I think we could test it locally for Firefox disabling SPDY with network.http.spdy.enabled and network.http.spdy.enabled.v3-1.

We don't get a first paint in FF but maybe the difference is so big so we will still see it.

Peter added a comment.Feb 3 2016, 1:43 PM

I couldn't reproduce locally on Firefox. I use Edge on the Network link conditioner, I get the same timings but I'm not sure it really works when I turn off SPDY, does it work for you @Gilles?

When I turn it off, I still get the internal extra Firefox header: X-Firefox-SPDY but I don't get our CP=H2 cookie?

Gilles added a comment.Feb 3 2016, 4:03 PM

@Peter it seems like you were able to run Chrome with and without SPDY on WPT, how did you do that? I would be nice to link to WPT runs for the bug report.

Gilles added a comment.Feb 3 2016, 5:29 PM

I couldn't reproduce locally on Firefox. I use Edge on the Network link conditioner, I get the same timings but I'm not sure it really works when I turn off SPDY, does it work for you @Gilles?
When I turn it off, I still get the internal extra Firefox header: X-Firefox-SPDY but I don't get our CP=H2 cookie?

With https://developer.mozilla.org/en-US/docs/Mozilla/Debugging/HTTP_logging I can confirm that those flags work, but you need to restart firefox after changing them for the change to take effect. When I disable them and restart I don't see the X-Firefox-Spdy header anymore, though. Maybe you didn't restart when you tried?

Gilles added a comment.Feb 3 2016, 6:14 PM

Ok, on Firefox I'm seeing the same problem on the Obama article using your patent-pending stopwatch technique. With Spdy turned off, the page appears in 7-8 seconds, with Spdy turned on a normal run takes around 26 seconds (sometimes up to 50, the lowest was 13). Firebug lets me see what's going on in quite a bit of detail, and it's really getting most of the images before the stylesheet/js requests.

It's quite possible that the issue is on our end with nginx. When bombarded with too many requests over the same Spdy connection, it might not serve the content back in the order it was requested or something like that?

It's definitely very interesting to see that Firefox is affected as well. It makes my theory of the browser being responsible a lot less likely.

Peter added a comment.Feb 4 2016, 8:38 AM

@Gilles argh, I missed to restart the browser, thanks.

FYI I have the runs in another task T124874. Using HTTP/1 and SPDY for Chrome. Now when we seen that it happens both on Firefox (good work!) and Chrome maybe we don't need them. But for reference, if you want to pass parameters to Chrome in WebPageTest, you have an extra field on the Chrome tab.

Gilles added a comment.Feb 4 2016, 8:56 AM

I've taken a deep dive in chrome://net-internals and I think I know what's going on.

The SPDY protocol has a priority system for streams within sessions. On our page, there are two SPDY sessions, one for en.m.wikipedia.org and the other for uploads.wikimedia.org. Within these sessions there are streams (requests), each of which have priorities. Priority values go from 0 to 7, 0 being the highest priority.

On en.m.wikipedia.org the page itself (the html) is given a priority of 0. The CSS a priority of 1, the async startup JS a priority of 3, images a priority of 4.

On uploads.wikimedia.org all images have a priority of 4.

Here's the chronology of what happens.

en.m.wikipedia.org receives the request for the page itself, with the highest priority possible. Shortly after the bits of the page start trickling down and the browser can see the <head>, it requests the CSS and the startup JS. However, the streams for those have a lower priority than the page itself. Which means that the only data coming down the SPDY session is the page itself until it's fully downloaded. That sounds crazy, but it makes sense given the priorities that are set.

The spec states:

The creator of a stream assigns a priority for that stream. Priority is represented as an integer from 0 to 7. 0 represents the highest priority and 7 represents the lowest priority.
The sender and recipient SHOULD use best-effort to process streams in the order of highest priority to lowest priority.

I think that nginx is taking that very literally and serves the entire contents of all assets that have priority 0 first, then priority 1, etc.

So what's I'm seeing is that after the whole page body itself, the CSS is streamed and then the startup JS.

"so far so good"

Meanwhile, uploads.wikimedia.org has its own competing session. The priorities are within a session, not across them. Which means that while thumbnails have a lower priority, their session is on the same level as en.m.wikipedia.org's and will compete for bandwidth!

Peter added a comment.Feb 4 2016, 9:02 AM

The images and the CSS is on different connections but downloaded at the same time if I understand the waterfall and connection chart correctly.

It's a little hard hard to see on the waterfall but it looks like the browser is downloading the HTML/CSS/JS and the Obama jpg at the same time, so they are competing on the limited bandwith. On HTTP/1 it's only CSS and the HTML:

But it's hard to say just with the waterfall, I don't know how correct it is with SPDY/H2. The download time (blue) is much longer when using SPDY.

Peter added a comment.Feb 4 2016, 9:08 AM

@Gilles hehe. Yes, you are right.

Gilles added a comment.Feb 4 2016, 9:10 AM

I think that given that the page will always get a priority of 0, the ideal scenario would be that all external assets would be on a different domain than the page itself, and all on the same domain, for priorities to work between them.

Yes, loading images async will help, but it won't solve everything, as the page body will always be streamed entirely before the CSS. Even in that scenario, we would need to serve the CSS/JS from a separate domain to work around that.

Now, in absolute terms it seems wasteful to me to purposefully have two domains to sidestep the SPDY prioritization shortcomings. It seems backwards because of the extra DNS requests, etc. In an ideal world you'd have a single SPDY session for everything. I think I understand why Chrome had no choice but set priority 0 for the page itself, though. If anything linked from the page had a higher priority than the page itself, the download of the page would literally halt until that other asset is fully downloaded. We might blame nginx a bit in this, taking the priorities a bit too literally, it could very well mix a little bit of data from the next priority with the one it's processing as its main priority. Overall it seems to me like there is a fatal flaw here, in that the browser cant adjust dynamically to what happens. It has to set a priority when it first requests something and then has to live with the consequences. Chrome made the least bad choice given those conditions. Clearly, though, the low bandwidth scenario on a large html page wasn't taken into account in this protocol design.

There is probably a case to be made for making everything async, including the article content. If our page body was tiny, this issue would become irrelevant and we could safely have everything served from the same domain on the same SPDY session. It's something to keep in mind for the ambitious ideas like a JS-only mobile site using SW, etc.

In the short term, we should probably consider having the JS, CSS and thumbnails served from the same domain. I think it's the smallest effort to fix that issue, and it will benefit high bandwidth clients as well.

Now the question is whether HTTP/2 has the same properties... I'll try to look into that.

Gilles added a comment.EditedFeb 4 2016, 9:18 AM

A-ha, HTTP/2 looks smarter in that respect, in that the client can adjust priorities as it goes:

A client can assign a priority for a new stream by including prioritization information in the HEADERS frame (Section 6.2) that opens the stream. At any other time, the PRIORITY frame (Section 6.3) can be used to change the priority of a stream.

Which means that browsers get more control and might decide to temporarily lower the page body priority will they get the head CSS, etc. Whether browsers actually do that, though, is a different matter. At least if the same issues show up on our large pages, we could report it and they might actually have a way to do something about it.

Also:

Streams can be prioritized by marking them as dependent on the completion of other streams (Section 5.3.1). Each dependency is assigned a relative weight, a number that is used to determine the relative proportion of available resources that are assigned to streams dependent on the same stream.

And it goes on... The priority/dependency capabilities of the protocol seem very advanced.

Given how different and smarter HTTP/2 seems in that respect, I think that the likelihood that we'll run into the same problem there is probably low. And if we do, we can report it to browser vendors who actually have control over that logic and can do something about it.

Gilles added a comment.Feb 4 2016, 9:21 AM

Another short-term action could be to disable SPDY in locations where we know bandwidth is low, like those geographic areas where we started serving thumbnails that are more compressed.

Peter added a comment.Feb 4 2016, 10:05 AM

I think it's not only on the browser side, I'm mean it's really complex nowadays with the browser prio and then check that the server is really following that. Even harder when/if we start pushing resources. How do we know that it works as we expect? :)

What I think we should do though is start testing for HTTP/2 so we know that it will work as we think?

I think it would be duplicated effort to try and reproduce these conditions with an HTTP/2 server and later actually try to set up HTTP/2 in labs or prod. We can just remember to test these condittions when deploying HTTP/2 becomes a priority. You should file a separate task if you want to keep track of that, this task is about the SPDY problem. That's high priority, exploring hypothetical scenarios about HTTP/2 isn't, imho.

We should discuss what we want to do about the SPDY problem in today's meeting.

Note that inlining above-fold CSS as discussed in T124966 will also avoid this issue, no matter which protocol is used.

Gilles added a comment.Feb 5 2016, 1:28 PM

A few more WPT SPDY vs HTTP1 runs to verify our assumptions. This time I'm going to look at real connections, as so far the traffic shaping has always been artificial, either at the browser or OS level. Which means that this artificial throttling might play a role in what we've seen.

2G/3G connection in Bangalore, Obama article on mobile site
http://www.webpagetest.org/video/compare.php?tests=160205_95_FJ9,160205_S9_FHT

3G connection in Rio, Obama article on mobile site
http://www.webpagetest.org/video/compare.php?tests=160205_S1_F29,160205_H3_F05

3G connection in Sao Paulo, Obama article on mobile site
http://www.webpagetest.org/video/compare.php?tests=160205_BS_FGY,160205_VA_FGH

LTE connection in Tokyo, Obama article on mobile site
http://www.webpagetest.org/video/compare.php?tests=160205_TD_FFK,160205_6R_FFA

LTE connection in SF, Obama article on mobile site
http://www.webpagetest.org/video/compare.php?tests=160205_FV_HG7,160205_RM_HG8

2G/3G connection in Bangalore, Octopus article on mobile site
WPT keeps failing/being buggy on that one, no idea why. The Bangalore instance seems to be hammered with tests compared to the other ones.

3G connection in Rio, Octopus article on mobile site
http://www.webpagetest.org/video/compare.php?tests=160205_P6_H55,160205_HF_H4W

3G connection in Sao Paulo, Octopus article on mobile site
http://www.webpagetest.org/video/compare.php?tests=160205_GJ_H74,160205_53_H6Y

LTE connection in Tokyo, Octopus article on mobile site
http://www.webpagetest.org/video/compare.php?tests=160205_9E_H7X,160205_TN_H7K

LTE connection in SF, Octopus article on mobile site
http://www.webpagetest.org/video/compare.php?tests=160205_1B_H94,160205_MY_H8W

It appears that under those real conditions, having SPDY turned off always results in the first paint being better or equal to having SPDY turned on. SPDY sometimes wins the race to having the page fully loaded, but not always.

I'm going to keep poking at the underlying issue to understand it better, but from a black box testing perspective, it seems clear that we should probably turn off SPDY as it currently works. We can revisit it if our setup has changed in ways that could make the issue go away, like inlining the top CSS.

Gilles added a comment.Feb 5 2016, 2:17 PM

I've stared some more at Chrome's net-internals with SPDY, without throttling my connection.

In that case I do see the order of assets sometimes swapped, for example the startup JS served before the CSS, but that happens over the course of 2ms, so it's hard to reason about in absolute terms. What seems consistent, though, is that the entirety of a given asset is appears on the stream before the next one. In that specific example it's the whole article html first, then the startup JS, then the CSS. Given that the page always start first (by definition), it still seems like a large article body gets in the way of the head CSS being read and processed as early as it can.

The waterfall graph is misleading, because it seems to happen on an entirely different level in the browser, and includes a lot of parallelized processing (gunzip content as it arrives, etc.). The overlaps there do make sense because of that bigger context, but they are unseen in the SPDY stream's log.

I ran the test one more time with Network Link Conditioner's 3G profile. This time I saw the headers alone of the CSS and JS come in while the body was being downloaded, then nothing until the whole html was done and then the whole of the CSS followed by the whole of the JS. Still no overlap of HTTP bodies, though.

Finally, I looked at facebook, twitter and google with simulated 3G. They all displayed the same behavior of having the whole html loaded before external dependencies would start being processed (even the waterfall fully agrees). This suggests that this part of the problem will still exist with HTTP/2 and that we should strongly consider inlining the CSS if we want to achieve the best first paint time possible. It also means that turning HTTP/2 on could be equally counter-productive as the way SPDY currently behaves in regards to first paint.

Gilles closed this task as Resolved.Feb 5 2016, 2:34 PM

I'm closing this, as the next practical step is filed: T125979

We can reopen if production experiments don't confirm those findings on webpagetest.

This seems to be a known issue, with some servers like h2o offering server-side work-arounds for Chrome specifically: http://www.slideshare.net/kazuho/h2o-making-http-better

This seems to be a known issue, with some servers like h2o offering server-side work-arounds for Chrome specifically: http://www.slideshare.net/kazuho/h2o-making-http-better

Neat, and it seems like it's doing the right thing and parsing the HTML to find out which CSS/JS needs to have that higher priority: https://h2o.examp1e.net/configure/http2_directives.html#http2-reprioritize-blocking-assets