Page MenuHomePhabricator

HTTP/1 vs HTTP/2
Closed, ResolvedPublic

Description

Background

We serve our content with HTTP/2. When we did turn on HTTPS we switched from HTTP/1 to SPDY (the fastest protocol at the moment). With SPDY we could see that HTTP/1 in some cases where faster (T125208). When nginx added support for HTTP/2 we switched to that. But we never did any performance testing to make sure HTTP/2 was faster.

The problems with the current studies on how "fast" HTTP/2 is that either it is CDN companies behind a HTTP/2 solution that prays how good it is or people testing has synthetically tested websites (taking a companies website and turning on HTTP/2 with a proxy and then comparing the onLoad time (see https://www.slideshare.net/LoadImpact/http11-vs-http2-a-performance-analysis for a good "bad" example of not how to do it.) On load has been known for years and years that it is not a good measurement for measure the impact on users.

Since there are no studies out there, it could be that HTTP/1 actually is faster for us. Let us verify that HTTP/2 is the fastest for us.

How we test

Lets talk about which browsers and pages we tests.

Browsers

It's important to test on more than one browser since different browsers have different implementations of HTTP/2 (different prios on requests etc). We test with Chrome 58 (and then 59), Firefox 53, Chrome on emulated mode (to test on mobile) and Chrome 56 on a real Android phone.

In a perfect world we would test on more devices, but let us start with what we have and see if we can see any patterns.

You can force Chrome and Firefox to use HTTP/1. For Chrome you add the command line switch

disable-http2

and for Firefox you add the following Firefox preferences

network.http.spdy.enabled false
network.http.spdy.enabled.http2 false
network.http.spdy.enabled.v3-1 false

The idea is that we can run the exact same test with both HTTP/2 and HTTP/1 and then compare the metrics we collect.

Pages

Test on desktop, emulated mobile (running on desktop, emulating a mobile device) and on a real Android phone.

Desktop

We test each URL 5 times every hour and take the median metric and graph it.

Chrome and Firefox

Chrome only

Emulated mobile (only Chrome, simulated 3G)

We test each URL 5 times every hour and take the median metric and graph it.

Chrome on Android

I don't have setup where I can automatically run Android devices so I'll focus on my Huawei. I'll run 31 on runs on each URL and then take the median.

Metrics

We collect a lot of metrics but let us focus on:

  • First Visual Render - the first time something was changed in the current view port
  • SpeedIndex - the average time at which visible parts of the page are displayed within the view port.
  • Last Visual Change - the last time something was changed in the current view port
  • On load - when the load event is fired in the browser

The first three metrics are focusing on the user experience and the best metrics we have at the moment.

Result

The original data: https://dashboard.sitespeed.io/dashboard/db/http1-vs-http2?orgId=1

At 23/5 16:00 in the graphs I've updated to Chrome 59 that uses the new HTTP/2 prio. see https://twitter.com/patmeenan/status/866996216923660288

Also on the Obama something was pushed/changed that makes the metrics higher a little before midnight 23rd. I'll look into that.

Desktop

Obama first view Chrome

Obama first view Firefox

Obama second view Chrome

Obama second view Firefox

Ingrid_Vang_Nyman first view Chrome

Ingrid_Vang_Nyman first view Firefox

NO data, I need add tests for that.

Elizabeth_I_of_England second view Chrome

Elizabeth_I_of_England second view Firefox

Obama authenticated Chrome

Ingrid_Vang_Nyman authenticated Chrome

Emulated mobile

Barack_Obama

Ingrid_Vang_Nyman

Second view Obama

Second view Ingrid_Vang_Nyman

Chrome on Android

Making 31 one runs, taking the median value of each metric.

URLversionfirstVisualChangeSpeedIndexLastVisualChangeonLoad
https://en.m.wikipedia.org/wiki/Barack_Obamahttp22.71s28316.73s5.21s
https://en.m.wikipedia.org/wiki/Barack_Obamahttp12.29s24155.83s5.09s
https://en.m.wikipedia.org/wiki/Barack_Obama second viewhttp21.25s13395.26s2.04s
https://en.m.wikipedia.org/wiki/Barack_Obama second viewhttp11.31s14085.39s2.08s
https://en.m.wikipedia.org/wiki/Ingrid_Vang_Nymanhttp22.20s22725.48s3.69s
https://en.m.wikipedia.org/wiki/Ingrid_Vang_Nymanhttp11.75s18274.84s3.53s
https://en.m.wikipedia.org/wiki/Elizabeth_I_of_England second articlehttp21.14s11984.78s1.59s
https://en.m.wikipedia.org/wiki/Elizabeth_I_of_England second articlehttp11.19s12474.83s1.65s

Analyze the result

TBD

Event Timeline

Peter created this task.May 23 2017, 1:35 PM
Restricted Application added a subscriber: Aklapper. · View Herald TranscriptMay 23 2017, 1:35 PM
Peter updated the task description. (Show Details)May 23 2017, 2:11 PM
Peter updated the task description. (Show Details)May 24 2017, 11:23 AM
Peter updated the task description. (Show Details)

Result

Here are the result running it for a couple of days. I've missed collecting data for small articles on desktop, I can add that next week so we have a complete view. Remember that Chrome was tested with version 59 not yet stable. When I upgraded to 59 almost every metric got better for HTTP/2.

These are the results from synthetic testing and will not be 100% the same for real users. However we pickup up more relevant metrics using synthetic testing than RUM.

TypePage typeView  BrowserHTTP/1 SpeedIndex/firstVisualChangeHTTP/1 lastVisualChange/onLoad
DesktopLarge articlefirstChrome10-20% faster5-6% slower
DesktopLarge articlefirstFirefox0.5-1% faster1% slower
DesktopLarge articlesecondChrome50% faster5-20% slower
DesktopLarge articlesecondFirefox20% faster9% faster
DesktopSmall articlefirstChrome3-5% slower10% slower
DesktopSmall articlefirstFirefoxMissing dataMissing data
DesktopSmall articlesecondChromeMissing dataMissing data
DesktopSmall articlesecondFirefoxMissing dataMissing data
DesktopArticle no extra JSsecondChrome2-3% slower2-3% slower
DesktopArticle no extra JSsecondFirefox3% slower5% slower
Desktop authenticatedLarge articlefirstChrome30% faster20% slower
Desktop authenticatedSmall articlefirstChrome12 % slower7% slower
Emulated mobileLarge articlefirstChrome30% faster4% slower
Emulated mobileSmall articlefirstChrome+-1%10% slower
Emulated mobileLarge articlesecondChrome30% slower20% slower
Emulated mobileSmall articlesecondChrome2% slower+-3% slower

One extra thing: We had a change the 23/5 between 21-23 CET that completely changed the large articles on mobile. HTTP/1 was 30-40% faster, but after the change HTTP/2 is 30 faster.

Comments

It's super interesting to see that HTTP/2 isn't always faster. In some conditions it seems like HTTP/1 would be favorable from a user experience perspective. Most notable are large articles in in Chrome second view on Desktop, those changes are really big. It's also interesting to see that implementations differ between Chrome and Firefox.

Also been trying to rerun the tests on my android phone for Chrome 59 (beta) but it seems to be some issues with that version and I cannot get through 31 runs in a row, let me see if it could be something with a Chromedriver.

Gilles moved this task from Inbox to Doing on the Performance-Team board.May 26 2017, 4:14 PM
Peter added a comment.May 29 2017, 3:57 AM

Comparing H/1 with H/2 on Facebook page: https://www.webpagetest.org/video/compare.php?tests=170529_23_deb246702c43a2c145b702e91af3939e%2C170529_WX_efc36aa124df7127a4db17d3d78805ca&thumbSize=200&ival=100&end=visual

Seeing the same thing as the others tests (however we get that super fast TTFB that's been hunting us for some time).

Another interesting thing: First contentful and meaningful paint for HTTP/1, it is 1,0 s, that is 600 ms off from when the video shows something, so I think there's still work 2do there. For HTTP/2 it is 400 ms off.

Peter added a comment.May 29 2017, 4:25 AM

Maybe the big difference is dependent on how fast Chrome closes the connections. If we wait 30s before we access the next page, it looks like this: https://www.webpagetest.org/video/compare.php?tests=170529_SD_86a2ac65cd52149520d6465af5183644%2C170529_J0_4d531cf9a19a1558fb87a37a45e4da27&thumbSize=200&ival=100&end=visual

Let us try the same with FF and see what we get.

Peter triaged this task as Normal priority.May 30 2017, 9:37 AM
Peter added a comment.May 30 2017, 2:24 PM

It's definitely the the thing with the second view, I see the same for FF too. Short wait time gives us much faster HTTP/1: https://www.webpagetest.org/video/compare.php?tests=170530_VV_f7482bce0cae1eee1e0c771c262fe27a,170530_50_dc6b6be6bfe759b565dee5882fa43d4d

And for 30 seconds wait time (HTTP/2 is much faster): https://www.webpagetest.org/video/compare.php?tests=170530_3W_8c05531097f886e95037101c4cba4520,170530_H1_f5bdedaa8ebbc186adce404d1aa6b39e

I'll verify this on our own instance too since I got black screens and wide range of TTFB for the public instance.

Gilles added a subscriber: Gilles.Jun 21 2017, 2:08 PM

One theory we came up with discussion this again, is that while we technically support HTTP/2, the various layers we use to do so might get in the way of proper priority management/content interleaving from the server, which is what makes HTTP/2 effective. We have Nginx in front of Varnish, talking to each other in HTTP/1.1, and Nginx takes care of talking in HTTP/2 to the client. It's not obvious how Nginx would manage to do proper prioritization of the content it merely proxies. I think we need to inspect the actual HTTP/2 traffic to see if we are actually sending content in a priority order that makes sense compared to what the client requested, or if we basically send things in the same order HTTP/1.1 would have, just over a single HTTP/2 connection instead of a few HTTP/1.1 connections.

Peter closed this task as Resolved.Sep 12 2017, 2:25 PM

We know it but there's nothing we will do about it for now.

@Peter: If there's nothing we will do that's not really resolved but rather declined?