# Background
We serve our content with HTTP/2. When we did turn on HTTPS we switched from HTTP/1 to SPDY (the fastest protocol at the moment). With SPDY we could see that HTTP/1 in some cases where faster (T125208). When nginx added support for HTTP/2 we switched to that. But we never did any performance testing to make sure HTTP/2 was faster.
The problems with the current studies on how "fast" HTTP/2 is that either it is CDN companies behind a HTTP/2 solution that prays how good it is or people testing has synthetically tested websites (taking a companies website and turning on HTTP/2 with a proxy and then comparing the onLoad time (see https://www.slideshare.net/LoadImpact/http11-vs-http2-a-performance-analysis for a good "bad" example of not how to do it.) On load has been known for years and years that it is not a good measurement for measure the impact on users.
Since there are no studies out there, it could be that HTTP/1 actually is faster for us. Let us verify that HTTP/2 is the fastest for us.
# How we test
Lets talk about which browsers and pages we tests.
## Browsers
It's important to test on more than one browser since different browsers have different implementations of HTTP/2 (different prios on requests etc). We test with Chrome 58 (and then 59), Firefox 53, Chrome on emulated mode (to test on mobile) and Chrome 56 on a real Android phone.
In a perfect world we would test on more devices, but let us start with what we have and see if we can see any patterns.
You can force Chrome and Firefox to use HTTP/1. For Chrome you add the command line switch
```
disable-http2
```
and for Firefox you add the following Firefox preferences
```
network.http.spdy.enabled false
network.http.spdy.enabled.http2 false
network.http.spdy.enabled.v3-1 false
```
The idea is that we can run the exact same test with both HTTP/2 and HTTP/1 and then compare the metrics we collect.
## Pages
Test on desktop, emulated mobile (running on desktop, emulating a mobile device) and on a real Android phone.
### Desktop
We test each URL 5 times every hour and take the median metric and graph it.
Chrome and Firefox
* https://en.wikipedia.org/wiki/Barack_Obama (large article page, both on Chrome and Firefox)
* https://en.wikipedia.org/wiki/Barack_Obama (second view, first hitting https://en.wikipedia.org/wiki/Main_Page)
* https://en.wikipedia.org/wiki/Ingrid_Vang_Nyman (small article)
* https://en.wikipedia.org/wiki/Elizabeth_I_of_England (second view first hitting https://en.wikipedia.org/wiki/Edward_VI_of_England to get the article to article experience without loading extra JavaScript)
Chrome only
* https://en.wikipedia.org/wiki/Barack_Obama (authenticated, large article)
* https://en.wikipedia.org/wiki/Ingrid_Vang_Nyman (authenticated, small article)
### Emulated mobile (only Chrome, simulated 3G)
We test each URL 5 times every hour and take the median metric and graph it.
* https://en.m.wikipedia.org/wiki/Barack_Obama (a large article page)
* https://en.m.wikipedia.org/wiki/Ingrid_Vang_Nyman (small article)
* https://en.m.wikipedia.org/wiki/Ingrid_Vang_Nyman (second view first hitting https://en.m.wikipedia.org/wiki/Main_Page)
### Chrome on Android
I don't have setup where I can automatically run Android devices so I'll focus on my Huawei. I'll run 31 on runs on each URL and then take the median.
* https://en.m.wikipedia.org/wiki/Barack_Obama (a large article page)
* https://en.m.wikipedia.org/wiki/Barack_Obama (second view first hitting https://en.m.wikipedia.org/wiki/Main_Page)
* https://en.m.wikipedia.org/wiki/Ingrid_Vang_Nyman (a small article page)
* https://en.m.wikipedia.org/wiki/Ingrid_Vang_Nyman (second view first hitting https://en.m.wikipedia.org/wiki/Main_Page)
* https://en.m.wikipedia.org/wiki/Elizabeth_I_of_England (second view first hitting https://en.m.wikipedia.org/wiki/Edward_VI_of_England to get the article to article experience)
### Metrics
We collect a lot of metrics but let us focus on:
* First Visual Render - the first time something was changed in the current view port
* SpeedIndex - the average time at which visible parts of the page are displayed within the view port.
* Last Visual Change - the last time something was changed in the current view port
* On load - when the load event is fired in the browser
The first three metrics are focusing on the user experience and the best metrics we have at the moment.
# Result
The original data: https://dashboard.sitespeed.io/dashboard/db/http1-vs-http2?orgId=1
At 23/5 16:00 in the graphs I've updated to Chrome 59 that uses the new HTTP/2 prio. see https://twitter.com/patmeenan/status/866996216923660288
## Desktop
### Obama first view Chrome
{F8174183 width=800}
### Obama first view Firefox
{F8174188 width=800}
### Obama second view Chrome
{F8174194 width=800}
### Obama second view Firefox
{F8174197 width=800}
### Ingrid_Vang_Nyman first view Chrome
{F8174199 width=800}
### Ingrid_Vang_Nyman first view Firefox
NO data, I need add tests for that.
### Elizabeth_I_of_England second view Chrome
{F8174215 width=800}
### Elizabeth_I_of_England second view Firefox
{F8174217 width=800}
### Obama authenticated Chrome
{F8174221 width=800}
### Ingrid_Vang_Nyman authenticated Chrome
{F8174231 width=800}
## Emulated mobile
### Barack_Obama
{F8174233 width=800}
### Ingrid_Vang_Nyman
{F8174241 width=800}
### Second view Obama
{F8174249 width=800}
### Second view Ingrid_Vang_Nyman
{F8174246 width=800}
## Chrome on Android
Making 31 one runs, taking the median value of each metric.
|URL|version|firstVisualChange|SpeedIndex|LastVisualChange|onLoad|
|https://en.m.wikipedia.org/wiki/Barack_Obama|http2|2.71s|2831|6.73s|5.21s|
|https://en.m.wikipedia.org/wiki/Barack_Obama|http1|2.29s|2415|5.83s|5.09s|
|https://en.m.wikipedia.org/wiki/Barack_Obama second view|http2|1.25s|1339|5.26s|2.04s|
|https://en.m.wikipedia.org/wiki/Barack_Obama second view|http1|1.31s|1408|5.39s|2.08s|
|https://en.m.wikipedia.org/wiki/Ingrid_Vang_Nyman|http2|2.20s|2272|5.48s|3.69s|
|https://en.m.wikipedia.org/wiki/Ingrid_Vang_Nyman|http1|1.75s|1827|4.84s|3.53s|
|https://en.m.wikipedia.org/wiki/Elizabeth_I_of_England second article|http2|1.14s|1198|4.78s|1.59s|
|https://en.m.wikipedia.org/wiki/Elizabeth_I_of_England second article|http1|1.19s|1247|4.83s|1.65s|
# Analyze the result
TBD