Documentation: https://wikitech.wikimedia.org/wiki/WebPageTest

Background

Today we collect performance metrics using RUM. That is super and helps us keep track of performance trends. Running our own synthetic testing (automatically testing pages in browsers) will help us find performance problems related to specific browsers, giving us better instruments when analyzing (and talking about) performance: HAR files, SpeedIndex (that is the best way today to show the above the fold content is loaded) & videos.

What makes WebPageTest especially good is that it's open-source and can handle Internet Explorer, Firefox, Chrome & Safari.

Why our own instance?

There's a public instance of WebPageTest where the limit is 200 page views per day. If we test one page first view and repeat view nine times, we can test 10 pages once a day. That is too low. Running our own instance also removes the limit of 9 runs per URL.

Setup our own instance(s)

WebPageTest contains of one web server (the main entry point) that can run on Linux/Windows and test agent (that actually runs the browser tests) that only runs on Windows.

Setting up WebPageTest instances can be quite a lot of work (even though the list of what needs to be done is not too long https://sites.google.com/a/webpagetest.org/docs/private-instances). WebPageTest is known for the lack/outdated documentation.

However, there are ready made AMIs on Amazon that we could could use. That will save us a lot of time on setup and will also add automatically up/down scaling of agents. By default an agent that hasn't been used in one hour will be shut down. It will also add the ability to run agents from different locations, something we can use in the future.

What kind of data will WebPageTest collect?

All the metrics we collect can be public (there's no secrets there, whoever who wants can collect them). It would be great if the instance could be publicly accessible. I think we should aim for that in the future but lets first set it up so it works. We can run the instance headless = no GUI for running tests, then use the API with an API-key. That way we can run the tests automatically that we want and the result will be public (if you know the URL).

Here's an example of WebPageTest run using Chrome for https://en.wikipedia.org/wiki/Barack_Obama:
http://www.webpagetest.org/result/150819_W0_19D3/

SPDY problems

We will have a problem in a way of that we will not have everything exactly how we want it with SPDY for different browsers until we change to HTTP2.

It's like this: WebPageTest has SPDY support for Chrome, meaning it will use SPDY and all the metrics will be right, except for sizes of different objects in the HAR file. That doesn't matter so much for us right now, but if we also wanna pick assets sizes from the run, we need to have workaround (and don't worry there is).

Firefox is using SPDY but the HAR/Waterfall graphs aren't generated because there's no SPDY decoder implemented for Firefox. But it will be there when we support HTTP2.

Internet Explorer 11 isn't supporting SPDY on older version of Windows (and that's what WPT uses).

The Safari version running on WPT is 7:ish meaning not supporting SPDY.

What we need to do on a high level

• Setup our own instance of WebPageTest (security etc) and mount the logs dir to EBS (how do we do that?)
• Automate run/trigger tests on WebPageTest (we can use sitespeed.io or the WebPageTest nodejs API wrapper )
• Define a couple of URLs to start with. I think it good to keep the list small for the beginning. Talked with @ori and logged in/anonymous users and a couple pages should do as a start. I think it's important to keep it as simple as possible as a start just to get something up and running.
• Define browsers and connectivity. We should use latest Chrome/Firefox and discuss how we do with Internet Explorer/Safari until WPT versions use SPDY or we switch to HTTP2. We should also run a browser that not will support SPDY/HTTP2 so we also keep track of that
• Decide how many times we want to test each URL.
• Push the data to Graphite
• Find a easy way to map metrics in Graphite to runs in WebPageTest (so we from a specific run easy can look up the data in our WebPageTest instance).