## Background
Today we collect performance metrics using RUM. That is super and helps us keep track of performance trends. Running our own synthetic testing (automatically testing pages in browsers) will help us find performance problems related to specific browsers, giving us better instruments when analyzing (and talking about) performance: [[ http://www.softwareishard.com/blog/har-12-spec/ | HAR ]] files, [[ https://sites.google.com/a/webpagetest.org/docs/using-webpagetest/metrics/speed-index | SpeedIndex ]] (that is the best way today to show the above the fold content is loaded) & videos.
What makes [[ http://www.webpagetest.org/ | WebPageTest ]] especially good is that it's open-source and can handle Internet Explorer, Firefox, Chrome & Safari.
## Why our own instance?
There's a public instance of WebPageTest where the limit is 200 page views per day. If we test one page first view and repeat view nine times, we can test 10 pages once a day. That is too low. Running our own instance also removes the limit of 9 runs per URL.
## Setup our own instance(s)
WebPageTest contains of one web server (the main entry point) that can run on Linux/Windows and test agent (that actually runs the browser tests) that only runs on Windows.
Setting up WebPageTest instances can be quite a lot of work (even though the list of what needs to be done is not too long https://sites.google.com/a/webpagetest.org/docs/private-instances). WebPageTest is known for the lack/outdated documentation.
However, there are ready made AMIs on Amazon that we could could use. That will save us a lot of time on setup and will also add automatically up/down scaling of agents. By default an agent that hasn't been used in one hour will be shut down. It will also add the ability to run agents from different locations, something we can use in the future.
## What kind of data will WebPageTest collect?
All the metrics we collect can be public (there's no secrets there, whoever who wants can collect them). It would be great if the instance could be publicly accessible. I think we should aim for that in the future but lets first set it up so it works. To make it public,We can run the instance headless = no GUI for running tests, then use the API with an API-key. we need to investigate how people can seeThat way we can run the tests automatically that we want and the test results but not run their own testsresult will be public (if you know the URL).
Here's an example of WebPageTest run using Chrome for https://en.wikipedia.org/wiki/Barack_Obama:
http://www.webpagetest.org/result/150819_W0_19D3/
## What we need to do on a high level
[ ] Setup our own instance of WebPageTest (security etc)
[ ] Automate run/trigger tests on WebPageTest (we can use [[ https://www.sitespeed.io/documentation/webpagetest/ | sitespeed.io ]] or the [[ https://www.npmjs.com/package/webpagetest | WebPageTest nodejs API wrapper ]] )
[ ] Define a couple of URLs to start with. I think it good to keep the list small for the beginning. Talked with @ori and logged in/anonymous users and a couple pages should do as a start. I think it's important to keep it as simple as possible as a start just to get something up and running.
[ ] Define browsers and connectivity. We should use latest IE/Chrome/Firefox/Safari and a browser that doesn't support SPDY.
[ ] Push the data to Graphite
[ ] Find a easy way to map metrics in Graphite to runs in WebPageTest (so we from a specific run easy can look up the data in our WebPageTest instance).