## Background
Today we collect performance metrics using RUM, that's super and helps us keep track of performance trends. Running our own synthetic testing (automatically test pages in browsers) will help us find performance problems related to specific browsers, giving us better instruments when analyzing (and talking about) performance: [[ http://www.softwareishard.com/blog/har-12-spec/ | HAR ]] files, [[ https://sites.google.com/a/webpagetest.org/docs/using-webpagetest/metrics/speed-index | SpeedIndex ]] (that is the best way today to show the above the fold content is loaded) & videos.
What's make [[ http://www.webpagetest.org/ | WebPageTest ]] extra good is that it's Open Source and can handle Internet Explorer, Firefox, Chrome & Safari.
## Why our own instance?
There's a public instance of WebPageTest where the limit is 200 page views per day. If we test one page first view and repeat view nine times, we can test 10 pages once a day. Yep that is too low. Running our own instance also removes the limit of 9 runs per URL.
## Setup our own instance(s)
WebPageTest contains of one web server (the main entry point) that can run on Linux/Windows and test agent (that actually runs the browser tests) that only runs on Windows.
Setting up WebPageTest instances can be quite much work (even though the list of what needs to be done is not too long https://sites.google.com/a/webpagetest.org/docs/private-instances). WebPageTest is known for the lack/outdated documentation.
However, there are ready made AMI:s on Amazon that we could could use. That will save us a lot of time on setup and will also add automatically up/down scaling of agents. By default an agent that hasn't been used in one hour will be shutdown. It will also add the ability to run agents from different locations, something we can use in the future.
## What kind of data will WebPageTest collect?
All the metrics we collect can be public (there's no secrets there, whoever who wants can collect them). It would be great if the instance could be public accessible, I think we should aim for that in the future but lets first set it up so it works. To make it public, we need to investigate how people can see the test results but not run their own tests.
Here's an example of WebPageTest run using Chrome for https://en.wikipedia.org/wiki/Barack_Obama:
http://www.webpagetest.org/result/150819_W0_19D3/
## What we need todo on a high level
* Setup our own instance of WebPageTest (security etc)
* Automate run/trigger tests on WebPageTest (we can use [[ https://www.sitespeed.io/documentation/webpagetest/ | sitespeed.io ]] or the [[ https://www.npmjs.com/package/webpagetest | WebPageTest nodejs API wrapper ]] )
* Define a couple of URL:s to start with. I think it good to keep the list small for the beginning. Talked with @ori and logged in/anonymous users and a couple pages should do as a start. Think it's important to keep it as simple as possible as a start just to get something up and running.
* Define browsers and connectivity. We should use latest IE/Chrome/Firefox/Safari and a browser that doesn't support SPDY.
* Push the data to Graphite and create
* Find a easy way to map metrics in Graphite to runs in WebPageTest (so we from a specific run easy can look up the data in our WebPageTest instance).