A first start could be to do our alerts more intelligent. Today we have many different tests/alerts that fire and I think we should aim to make them work together.

Example:

• To know if we have a regression using WebPageTest we have three different queries that runs using Chrome: We check three URLs and compare if the first visual increased since X days. If all three URLs has increased, we fire an alert.
• We also have another alert (another graph) that checks if we got metrics from WebPageTests the last Y hours. That check is there to make sure that WebPageTest works. That alert should have highest priority and the performance team needs to investigate why there's no metrics from WebPageTest.
• We can also check the TTFB for our tests the same way we check for the If we also have high TTFB we know something happened with the backend. If there's no increased in backend time, we know something has been pushed.
• We also can check TTFB standard deviation, I've seen that over time it can happens that our standard deviation increase and if that's the case, we need to look into the WebPageTest server to see if we can understand what's going on.
• We could also know better what has caused a regression: if the regression isn't not tied to the main train going out, it should be easier to automatically narrow down what's been pushed by the data that is in https://wikitech.wikimedia.org/wiki/Server_Admin_Log (or a better format)

That is one alert for WebPageTest for Chrome. Ideally we should combine that with Firefox tests, tests using WebPageReplay to "know" if its a front end regression. And then we could also combine it with the Navigation Timing data we have to better narrow it down.

Let me start creating runbooks for the alerts and see what we can automate.