Today with the current version of our synthetic testing WebPageTest and Browsertime we run our test agent on AWS because the AWS gives us the most stable metrics. We send metrics through statsv to Graphite, so we have limited the number of metrics that we send (for WebPageReplay/Browsertime we send 17 metrics) ... but with the tools we use there are so much more metric that we can use. For one tested URL there are 500+ metrics that we get automatically and graph. Storing those metrics gives us a bigger picture of whats going on + makes it easier to investigate regressions.
We have a new Graphite server up and running on AWS where we want to store only synthetic metrics that is generated on the outside of our network. That also gives us annotations per test as in T156252 without overloading anyone else's annotations as in T175708
We need to add it as an datasource to https://grafana.wikimedia.org, need to look into how we do that the correct way.