I think what we have today is really fantastic: WebPageTest testing important enwiki pages, Browsertime/WebPageReplay keeping track of frontend performance for 10+ wikis and Fresnel starting to test on commits.

And there are always room for improvements. I'll list what I think should be done the coming year to have our performance testing as good as possible.

Here's my wish list:

• Automate performance testing on real mobile devices - We are ready to do it, but still need a place for the devices T197847
• Graph more metrics from our tests. - One of the problems today is that we send all the metrics to Graphite through statsv. That limits us of what we graph. For example we should minimum collect the same metrics as we do with RUM. Today WebPageTest misses collecting navigation timing data, Browsertime has it all, but we do not graph it. Today we graph time spent in JS for Chrome, but we could also graph time spent per event type. I do that for sitespeed.io and it useful to better see what's actually is happening. I think the easiest way for us is to deploy our own Graphite instance for synthetic performance metrics. It will give us more flexibility and ability to collect more data. It also enables having annotations with links to the actual results. What we will miss is the sync-wikiversions annotation but maybe we can fix that too, with sending it to multiple endpoints.
• Move WebPageTest out of Jenkins - Today our runs gets started by Jenkins, by pulling the project from Gerrit. So for example when Gerrit was down, our tests stopped working. From the beginning we did our runs from the crontab of the WebPageTest server. I think we should aim for that all our testing works independently of our other tools.
• More focus on testing user flows - https://wikitech.wikimedia.org/wiki/Performance/Synthetic_Measurement_Experiment_2019 showed us what a big different the performance is depending on if you as a user hit Wikipedia with an empty browser cache vs hitting multiple pages. We should add more tests to cover that.
• Enable performance alerts for all wikis - We should make sure we automate alerts for more Wikis as in T198287. What's holding me back is that there's a lot of work since alerting doesn't work with templating in Grafana (and I still want the Graphs to be). But we still should do it. I think the way forward is to create one dashboard per wiki instead of one dashboard for desktop and one for mobile.