As a follow up on T168526 where we run performance test against Mahimahi as a proxy to get stable values, we wanna try out to do the same on Labs. If we can do that we can easily scale our testing to reach our end goal: test on commits and earlier find performance regressions.
The modified version of Mahimahi isn't open source yet but since we tested it out, there's another alternative: WebPageReplay. I've tried it out during the offsite and it is much easier to get to work. The proxy is more of an implementation issue, so we can safely start off with WebPageReplay and then add support for mahimahi if we want that in the future.
In this task we should:
- Make it easy to run tests collecting SpeedIndex/visualCompleteness on Labs by installing WebPageReplay/FFmpeg/Browsertime and document the setup.
- Run tests continuously through out the day and see how much jitter we get and fine tune and see how many runs we need to do per URL to get stable values.
- Create the follow up tasks to finish our main goal: tests on commits.