There's a couple issues with running WebPageTest that we need to fix before we can make the switch. Lets keep this issue open until all of them are fixed.
At the moment there's two things:
- Annotations are not sent automatically when running WebPageTest when Browsertime is disabled. It should but I cannot see them. Let me debug that.
- We cannot get the image from the WebPageTest server. The log is filled with messages like: WARN: Couldn't get screenshot for id 190828_0F_a7049b18fbc27a44d975bed0d0268d52, run 2 from the WebPageTest API with the error Not Found (url = https://www.wikipedia.beta.wmflabs.org/) - I think this is a upstream issue in WebPageTest but also thinking maybe our server instance is too small and cannot handle the API load? Let me create a upstream issue later today.