History

WebPageTest is the synthetic web performance tool we used the longest. Here's a brief timeline:

• In 2015 in task T109666 I setup our first WebPageTest instance. The tool is Open Source and released under the Apache License. The agent (the machine that runs the actual browser tests) only used Windows at that time, so we deployed the instance on AWS.
• In 2017 in task T165626 I started the work to move to Linux. There was a lot of things that didn't work in the upstream project and I think my testing and feedback moved the project so it fully worked on Linux.
• In 2020/2021 @dpifke started the work to explore if we could move the project in-house in task T262962. The idea was that we could enable the GUI so Wikimedia's could run tests on demand.
• In late 2020 Catchpoint acquires WebPageTest and for a while the creator (Patrick Meenan) works for Catchpoint but then leaves the company. The development of the Open Source licensed version stops and all new works is licensed under PolyForm Shield License.
• In 2021/2022 in T278164 the work is done to make sure we only use the (old) Apache branch

The future

First of all I just want to make clear that we have sitespeed.io/browsertime tests that runs against Wikipedia and had for years, we test the same pages (and more). So today we test the same thing twice.

There's a couple of different actions we could do:

• Continue use the unsupported Apache branch and do the changes needed ourself
• Pay for the service and use Catchpoint
• Stop using WebPageTest

Let me go through the pro/cons with the different approaches.

Continue use the unsupported Apache branch and do the changes needed ourself

One thing that is sure is that we will come to a point where the current branch will not work any more. Browsers change over time and you need to keep tune the code to make it continue to work. We already have some functionality that do not work (the trace log from Chrome).

To continue running the Apache branch, we should make our own branch and commit a couple of the changes I've done so far to keep it running locally on our server. And start fixing things that is broken, first start with the trace log for Chrome.

I asked a couple months ago if someone was interesting in helping maintaining Apache branch and I got one yes, so it doesn't seem like the performance community care so much. I think its up to us to maintain it.

Pay for the service and use Catchpoint

Since Pat Meenan left there are new people maintaining WebPageTest so his almost 15+ of experience is gone and I think you can see that by just testing the public version. We didn't have a nice experience when we tested out Catchpoint for web performance measurements in T162515 and I think we would see the same issue today. They are a couple of years behind us in experience running synthetic testing tools :(

Stop using WebPageTest

We could continue with the sitespeed.io/Browsertime tests and just stop with WebPageTest.
The reason for continuing having WebPageTest is that its nice to have two different tools, different release cycles etc. It makes me more sure when for example we have regression coming from Chrome. Some of the browser vendors are kind of stubborn when you point out a bug in the browser and being able to reproduce on multiple tools are good.

Change 840191 had a related patch set uploaded (by Phedenskog; author: Phedenskog):

[performance/synthetic-monitoring-tests@master] Add Wikidata tests to Browsertime tests.

https://gerrit.wikimedia.org/r/840191