We use WebPageTest to synthetic test the performance of Wikipedia.org. WebPageTest desktop agents need to run on Windows, so the choice for us when we started using it, was to use the ready made AWS images. How it's setup and used is documented here: https://wikitech.wikimedia.org/wiki/WebPageTest
We have one server instance and one agent constantly running and then spin up another agent when we need to test changes it our test scripts. The test agent for testing our changes is a small AWS instance, the performance isn't important. The agent constantly running tests need to be large. I think we have problems with the current instance being too small when we test using Firefox, the CPU reaches the limit and we can see that the JS sometimes halts/takes long time to execute.
Limitations running on AWS
- We cannot test on mobile devices - however we maybe want to handle that differently in the future see T136850
- The cost. Is it expensive or is it cheap? We need to calculate and compare. Plus think about how many agents we wanna run in the future.
- How large instance do we need to use? I think we have problem with Firefox today so we need a larger instance.
Pros running in AWS
- We have done the server setup once and it was a lot of work and it will be even more setting it up ourselves. The documentation for WebPageTest could be better.
- It super easy to add new desktop agents, just one line of configuration (however we don't use that today).
- No hustle updating software, everything fixes it self (auto updating agents etc).
- If we run the server in-house we need to make sure we can publish the result to https://www.webpagetest.org so it's public and we can share specific problems.
What we need to find out
- What version of Windows do we need to use? I know before it was Windows 7 but when Edge will be supported it will be Windows 10.
There's three different ways we can go forward:
A. Keep it as is (running only on AWS)
B. Running desktop and server on AWS and have mobile agents running Linux/OS X in-house
C. Running everything in-house