Page MenuHomePhabricator

Consider moving WebPageTest from Amazon to our own setup
Closed, DeclinedPublic

Description

Background

We use WebPageTest to synthetic test the performance of Wikipedia.org. WebPageTest desktop agents need to run on Windows, so the choice for us when we started using it, was to use the ready made AWS images. How it's setup and used is documented here: https://wikitech.wikimedia.org/wiki/WebPageTest

We have one server instance and one agent constantly running and then spin up another agent when we need to test changes it our test scripts. The test agent for testing our changes is a small AWS instance, the performance isn't important. The agent constantly running tests need to be large. I think we have problems with the current instance being too small when we test using Firefox, the CPU reaches the limit and we can see that the JS sometimes halts/takes long time to execute.

Limitations running on AWS

  • We cannot test on mobile devices - however we maybe want to handle that differently in the future see T136850
  • The cost. Is it expensive or is it cheap? We need to calculate and compare. Plus think about how many agents we wanna run in the future.
  • How large instance do we need to use? I think we have problem with Firefox today so we need a larger instance.

Pros running in AWS

  • We have done the server setup once and it was a lot of work and it will be even more setting it up ourselves. The documentation for WebPageTest could be better.
  • It super easy to add new desktop agents, just one line of configuration (however we don't use that today).
  • No hustle updating software, everything fixes it self (auto updating agents etc).
  • If we run the server in-house we need to make sure we can publish the result to https://www.webpagetest.org so it's public and we can share specific problems.

What we need to find out

  • What version of Windows do we need to use? I know before it was Windows 7 but when Edge will be supported it will be Windows 10.

Possibile solutions

There's three different ways we can go forward:
A. Keep it as is (running only on AWS)
B. Running desktop and server on AWS and have mobile agents running Linux/OS X in-house
C. Running everything in-house

Event Timeline

Peter created this task.Dec 15 2016, 8:51 PM
Restricted Application added a subscriber: Aklapper. · View Herald TranscriptDec 15 2016, 8:51 PM
Krinkle renamed this task from Move WebPageTest from Amazon to our own setup to Consider moving WebPageTest from Amazon to our own setup.Dec 15 2016, 9:35 PM
Krinkle triaged this task as Medium priority.
Peter updated the task description. (Show Details)Dec 16 2016, 1:27 PM
Peter added a comment.Apr 12 2017, 8:43 AM

Status update: There's experimental Linux version of Chrome now for WebPageTest (you can choose it in the dropdown). According to the issues there's a Firefox Linux versions coming out in a couple of weeks, so I think we should move to Linux ASAP when Firefox is finished. We will miss the IE test we do, but that one only test HTTP/1.

However if we feel that it is ok to drop IE testing, we should move to sitespeed.io instead, since it supported Linux for years instead of weeks for WPT. What I will miss is the compare functionality between two runs, but maybe we can find a creative solution to fix that.

Krinkle closed this task as Declined.Jan 17 2018, 8:54 AM
Krinkle added a subscriber: Krinkle.

Closing. Based on various tests (T176361), we found that AWS remains significantly more stable for WPT metrics.

Peter added a comment.Jan 17 2018, 9:15 AM

Been thinking about it, I think the reason to have our own agent is to drive mobile phones.