| Peter | |
| Jun 20 2017, 4:06 PM |
| F8600502: Screen Shot 2017-07-03 at 9.40.39 AM.png | |
| Jul 3 2017, 7:47 AM |
| F8540963: Screen Shot 2017-06-28 at 5.42.25 AM.png | |
| Jun 28 2017, 4:03 AM |
We can choose between the following running an agent on AWS:
We need to add the configuration on our WebPageTest server, then add the test(s) (I think doing another Jenkins job would be fine for now, or even just add it to the crontab depending on what kind of time limit we have) and choose URLs. Then after the change, we keep it running for a couple of weeks, collect the metrics and close down the instance(s).
I've added the locations in https://github.com/WPO-Foundation/webpagetest/blob/561541499dc4a3f83fc207971663f515189b8c7a/www/settings/ec2_locations.ini restarted and tested an instance from Mumbai:
http://wpt.wmftest.org/result/170621_Z2_6G/
Adding this as a reference: http://wpt.wmftest.org/video/compare.php?tests=170621_NK_8R-r:3,170621_NK_8R-r:1
The difference between two runs (empty cache) can be really big, the diff here is 1.5 s first visual change, However it will probably not matter as long as we pick median/fastest run, so let see when we got permanent agent up and running.
I had some test last weekend and just tested adding it to my own AWS instance in Mumbai. I added two URLs, testing on desktop (cable) and two on mobile (3g and 3gem).
https://hi.m.wikipedia.org/wiki/%E0%A4%AD%E0%A4%BE%E0%A4%B0%E0%A4%A4%E0%A5%80%E0%A4%AF_%E0%A4%85%E0%A4%82%E0%A4%A4%E0%A4%B0%E0%A4%BF%E0%A4%95%E0%A5%8D%E0%A4%B7_%E0%A4%85%E0%A4%A8%E0%A5%81%E0%A4%B8%E0%A4%82%E0%A4%A7%E0%A4%BE%E0%A4%A8_%E0%A4%B8%E0%A4%82%E0%A4%97%E0%A4%A0%E0%A4%A8
and
https://hi.m.wikipedia.org/wiki/%E0%A4%AB%E0%A5%89%E0%A4%B8%E0%A4%8F%E0%A4%B6%E0%A4%BF%E0%A4%AF%E0%A4%BE
I'll remove them next week and we can just add them to WPT when we decide 2do it. One interesting thing is that with the slowest connection we have really steady first visual change where it differs 66 ms over a couple of days:
We need to decide:
We tested out Catchpoint to see if the SpeedIndex there where usable, setting connectivity to 4g, running one test every hour:
The difference is much higher than for example using WebPageTest, so lets concentrate on using that.
Change 362971 had a related patch set uploaded (by Phedenskog; owner: Phedenskog):
[performance/WebPageTest@master] Add test for run in Tokyo and Mumbai for testing the new Asian cache pop.
Change 362972 had a related patch set uploaded (by Phedenskog; owner: Phedenskog):
[integration/config@master] Run WebPageTest tests from Asia to verify the new cache pop.
I've changed those to the Linux instance, think that will work fine and a good test for us. We can make this run maybe every 4 hour or something like that, then we can have Mumbai and Tokyo run at the same time.
It would be nice though to also have a second view test, I can add that too next week.
Let me try Windows vs Linux and make sure Linux have the same run time. When I fire away the tests for Linux it seems like it hangs ...
So I've been testing now on Linux it looks kind of worrying, I'll switch to Windows and test there.
First run with the URL starting the instances and running the tests:
10:07:55 - 11:03:31 - runtime is 56 minutes but then we started the agent too.
Running on an already started instance:
11:28:02 - 12:44:08 - runtime 1h 16 min, it seems something is wrong.
Let me try now with Windows, maybe there's something else.
No it is the same thing with Windows. Four runs take over one hour. I'll retest again on Monday. Maybe it's something going on with having the server running on one location and the agent on others (far) far away. hmm.
I switched to use Windows. Even though it seems we have the same problem there, I'll not investigate it since we will only run this for a couple of weeks. If we wanna test from different locations in the future then we need to look into it.
Change 362971 merged by jenkins-bot:
[performance/WebPageTest@master] Add test for run in Tokyo and Mumbai for testing the new Asian cache pop.
Change 362972 abandoned by Phedenskog:
Run WebPageTest tests from Asia to verify the new cache pop.
Reason:
Let do this job on the main job instead, that will simplify things.
Change 421266 had a related patch set uploaded (by Phedenskog; owner: Phedenskog):
[integration/config@master] Run WebPageTest tests from Singapore to verify the new cache pop.
Change 421266 abandoned by Phedenskog:
Run WebPageTest tests from Singapore to verify the new cache pop.
Reason:
Let me trigger those runs from the WebPageTest server instead, that will be super clean.