Page MenuHomePhabricator

Test enwiki configuration with minimal WebDriver impact
Closed, ResolvedPublic

Description

After discussing with Mozilla I wanna test to minimize the impact of the WebDriver when we run the tests. At the moment it works like this: We start a test and the WebDriver gives us back control after the onload event. Its unclear how chatty the communication is. But when we get back control we run a JS every 200 ms to check if everything is really finished.

Another way of doing it is to configure the driver to give back control immediately and then have a configurable wait time before check if the page has been finished loading and then have another configurable poll time. That we can minimize the impact of other things that runs against the browser at the same time as we runs the tests.

Event Timeline

Change 554831 had a related patch set uploaded (by Phedenskog; owner: Phedenskog):
[performance/synthetic-monitoring-tests@master] Disable WebDriver interruptions

https://gerrit.wikimedia.org/r/554831

Change 554927 had a related patch set uploaded (by Phedenskog; owner: Phedenskog):
[performance/synthetic-monitoring-tests@master] Upgrade to 11.5 to be able to try minimize WebDriver interruptions

https://gerrit.wikimedia.org/r/554927

Change 554927 merged by jenkins-bot:
[performance/synthetic-monitoring-tests@master] Upgrade to 11.5 to be able to try minimize WebDriver interruptions

https://gerrit.wikimedia.org/r/554927

Change 554831 merged by jenkins-bot:
[performance/synthetic-monitoring-tests@master] Disable WebDriver interruptions

https://gerrit.wikimedia.org/r/554831

Change 555375 had a related patch set uploaded (by Phedenskog; owner: Phedenskog):
[performance/synthetic-monitoring-tests@master] Upgrade to 11.6.0 to be able to use new Firefox preferences from Mozilla.

https://gerrit.wikimedia.org/r/555375

Change 555375 merged by jenkins-bot:
[performance/synthetic-monitoring-tests@master] Upgrade to 11.6.0 to be able to use new Firefox preferences from Mozilla.

https://gerrit.wikimedia.org/r/555375

Change 555661 had a related patch set uploaded (by Phedenskog; owner: Phedenskog):
[performance/synthetic-monitoring-tests@master] Disable safe browsing for Firefox

https://gerrit.wikimedia.org/r/555661

Change 555661 merged by jenkins-bot:
[performance/synthetic-monitoring-tests@master] Disable safe browsing for Firefox

https://gerrit.wikimedia.org/r/555661

This done now, but I cannot see any change for Firefox. One thing left todo is to use verify that FF doesn't any extra requests during loading the page, I've only verified that on Mac not Linux.

Summary: I've changed so that the WebDriver leave the control to Browsertime directly after navigation and then we wait X seconds before we run any JavaScript to check if the page has finished loading. I've also removed our use of a WebExtension (it isn't loaded anymore) and then I've also changed configuration for Firefox to use preferences that I got from the Mozilla performance team.

The changes has made great impact on stdev on First Visual Change for Chrome running with WebPageReplay:

WebPageReplay drilldown - Grafana-chrome.jpg (920×1 px, 114 KB)

Running without replaying I cannot see any change.

For Firefox there's maybe a little change but not super obvious:

WebPageReplay drilldown - Grafanafirefox.jpg (928×2 px, 166 KB)

Here are some more visuals. Chinese wiki first visual change:

Screen Shot 2019-12-09 at 3.09.29 PM.png (726×1 px, 130 KB)

Speed Index:

Screen Shot 2019-12-09 at 3.09.37 PM.png (732×1 px, 128 KB)

I think this is done. The effect hasn't lasted as good as it was but I think that's something going on on the server

Here you can see First Visual Change:

Screen Shot 2019-12-14 at 2.41.33 PM.png (722×1 px, 133 KB)

And you can see the stdev increased, not as high as before but definitely higher than then when we started to test.

Screen Shot 2019-12-14 at 2.42.31 PM.png (394×586 px, 65 KB)