Adding also Obama for reference. Windows:

Linux

Not as big difference but not as good as before.

I created an upstream issue https://github.com/WPO-Foundation/wptagent/issues/79 and will also do a switch to pickup the median run instead of the fastest.

Change 408804 had a related patch set uploaded (by Phedenskog; owner: Phedenskog):
[performance/WebPageTest@master] Add two extra runs where we pick the median run

https://gerrit.wikimedia.org/r/408804

Change 408804 merged by jenkins-bot:
[performance/WebPageTest@master] Add two extra runs where we pick the median run

https://gerrit.wikimedia.org/r/408804

Here's an example of where we have a big difference (500 ms) between different runs (1st and 3rd):

http://wpt.wmftest.org/video/compare.php?tests=180213_4D_BK-r%3A1%2C180213_4D_BK-r%3A3&thumbSize=200&ival=100&end=visual#

The only other thing that differs is the amount of bytes (that is strange).

The same here (another URL, same different between different runs):
http://wpt.wmftest.org/video/compare.php?tests=180213_YQ_16Y-r:1,180213_YQ_16Y-r:3

But when I download the HAR file, there is no difference in the amount of data, so maybe something else is going on.

@Gilles @Krinkle do you see something else?

Pat did some tests yesterday when he got 200 ms in variance (much better): https://github.com/WPO-Foundation/wptagent/issues/79#issuecomment-365694467

I went through the numbers more and the Firefox metrics starts to look better but I still think there's something wrong with the implementation for Chrome (and not on our side). When I checked the metrics for mobile we have a diff there too, but not as much as on Desktop. I'll compare some more during the day.

Change 412885 had a related patch set uploaded (by Phedenskog; owner: Phedenskog):
[performance/WebPageTest@master] Test slower connection for more stable metrics on Linux

https://gerrit.wikimedia.org/r/412885

Change 412885 merged by jenkins-bot:
[performance/WebPageTest@master] Test slower connection for more stable metrics on Linux

https://gerrit.wikimedia.org/r/412885

I did try slowing down the connection, comparing cable vs 3gfast. For Obama we have the same difference, just moving the start values:

Facebook is another story:

We got super stable metrics (instead of the other way around) with 3gfast. I would say that we hit a really bad spot going to Linux on that connectivity.

I think we should move on with the move to Linux and then think what connectivities we should use for our tests. I mean now for alerting I think we are ready to switch to WebPageReplay/Browsertime and then we can continue to have WebPageTest to show what it can look like for a user.

Change 413311 had a related patch set uploaded (by Phedenskog; owner: Phedenskog):
[performance/WebPageTest@master] Remove our tests for 3gfast on Linux.

https://gerrit.wikimedia.org/r/413311

Change 413311 merged by jenkins-bot:
[performance/WebPageTest@master] Remove our tests for 3gfast on Linux.

https://gerrit.wikimedia.org/r/413311

WebPageTest on Linux switch how to setup traffic shaping https://github.com/WPO-Foundation/wptagent/commit/f90afeef410cf522b7bafde8ea906901a150004f and after that our metrics looks better.

Interesting, worth testing that different tool for webpagereplay