Page MenuHomePhabricator

Firefox gives us unstable metrics in synthetic testing
Closed, ResolvedPublic

Description

At the moment we run our testing on AWS c4.large, using WebPageReplay to replay our pages. When we run our tests on desktop with Chrome we can find regressions in first visual change that are 40 ms or higher. With Chrome we also use the Chrome trace log (where we can get CPU stats of where Chrome spends time) where we can find regressions that are 20 ms or higher. For example this setup helped us find a regression in Chrome 67 when the Chrome team changed their music/video player https://bugs.chromium.org/p/chromium/issues/detail?id=849108).

With Firefox we don't get that stable metrics so we can alert on them. Our alert threshold would then be something like 500 ms. We have turned off all the things (that we know about) that can make metrics with Firefox unstable (like the background update of 3rd party blacklists etc) but it doesn't help. Most of the pages for us have two buckets of metrics. For example if we make 11 test runs, X runs is fast and Y runs is slow (and the difference could be like 1 second, so it is a big diff). Using Chrome the diff is max 100 ms.

We see unstable metrics with WebPageTest without WebPageReplay too, so it seems not to be tool related. However some URLs are more sensitive than others, so content matters.

This could either be something special with Wikipedia pages that doesn't work well with Firefox, something in our Firefox setup that we do wrong with the tools, or a generic Firefox problem.

Event Timeline

Peter created this task.Jan 18 2018, 9:05 AM
Restricted Application added a subscriber: Aklapper. · View Herald TranscriptJan 18 2018, 9:05 AM
Peter renamed this task from Show the to Explain the sudden change in First Visual Change in the Obama article for FF 57.Jan 18 2018, 9:07 AM
Peter updated the task description. (Show Details)
Peter added a comment.EditedJan 18 2018, 9:10 AM

Adding these for reference (it seems like we got more stable metrics before 57):

FF 57

FF 54

Peter updated the task description. (Show Details)Jan 18 2018, 9:13 AM
Imarlier triaged this task as Normal priority.Jan 18 2018, 3:07 PM
Imarlier moved this task from Inbox to Doing on the Performance-Team board.
Alsee added a subscriber: Alsee.Feb 8 2018, 4:50 PM
Peter renamed this task from Explain the sudden change in First Visual Change in the Obama article for FF 57 to Firefox Quantum (and later) seems extremely sensitive in First Visual Change/Speed Index to content changes .Mar 9 2018, 11:49 AM
Peter updated the task description. (Show Details)
Peter added a comment.Mar 20 2018, 7:17 PM

Here's another example. These changes made First Visual Change go down 400 ms.

No change at all for Chrome at the same time:

Hmm been checking individual runs and for Firefox when we do 11 runs the variance can be 500 ms (for chrome it is 100 ms) between runs so I wonder maybe it is some configuration issue? We turned off OCSP and black/white list update but maybe it's something more that needs to be disabled.

Peter added a comment.Mar 22 2018, 5:34 AM

I've done a test by switching to the same Firefox preferences that WebPageTest uses instead of the default one that Geckodriver uses (+ some extras) and the median SpeedIndex looks a lot better:

The mdev are the same though. Let me try to turn on the HAR exporter again and also see if I can identify which preferences that do the trick.

This is driving me crazy :( I've verified that Firefox is using the proxy (by removing internet when replying the data), it works. However both using the proxy and doing run without the proxy the Firefox metrics has high variance: Running Chrome without setting any connectivity, not using the proxy the variance is 100 ms, the same and less when we use the proxy. For Firefox the variance is 1000 ms of firstVisualChange both with or without the proxy. I wonder if there's something in the Docker setup that can explain it? When I check the HAR file and compare with a run with the same connectivity, the HAR is mostly identical except that the timings differs 1000ms (so the HAR doesn't help). I'll test if I see the same without Docker and also collect the MOZ log but I think it is a trace log of what Firefox is doing that's missing. So far I tried to rollback to stable 58, disable ipv6 , increased the file limit on Linux but no luck so far.

Peter renamed this task from Firefox Quantum (and later) seems extremely sensitive in First Visual Change/Speed Index to content changes to Firefox Quantum (and later or earlier) gives us really unstable metrics in Docker.Mar 23 2018, 10:25 AM
Peter updated the task description. (Show Details)

I've tried out different preferences, and stopped more type of requests in the background (setting network.captive-portal-service.enabled to false). Haven't been able to track down more requests in the MOZ-log so it is probably ok for now.

I've been going through the logs (and comparing Chrome vs Firefox) and Firefox seems to always render in about DOMInteractive. For both of them we have two buckets of DOMInteractive: For Chrome it is more stable, for example when we test https://en.wikipedia.org/wiki/Facebook we have one bucket if 515 ms and one 1014 ms. If we do 11 runs, we get a couple of runs in the first bucket. But a DOMInteractive 512 generates a first visual change at 1100 and DOMInteractive: 1019 also lands at first visual change 1100.

Firefox when we test Sweden and get a DOMInteractive at 1626 the first visual change is 1600. When we get a DomInteractiveTime at 2042 the visual change happend at 2034. The same for the rest of the URLs.

Also another interesting: When I turn the MOZ log it adds aprox 1s to each metrics, so that could be one of the reasons why FF is so much slower than Chrome on WebPageTest.

Peter added a comment.Apr 23 2018, 8:19 AM

I've been looking into the variation and we have two buckets: one early and one slower first paint (it can vary up to 1 s depending on the page). I think it depends on when the JavaScript kicks in see: https://phabricator.wikimedia.org/T160315#4148868

Attaching a HAR where we have the diff (almost 1 second) in first visual change between first run and third run.

Hmm we miss the user timings in the HAR, let me add those.

Peter added a comment.Apr 24 2018, 1:16 PM

I think my theory with when the JavaScript is parsed don't hold when I run Firefox on my local. I can get the same variance, but the mwLoadStart doesn't match firstVisualChange.

However in the "slow" runs we have larger gap before the second request starts:


That time makes almost up the big difference in first visual change. There's no way to get more info out of FF when we automate it, so let me try running it manually with some connectivity limits.

Peter added a comment.Apr 25 2018, 4:18 AM

One thing that I've seen is that on Linux Nightly, Firefox calls home a couple of time to check for updates:
https://aus5.mozilla.org/update/3/GMP/61.0a1/20180317134556/Linux_x86_64-gcc3/null/nightly-cck-ubuntu/Linux%204.9.87-linuxkit-aufs%20(GTK%203.22.25%2Clibpulse%20not-available)/canonical/1.0/update.xml (love that null in the URL).

When I run on Mac OS X there's no requests to any .mozilla.org domain (using the same configuration for the preferences).

But I cannot say it's that that cause the variance because it looks mostly the same for different runs, but let me at least file an upstream bug.

Peter claimed this task.May 21 2018, 8:27 PM
Peter moved this task from Doing to This Quarter (FY1920Q1 Jul-Sep) on the Performance-Team board.
Imarlier raised the priority of this task from Normal to High.Jun 21 2018, 10:51 AM
Imarlier added a subscriber: Imarlier.

Blocking us from setting up alerts on Firefox.

Peter added a comment.Sep 11 2018, 6:09 AM

I would say this is mainly a problem for us with Browsertime/WebPageReplay since it is there we get so low variance for Chrome. But we see the same kind of thing on some pages on WebPageTest too.

For WebPageTest checking the latest runs one page has a diff of 500 ms in First Visual Change (16%):
http://wpt.wmftest.org/result/180911_2P_3W/

And here 700 ms (25%):
http://wpt.wmftest.org/result/180910_AB_99/

25% is high. The same page in Chrome on WebPageTest the diff is 100 ms (7%):
http://wpt.wmftest.org/result/180911_7P_4N/

For Browsertime/WebPageReplay it looks like this. This is Chrome first visual change, check the right graph where we graph the deviation:

And compare with Firefox:

It seems like the difference depends on content but it looks like we have worse metrics for Firefox across the board, here's the Swedish wiki:


And then on beta:


Peter renamed this task from Firefox Quantum (and later or earlier) gives us really unstable metrics in Docker to Firefox gives us unstable metrics in synthetic testing.Sep 11 2018, 6:12 AM
Peter updated the task description. (Show Details)
Peter added a comment.Sep 13 2018, 8:40 AM

I've been looking at other sites to see if I could find the same pattern. I thought I found one but digging to the problem it seems that the site loaded blocking CSS from Google and in Firefox that sometimes took a looong time. Interesting but nothing for us to look into.

Peter added a comment.EditedSep 14 2018, 12:00 PM

I think the reason is that a c4.large is a too small instance for Firefox: Here's the log running Browsertime/WebPageReplay on a c4.large (look at FirstVisualChange):

[2018-09-14 11:38:28] INFO: Running firefox for url: https://en.wikipedia.org/wiki/Barack_Obama
[2018-09-14 11:38:28] INFO: Changing network interfaces needs sudo rights.
[2018-09-14 11:38:28] INFO: Testing url https://en.wikipedia.org/wiki/Barack_Obama iteration 1
[2018-09-14 11:39:03] INFO: Waiting on har-export-trigger to collect the HAR
[2018-09-14 11:39:18] INFO: BackEndTime: 409 DomInteractiveTime: 3153 DomContentLoadedTime: 3631 FirstPaint: 2869 PageLoadTime: 6452
[2018-09-14 11:39:18] INFO: VisualMetrics FirstVisualChange: 4300 SpeedIndex: 4437 PerceptualSpeedIndex: 6036 LastVisualChange: 7233
[2018-09-14 11:39:18] INFO: Testing url https://en.wikipedia.org/wiki/Barack_Obama iteration 2
[2018-09-14 11:39:54] INFO: Waiting on har-export-trigger to collect the HAR
[2018-09-14 11:40:08] INFO: BackEndTime: 405 DomInteractiveTime: 3038 DomContentLoadedTime: 3459 FirstPaint: 2818 PageLoadTime: 6860
[2018-09-14 11:40:08] INFO: VisualMetrics FirstVisualChange: 4933 SpeedIndex: 5012 PerceptualSpeedIndex: 5922 LastVisualChange: 6733
[2018-09-14 11:40:09] INFO: Testing url https://en.wikipedia.org/wiki/Barack_Obama iteration 3
[2018-09-14 11:40:44] INFO: Waiting on har-export-trigger to collect the HAR
[2018-09-14 11:40:59] INFO: BackEndTime: 402 DomInteractiveTime: 3149 DomContentLoadedTime: 3582 FirstPaint: 2735 PageLoadTime: 6611
[2018-09-14 11:40:59] INFO: VisualMetrics FirstVisualChange: 2833 SpeedIndex: 3078 PerceptualSpeedIndex: 5626 LastVisualChange: 7400
[2018-09-14 11:40:59] INFO: Testing url https://en.wikipedia.org/wiki/Barack_Obama iteration 4
[2018-09-14 11:41:34] INFO: Waiting on har-export-trigger to collect the HAR
[2018-09-14 11:41:47] INFO: BackEndTime: 396 DomInteractiveTime: 3387 DomContentLoadedTime: 3522 FirstPaint: 3056 PageLoadTime: 6265
[2018-09-14 11:41:47] INFO: VisualMetrics FirstVisualChange: 3800 SpeedIndex: 3950 PerceptualSpeedIndex: 5728 LastVisualChange: 7033
[2018-09-14 11:41:48] INFO: Testing url https://en.wikipedia.org/wiki/Barack_Obama iteration 5
[2018-09-14 11:42:23] INFO: Waiting on har-export-trigger to collect the HAR
[2018-09-14 11:42:37] INFO: BackEndTime: 409 DomInteractiveTime: 3408 DomContentLoadedTime: 3591 FirstPaint: 2635 PageLoadTime: 6441
[2018-09-14 11:42:37] INFO: VisualMetrics FirstVisualChange: 2733 SpeedIndex: 3017 PerceptualSpeedIndex: 5467 LastVisualChange: 7200
[2018-09-14 11:42:37] INFO: Testing url https://en.wikipedia.org/wiki/Barack_Obama iteration 6
[2018-09-14 11:43:11] INFO: Waiting on har-export-trigger to collect the HAR
[2018-09-14 11:43:25] INFO: BackEndTime: 406 DomInteractiveTime: 3171 DomContentLoadedTime: 3396 FirstPaint: 2768 PageLoadTime: 6059
[2018-09-14 11:43:25] INFO: VisualMetrics FirstVisualChange: 3366 SpeedIndex: 3591 PerceptualSpeedIndex: 5585 LastVisualChange: 7033
[2018-09-14 11:43:26] INFO: Testing url https://en.wikipedia.org/wiki/Barack_Obama iteration 7
[2018-09-14 11:44:01] INFO: Waiting on har-export-trigger to collect the HAR
[2018-09-14 11:44:17] INFO: BackEndTime: 407 DomInteractiveTime: 3264 DomContentLoadedTime: 3822 FirstPaint: 2886 PageLoadTime: 6955
[2018-09-14 11:44:17] INFO: VisualMetrics FirstVisualChange: 3300 SpeedIndex: 3468 PerceptualSpeedIndex: 5447 LastVisualChange: 6900
[2018-09-14 11:44:17] INFO: Testing url https://en.wikipedia.org/wiki/Barack_Obama iteration 8
[2018-09-14 11:44:53] INFO: Waiting on har-export-trigger to collect the HAR
[2018-09-14 11:45:09] INFO: BackEndTime: 426 DomInteractiveTime: 3400 DomContentLoadedTime: 3590 FirstPaint: 2735 PageLoadTime: 6351
[2018-09-14 11:45:09] INFO: VisualMetrics FirstVisualChange: 2833 SpeedIndex: 3146 PerceptualSpeedIndex: 5447 LastVisualChange: 7066
[2018-09-14 11:45:10] INFO: Testing url https://en.wikipedia.org/wiki/Barack_Obama iteration 9
[2018-09-14 11:45:46] INFO: Waiting on har-export-trigger to collect the HAR
[2018-09-14 11:46:01] INFO: BackEndTime: 403 DomInteractiveTime: 3127 DomContentLoadedTime: 3575 FirstPaint: 2835 PageLoadTime: 6536
[2018-09-14 11:46:01] INFO: VisualMetrics FirstVisualChange: 3766 SpeedIndex: 3931 PerceptualSpeedIndex: 5891 LastVisualChange: 7233
[2018-09-14 11:46:02] INFO: Testing url https://en.wikipedia.org/wiki/Barack_Obama iteration 10
[2018-09-14 11:46:38] INFO: Waiting on har-export-trigger to collect the HAR
[2018-09-14 11:46:54] INFO: BackEndTime: 397 DomInteractiveTime: 3208 DomContentLoadedTime: 3814 FirstPaint: 2837 PageLoadTime: 6153
[2018-09-14 11:46:54] INFO: VisualMetrics FirstVisualChange: 2900 SpeedIndex: 3157 PerceptualSpeedIndex: 5355 LastVisualChange: 7033
[2018-09-14 11:46:54] INFO: Testing url https://en.wikipedia.org/wiki/Barack_Obama iteration 11
[2018-09-14 11:47:31] INFO: Waiting on har-export-trigger to collect the HAR
[2018-09-14 11:47:46] INFO: BackEndTime: 404 DomInteractiveTime: 3293 DomContentLoadedTime: 3730 FirstPaint: 3002 PageLoadTime: 6839
[2018-09-14 11:47:46] INFO: VisualMetrics FirstVisualChange: 3467 SpeedIndex: 3623 PerceptualSpeedIndex: 5240 LastVisualChange: 6700
[2018-09-14 11:47:46] INFO: 111 requests, 1426.71 kb, backEndTime: 406ms (±2.28ms), firstPaint: 2.83s (±34.65ms), firstVisualChange: 3.48s (±198.02ms), DOMContentLoaded: 3.61s (±38.62ms), Load: 6.50s (±84.46ms), speedIndex: 3674 (±179.52), visualComplete85: 3.48s (±198.02ms), lastVisualChange: 7.05s (±61.56ms), rumSpeedIndex: 1223 (±21.61) (11 runs)

Check the big difference for first visual change. It differs over one second.

I also runs the test on a CPU Optimized Droplets on Digital Ocean (4 GB - 2 vCPUs):

[2018-09-14 10:44:08] INFO: Versions OS: linux 4.4.0-134-generic nodejs: v8.11.1 sitespeed.io: 7.3.6 browsertime: 3.4.1 coach: 2.0.6
[2018-09-14 10:44:08] INFO: Starting firefox for analysing https://en.wikipedia.org/wiki/Barack_Obama 11 time(s)
[2018-09-14 10:44:08] INFO: Changing network interfaces needs sudo rights.
[2018-09-14 10:44:08] INFO: Testing url https://en.wikipedia.org/wiki/Barack_Obama iteration 1
[2018-09-14 10:44:52] INFO: BackEndTime: 379 DomInteractiveTime: 1724 DomContentLoadedTime: 1945 FirstPaint: 1551 PageLoadTime: 4196
[2018-09-14 10:44:52] INFO: VisualMetrics FirstVisualChange: 1616 SpeedIndex: 1883 PerceptualSpeedIndex: 3096 LastVisualChange: 4150
[2018-09-14 10:44:52] INFO: Testing url https://en.wikipedia.org/wiki/Barack_Obama iteration 2
[2018-09-14 10:45:33] INFO: BackEndTime: 369 DomInteractiveTime: 1721 DomContentLoadedTime: 1973 FirstPaint: 1618 PageLoadTime: 3469
[2018-09-14 10:45:33] INFO: VisualMetrics FirstVisualChange: 1683 SpeedIndex: 1926 PerceptualSpeedIndex: 3023 LastVisualChange: 4000
[2018-09-14 10:45:34] INFO: Testing url https://en.wikipedia.org/wiki/Barack_Obama iteration 3
[2018-09-14 10:46:17] INFO: BackEndTime: 377 DomInteractiveTime: 1777 DomContentLoadedTime: 1994 FirstPaint: 1602 PageLoadTime: 4115
[2018-09-14 10:46:17] INFO: VisualMetrics FirstVisualChange: 1650 SpeedIndex: 1894 PerceptualSpeedIndex: 3002 LastVisualChange: 4050
[2018-09-14 10:46:17] INFO: Testing url https://en.wikipedia.org/wiki/Barack_Obama iteration 4
[2018-09-14 10:47:00] INFO: BackEndTime: 366 DomInteractiveTime: 1689 DomContentLoadedTime: 1962 FirstPaint: 1605 PageLoadTime: 4197
[2018-09-14 10:47:00] INFO: VisualMetrics FirstVisualChange: 1633 SpeedIndex: 1887 PerceptualSpeedIndex: 3041 LastVisualChange: 4000
[2018-09-14 10:47:00] INFO: Testing url https://en.wikipedia.org/wiki/Barack_Obama iteration 5
[2018-09-14 10:47:41] INFO: BackEndTime: 365 DomInteractiveTime: 1767 DomContentLoadedTime: 1823 FirstPaint: 1736 PageLoadTime: 3450
[2018-09-14 10:47:41] INFO: VisualMetrics FirstVisualChange: 1817 SpeedIndex: 2046 PerceptualSpeedIndex: 3094 LastVisualChange: 4083
[2018-09-14 10:47:41] INFO: Testing url https://en.wikipedia.org/wiki/Barack_Obama iteration 6
[2018-09-14 10:48:22] INFO: BackEndTime: 372 DomInteractiveTime: 1780 DomContentLoadedTime: 2000 FirstPaint: 1584 PageLoadTime: 3482
[2018-09-14 10:48:22] INFO: VisualMetrics FirstVisualChange: 1600 SpeedIndex: 1851 PerceptualSpeedIndex: 3004 LastVisualChange: 4033
[2018-09-14 10:48:22] INFO: Testing url https://en.wikipedia.org/wiki/Barack_Obama iteration 7
[2018-09-14 10:49:03] INFO: BackEndTime: 382 DomInteractiveTime: 1725 DomContentLoadedTime: 1959 FirstPaint: 1585 PageLoadTime: 3581
[2018-09-14 10:49:03] INFO: VisualMetrics FirstVisualChange: 1633 SpeedIndex: 1888 PerceptualSpeedIndex: 3050 LastVisualChange: 4050
[2018-09-14 10:49:04] INFO: Testing url https://en.wikipedia.org/wiki/Barack_Obama iteration 8
[2018-09-14 10:49:45] INFO: BackEndTime: 367 DomInteractiveTime: 1774 DomContentLoadedTime: 1857 FirstPaint: 1601 PageLoadTime: 3950
[2018-09-14 10:49:45] INFO: VisualMetrics FirstVisualChange: 1633 SpeedIndex: 1866 PerceptualSpeedIndex: 2928 LastVisualChange: 3833
[2018-09-14 10:49:45] INFO: Testing url https://en.wikipedia.org/wiki/Barack_Obama iteration 9
[2018-09-14 10:50:26] INFO: BackEndTime: 392 DomInteractiveTime: 1735 DomContentLoadedTime: 1997 FirstPaint: 1651 PageLoadTime: 3347
[2018-09-14 10:50:26] INFO: VisualMetrics FirstVisualChange: 1716 SpeedIndex: 1955 PerceptualSpeedIndex: 3046 LastVisualChange: 4033
[2018-09-14 10:50:27] INFO: Testing url https://en.wikipedia.org/wiki/Barack_Obama iteration 10
[2018-09-14 10:51:08] INFO: BackEndTime: 383 DomInteractiveTime: 1797 DomContentLoadedTime: 2464 FirstPaint: 1769 PageLoadTime: 3699
[2018-09-14 10:51:08] INFO: VisualMetrics FirstVisualChange: 1833 SpeedIndex: 2080 PerceptualSpeedIndex: 3200 LastVisualChange: 4250
[2018-09-14 10:51:09] INFO: Testing url https://en.wikipedia.org/wiki/Barack_Obama iteration 11
[2018-09-14 10:51:51] INFO: BackEndTime: 379 DomInteractiveTime: 1752 DomContentLoadedTime: 2019 FirstPaint: 1586 PageLoadTime: 4142
[2018-09-14 10:51:51] INFO: VisualMetrics FirstVisualChange: 1666 SpeedIndex: 1909 PerceptualSpeedIndex: 3017 LastVisualChange: 3950
[2018-09-14 10:51:51] INFO: 0, backEndTime: 376ms (±2.45ms), firstPaint: 1.63s (±19.40ms), firstVisualChange: 1.68s (±22.58ms), DOMContentLoaded: 2s (±47.62ms), Load: 3.78s (±97.36ms), speedIndex: 1926 (±21.19), visualComplete85: 1.68s (±22.58ms), lastVisualChange: 4.04s (±30.49ms), rumSpeedIndex: 804 (±12.15) (11 runs)

On that machine the difference is a little bit more than 200 ms.

I think we should try to beef up one machine on AWS and try on that.

I've pushed c4.xlarge and a c5.large to see if there is any diff in the metrics. We run tests for FF and Chrome but emulated mobile, the same setup as our normal tests (30fps for the video). Let us run that today and then we can also try to increase fps to 60 (as we do for mobile) to see if we can get even better metrics.

I tried out a couple of different settings and will keep the instance running for the rest of tomorrow.

First things first: Is Firefox more stable on a bigger server?

c5.large

First look at Chrome:

[2018-09-17 11:47:27] INFO: Running chrome for url: https://en.wikipedia.org/wiki/Barack_Obama
[2018-09-17 11:47:27] INFO: Changing network interfaces needs sudo rights.
[2018-09-17 11:47:27] INFO: Testing url https://en.wikipedia.org/wiki/Barack_Obama iteration 1
[2018-09-17 11:48:18] INFO: BackEndTime: 353 DomInteractiveTime: 1428 DomContentLoadedTime: 1429 FirstPaint: 860 PageLoadTime: 2331
[2018-09-17 11:48:18] INFO: VisualMetrics FirstVisualChange: 867 SpeedIndex: 943 PerceptualSpeedIndex: 1761 LastVisualChange: 2333
[2018-09-17 11:48:18] INFO: Testing url https://en.wikipedia.org/wiki/Barack_Obama iteration 2
[2018-09-17 11:49:09] INFO: BackEndTime: 356 DomInteractiveTime: 1268 DomContentLoadedTime: 1268 FirstPaint: 842 PageLoadTime: 2412
[2018-09-17 11:49:09] INFO: VisualMetrics FirstVisualChange: 833 SpeedIndex: 923 PerceptualSpeedIndex: 1797 LastVisualChange: 2467
[2018-09-17 11:49:09] INFO: Testing url https://en.wikipedia.org/wiki/Barack_Obama iteration 3
[2018-09-17 11:50:00] INFO: BackEndTime: 355 DomInteractiveTime: 1342 DomContentLoadedTime: 1342 FirstPaint: 1058 PageLoadTime: 2531
[2018-09-17 11:50:00] INFO: VisualMetrics FirstVisualChange: 1067 SpeedIndex: 1140 PerceptualSpeedIndex: 1926 LastVisualChange: 2533
[2018-09-17 11:50:00] INFO: Testing url https://en.wikipedia.org/wiki/Barack_Obama iteration 4
[2018-09-17 11:50:53] INFO: BackEndTime: 355 DomInteractiveTime: 1367 DomContentLoadedTime: 1367 FirstPaint: 811 PageLoadTime: 2485
[2018-09-17 11:50:53] INFO: VisualMetrics FirstVisualChange: 800 SpeedIndex: 898 PerceptualSpeedIndex: 1822 LastVisualChange: 2533
[2018-09-17 11:50:53] INFO: Testing url https://en.wikipedia.org/wiki/Barack_Obama iteration 5
[2018-09-17 11:51:46] INFO: BackEndTime: 350 DomInteractiveTime: 1314 DomContentLoadedTime: 1314 FirstPaint: 806 PageLoadTime: 2501
[2018-09-17 11:51:46] INFO: VisualMetrics FirstVisualChange: 800 SpeedIndex: 900 PerceptualSpeedIndex: 1844 LastVisualChange: 2533
[2018-09-17 11:51:47] INFO: Testing url https://en.wikipedia.org/wiki/Barack_Obama iteration 6
[2018-09-17 11:52:35] INFO: BackEndTime: 359 DomInteractiveTime: 1214 DomContentLoadedTime: 1214 FirstPaint: 840 PageLoadTime: 2476
[2018-09-17 11:52:35] INFO: VisualMetrics FirstVisualChange: 833 SpeedIndex: 930 PerceptualSpeedIndex: 1837 LastVisualChange: 2500
[2018-09-17 11:52:36] INFO: Testing url https://en.wikipedia.org/wiki/Barack_Obama iteration 7
[2018-09-17 11:53:24] INFO: BackEndTime: 356 DomInteractiveTime: 1299 DomContentLoadedTime: 1300 FirstPaint: 859 PageLoadTime: 2410
[2018-09-17 11:53:24] INFO: VisualMetrics FirstVisualChange: 867 SpeedIndex: 955 PerceptualSpeedIndex: 1814 LastVisualChange: 2467
[2018-09-17 11:53:24] INFO: Testing url https://en.wikipedia.org/wiki/Barack_Obama iteration 8
[2018-09-17 11:54:12] INFO: BackEndTime: 355 DomInteractiveTime: 1137 DomContentLoadedTime: 1137 FirstPaint: 902 PageLoadTime: 2414
[2018-09-17 11:54:12] INFO: VisualMetrics FirstVisualChange: 900 SpeedIndex: 980 PerceptualSpeedIndex: 1834 LastVisualChange: 2467
[2018-09-17 11:54:12] INFO: Testing url https://en.wikipedia.org/wiki/Barack_Obama iteration 9
[2018-09-17 11:55:01] INFO: BackEndTime: 350 DomInteractiveTime: 1223 DomContentLoadedTime: 1223 FirstPaint: 841 PageLoadTime: 2425
[2018-09-17 11:55:01] INFO: VisualMetrics FirstVisualChange: 833 SpeedIndex: 927 PerceptualSpeedIndex: 1783 LastVisualChange: 2433
[2018-09-17 11:55:01] INFO: Testing url https://en.wikipedia.org/wiki/Barack_Obama iteration 10
[2018-09-17 11:55:51] INFO: BackEndTime: 351 DomInteractiveTime: 1276 DomContentLoadedTime: 1277 FirstPaint: 855 PageLoadTime: 2435
[2018-09-17 11:55:51] INFO: VisualMetrics FirstVisualChange: 867 SpeedIndex: 961 PerceptualSpeedIndex: 1831 LastVisualChange: 2500
[2018-09-17 11:55:51] INFO: Testing url https://en.wikipedia.org/wiki/Barack_Obama iteration 11
[2018-09-17 11:56:41] INFO: BackEndTime: 350 DomInteractiveTime: 1181 DomContentLoadedTime: 1181 FirstPaint: 842 PageLoadTime: 2435
[2018-09-17 11:56:41] INFO: VisualMetrics FirstVisualChange: 833 SpeedIndex: 926 PerceptualSpeedIndex: 1807 LastVisualChange: 2500
[2018-09-17 11:56:41] INFO: 112 requests, 1365.48 kb, backEndTime: 354ms (±0.87ms), firstPaint: 865ms (±19.80ms), firstVisualChange: 864ms (±21.24ms), DOMContentLoaded: 1.28s (±24.58ms), Load: 2.44s (±15.70ms), speedIndex: 953 (±19.18), visualComplete85: 864ms (±21.24ms), lastVisualChange: 2.48s (±16.75ms), rumSpeedIndex: 915 (±19.12) (11 runs)

And then FF:

[2018-09-17 11:57:12] INFO: Running firefox for url: https://en.wikipedia.org/wiki/Barack_Obama
[2018-09-17 11:57:12] INFO: Changing network interfaces needs sudo rights.
[2018-09-17 11:57:12] INFO: Testing url https://en.wikipedia.org/wiki/Barack_Obama iteration 1
[2018-09-17 11:57:43] INFO: Waiting on har-export-trigger to collect the HAR
[2018-09-17 11:57:55] INFO: BackEndTime: 410 DomInteractiveTime: 2417 DomContentLoadedTime: 2475 FirstPaint: 2204 PageLoadTime: 4261
[2018-09-17 11:57:55] INFO: VisualMetrics FirstVisualChange: 2300 SpeedIndex: 2448 PerceptualSpeedIndex: 3917 LastVisualChange: 5033
[2018-09-17 11:57:55] INFO: Testing url https://en.wikipedia.org/wiki/Barack_Obama iteration 2
[2018-09-17 11:58:26] INFO: Waiting on har-export-trigger to collect the HAR
[2018-09-17 11:58:37] INFO: BackEndTime: 414 DomInteractiveTime: 2302 DomContentLoadedTime: 2629 FirstPaint: 2001 PageLoadTime: 4563
[2018-09-17 11:58:37] INFO: VisualMetrics FirstVisualChange: 2100 SpeedIndex: 2262 PerceptualSpeedIndex: 3937 LastVisualChange: 5100
[2018-09-17 11:58:37] INFO: Testing url https://en.wikipedia.org/wiki/Barack_Obama iteration 3
[2018-09-17 11:59:08] INFO: Waiting on har-export-trigger to collect the HAR
[2018-09-17 11:59:19] INFO: BackEndTime: 395 DomInteractiveTime: 2390 DomContentLoadedTime: 2747 FirstPaint: 2218 PageLoadTime: 4528
[2018-09-17 11:59:19] INFO: VisualMetrics FirstVisualChange: 2333 SpeedIndex: 2476 PerceptualSpeedIndex: 4081 LastVisualChange: 5200
[2018-09-17 11:59:19] INFO: Testing url https://en.wikipedia.org/wiki/Barack_Obama iteration 4
[2018-09-17 11:59:50] INFO: Waiting on har-export-trigger to collect the HAR
[2018-09-17 12:00:01] INFO: BackEndTime: 384 DomInteractiveTime: 2330 DomContentLoadedTime: 2385 FirstPaint: 2002 PageLoadTime: 4156
[2018-09-17 12:00:01] INFO: VisualMetrics FirstVisualChange: 2100 SpeedIndex: 2241 PerceptualSpeedIndex: 3678 LastVisualChange: 4700
[2018-09-17 12:00:01] INFO: Testing url https://en.wikipedia.org/wiki/Barack_Obama iteration 5
[2018-09-17 12:00:32] INFO: Waiting on har-export-trigger to collect the HAR
[2018-09-17 12:00:43] INFO: BackEndTime: 378 DomInteractiveTime: 2446 DomContentLoadedTime: 2811 FirstPaint: 2269 PageLoadTime: 4553
[2018-09-17 12:00:43] INFO: VisualMetrics FirstVisualChange: 2433 SpeedIndex: 2566 PerceptualSpeedIndex: 4051 LastVisualChange: 5100
[2018-09-17 12:00:43] INFO: Testing url https://en.wikipedia.org/wiki/Barack_Obama iteration 6
[2018-09-17 12:01:14] INFO: Waiting on har-export-trigger to collect the HAR
[2018-09-17 12:01:26] INFO: BackEndTime: 379 DomInteractiveTime: 2396 DomContentLoadedTime: 2742 FirstPaint: 2202 PageLoadTime: 4390
[2018-09-17 12:01:26] INFO: VisualMetrics FirstVisualChange: 2300 SpeedIndex: 2457 PerceptualSpeedIndex: 3903 LastVisualChange: 4933
[2018-09-17 12:01:26] INFO: Testing url https://en.wikipedia.org/wiki/Barack_Obama iteration 7
[2018-09-17 12:01:57] INFO: Waiting on har-export-trigger to collect the HAR
[2018-09-17 12:02:08] INFO: BackEndTime: 381 DomInteractiveTime: 2395 DomContentLoadedTime: 2664 FirstPaint: 2168 PageLoadTime: 4326
[2018-09-17 12:02:08] INFO: VisualMetrics FirstVisualChange: 2300 SpeedIndex: 2435 PerceptualSpeedIndex: 3876 LastVisualChange: 4966
[2018-09-17 12:02:08] INFO: Testing url https://en.wikipedia.org/wiki/Barack_Obama iteration 8
[2018-09-17 12:02:40] INFO: Waiting on har-export-trigger to collect the HAR
[2018-09-17 12:02:51] INFO: BackEndTime: 413 DomInteractiveTime: 2296 DomContentLoadedTime: 2411 FirstPaint: 2001 PageLoadTime: 4732
[2018-09-17 12:02:51] INFO: VisualMetrics FirstVisualChange: 2100 SpeedIndex: 2243 PerceptualSpeedIndex: 3682 LastVisualChange: 4833
[2018-09-17 12:02:51] INFO: Testing url https://en.wikipedia.org/wiki/Barack_Obama iteration 9
[2018-09-17 12:03:22] INFO: Waiting on har-export-trigger to collect the HAR
[2018-09-17 12:03:34] INFO: BackEndTime: 391 DomInteractiveTime: 2426 DomContentLoadedTime: 2500 FirstPaint: 2068 PageLoadTime: 4413
[2018-09-17 12:03:34] INFO: VisualMetrics FirstVisualChange: 2166 SpeedIndex: 2351 PerceptualSpeedIndex: 3934 LastVisualChange: 5100
[2018-09-17 12:03:34] INFO: Testing url https://en.wikipedia.org/wiki/Barack_Obama iteration 10
[2018-09-17 12:04:05] INFO: Waiting on har-export-trigger to collect the HAR
[2018-09-17 12:04:16] INFO: BackEndTime: 404 DomInteractiveTime: 2407 DomContentLoadedTime: 2743 FirstPaint: 2220 PageLoadTime: 4239
[2018-09-17 12:04:16] INFO: VisualMetrics FirstVisualChange: 2333 SpeedIndex: 2476 PerceptualSpeedIndex: 4015 LastVisualChange: 5066
[2018-09-17 12:04:16] INFO: Testing url https://en.wikipedia.org/wiki/Barack_Obama iteration 11
[2018-09-17 12:04:48] INFO: Waiting on har-export-trigger to collect the HAR
[2018-09-17 12:04:59] INFO: BackEndTime: 385 DomInteractiveTime: 2322 DomContentLoadedTime: 2390 FirstPaint: 2001 PageLoadTime: 4802
[2018-09-17 12:04:59] INFO: VisualMetrics FirstVisualChange: 2100 SpeedIndex: 2270 PerceptualSpeedIndex: 3744 LastVisualChange: 5066
[2018-09-17 12:04:59] INFO: 111 requests, 1426.28 kb, backEndTime: 394ms (±4.02ms), firstPaint: 2.12s (±31.08ms), firstVisualChange: 2.23s (±35.06ms), DOMContentLoaded: 2.59s (±46.57ms), Load: 4.45s (±58.91ms), speedIndex: 2384 (±32.95), visualComplete85: 2.23s (±35.06ms), lastVisualChange: 5.01s (±40.91ms), rumSpeedIndex: 953 (±14.67) (11 runs)

It's not perfect but much better than c4.large. We also have the problem URL https://en.wikipedia.org/wiki/Sweden
Lets check Chrome first:

[2018-09-17 10:57:01] INFO: Running chrome for url: https://en.wikipedia.org/wiki/Sweden
[2018-09-17 10:57:01] INFO: Changing network interfaces needs sudo rights.
[2018-09-17 10:57:01] INFO: Testing url https://en.wikipedia.org/wiki/Sweden iteration 1
[2018-09-17 10:57:49] INFO: BackEndTime: 340 DomInteractiveTime: 1197 DomContentLoadedTime: 1197 FirstPaint: 967 PageLoadTime: 2583
[2018-09-17 10:57:49] INFO: VisualMetrics FirstVisualChange: 967 SpeedIndex: 1057 PerceptualSpeedIndex: 1709 LastVisualChange: 2500
[2018-09-17 10:57:49] INFO: Testing url https://en.wikipedia.org/wiki/Sweden iteration 2
[2018-09-17 10:58:37] INFO: BackEndTime: 338 DomInteractiveTime: 1070 DomContentLoadedTime: 1070 FirstPaint: 927 PageLoadTime: 2601
[2018-09-17 10:58:37] INFO: VisualMetrics FirstVisualChange: 933 SpeedIndex: 1019 PerceptualSpeedIndex: 1695 LastVisualChange: 2533
[2018-09-17 10:58:37] INFO: Testing url https://en.wikipedia.org/wiki/Sweden iteration 3
[2018-09-17 10:59:26] INFO: BackEndTime: 339 DomInteractiveTime: 1030 DomContentLoadedTime: 1030 FirstPaint: 904 PageLoadTime: 2663
[2018-09-17 10:59:26] INFO: VisualMetrics FirstVisualChange: 900 SpeedIndex: 989 PerceptualSpeedIndex: 1672 LastVisualChange: 2500
[2018-09-17 10:59:26] INFO: Testing url https://en.wikipedia.org/wiki/Sweden iteration 4
[2018-09-17 11:00:14] INFO: BackEndTime: 342 DomInteractiveTime: 1082 DomContentLoadedTime: 1082 FirstPaint: 864 PageLoadTime: 2595
[2018-09-17 11:00:14] INFO: VisualMetrics FirstVisualChange: 867 SpeedIndex: 959 PerceptualSpeedIndex: 1673 LastVisualChange: 2500
[2018-09-17 11:00:14] INFO: Testing url https://en.wikipedia.org/wiki/Sweden iteration 5
[2018-09-17 11:01:02] INFO: BackEndTime: 341 DomInteractiveTime: 1202 DomContentLoadedTime: 1202 FirstPaint: 958 PageLoadTime: 2579
[2018-09-17 11:01:02] INFO: VisualMetrics FirstVisualChange: 967 SpeedIndex: 1048 PerceptualSpeedIndex: 1703 LastVisualChange: 2533
[2018-09-17 11:01:02] INFO: Testing url https://en.wikipedia.org/wiki/Sweden iteration 6
[2018-09-17 11:01:51] INFO: BackEndTime: 341 DomInteractiveTime: 1216 DomContentLoadedTime: 1216 FirstPaint: 877 PageLoadTime: 2648
[2018-09-17 11:01:51] INFO: VisualMetrics FirstVisualChange: 867 SpeedIndex: 991 PerceptualSpeedIndex: 1649 LastVisualChange: 2533
[2018-09-17 11:01:51] INFO: Testing url https://en.wikipedia.org/wiki/Sweden iteration 7
[2018-09-17 11:02:40] INFO: BackEndTime: 342 DomInteractiveTime: 1153 DomContentLoadedTime: 1153 FirstPaint: 901 PageLoadTime: 2607
[2018-09-17 11:02:40] INFO: VisualMetrics FirstVisualChange: 900 SpeedIndex: 993 PerceptualSpeedIndex: 1680 LastVisualChange: 2500
[2018-09-17 11:02:40] INFO: Testing url https://en.wikipedia.org/wiki/Sweden iteration 8
[2018-09-17 11:03:28] INFO: BackEndTime: 340 DomInteractiveTime: 940 DomContentLoadedTime: 940 FirstPaint: 972 PageLoadTime: 2615
[2018-09-17 11:03:28] INFO: VisualMetrics FirstVisualChange: 967 SpeedIndex: 1044 PerceptualSpeedIndex: 1707 LastVisualChange: 2567
[2018-09-17 11:03:28] INFO: Testing url https://en.wikipedia.org/wiki/Sweden iteration 9
[2018-09-17 11:04:17] INFO: BackEndTime: 342 DomInteractiveTime: 1206 DomContentLoadedTime: 1206 FirstPaint: 897 PageLoadTime: 2666
[2018-09-17 11:04:17] INFO: VisualMetrics FirstVisualChange: 867 SpeedIndex: 999 PerceptualSpeedIndex: 1717 LastVisualChange: 2600
[2018-09-17 11:04:17] INFO: Testing url https://en.wikipedia.org/wiki/Sweden iteration 10
[2018-09-17 11:05:05] INFO: BackEndTime: 340 DomInteractiveTime: 1154 DomContentLoadedTime: 1154 FirstPaint: 925 PageLoadTime: 2622
[2018-09-17 11:05:05] INFO: VisualMetrics FirstVisualChange: 900 SpeedIndex: 1000 PerceptualSpeedIndex: 1665 LastVisualChange: 2533
[2018-09-17 11:05:05] INFO: Testing url https://en.wikipedia.org/wiki/Sweden iteration 11
[2018-09-17 11:05:53] INFO: BackEndTime: 341 DomInteractiveTime: 1176 DomContentLoadedTime: 1176 FirstPaint: 911 PageLoadTime: 2642
[2018-09-17 11:05:53] INFO: VisualMetrics FirstVisualChange: 900 SpeedIndex: 1000 PerceptualSpeedIndex: 1670 LastVisualChange: 2533
[2018-09-17 11:05:53] INFO: 149 requests, 2084.67 kb, backEndTime: 341ms (±0.37ms), firstPaint: 918ms (±10.21ms), firstVisualChange: 912ms (±11.56ms), DOMContentLoaded: 1.13s (±25.40ms), Load: 2.62s (±8.85ms), speedIndex: 1009 (±8.57), visualComplete85: 921ms (±9.86ms), lastVisualChange: 2.53s (±9.05ms), rumSpeedIndex: 1045 (±9.58) (11 runs)

Looks good! And then FF:

[2018-09-17 11:06:24] INFO: Running firefox for url: https://en.wikipedia.org/wiki/Sweden
[2018-09-17 11:06:24] INFO: Changing network interfaces needs sudo rights.
[2018-09-17 11:06:24] INFO: Testing url https://en.wikipedia.org/wiki/Sweden iteration 1
[2018-09-17 11:06:54] INFO: Waiting on har-export-trigger to collect the HAR
[2018-09-17 11:07:05] INFO: BackEndTime: 387 DomInteractiveTime: 2182 DomContentLoadedTime: 2267 FirstPaint: 1968 PageLoadTime: 3960
[2018-09-17 11:07:05] INFO: VisualMetrics FirstVisualChange: 2566 SpeedIndex: 2647 PerceptualSpeedIndex: 3276 LastVisualChange: 4000
[2018-09-17 11:07:06] INFO: Testing url https://en.wikipedia.org/wiki/Sweden iteration 2
[2018-09-17 11:07:36] INFO: Waiting on har-export-trigger to collect the HAR
[2018-09-17 11:07:47] INFO: BackEndTime: 376 DomInteractiveTime: 2327 DomContentLoadedTime: 2404 FirstPaint: 1970 PageLoadTime: 4070
[2018-09-17 11:07:47] INFO: VisualMetrics FirstVisualChange: 2033 SpeedIndex: 2139 PerceptualSpeedIndex: 3199 LastVisualChange: 4166
[2018-09-17 11:07:48] INFO: Testing url https://en.wikipedia.org/wiki/Sweden iteration 3
[2018-09-17 11:08:18] INFO: Waiting on har-export-trigger to collect the HAR
[2018-09-17 11:08:29] INFO: BackEndTime: 363 DomInteractiveTime: 2034 DomContentLoadedTime: 2131 FirstPaint: 1853 PageLoadTime: 3866
[2018-09-17 11:08:29] INFO: VisualMetrics FirstVisualChange: 3066 SpeedIndex: 3102 PerceptualSpeedIndex: 3474 LastVisualChange: 4000
[2018-09-17 11:08:29] INFO: Testing url https://en.wikipedia.org/wiki/Sweden iteration 4
[2018-09-17 11:08:59] INFO: Waiting on har-export-trigger to collect the HAR
[2018-09-17 11:09:09] INFO: BackEndTime: 362 DomInteractiveTime: 2204 DomContentLoadedTime: 2327 FirstPaint: 1935 PageLoadTime: 3913
[2018-09-17 11:09:09] INFO: VisualMetrics FirstVisualChange: 2833 SpeedIndex: 2906 PerceptualSpeedIndex: 3493 LastVisualChange: 3933
[2018-09-17 11:09:10] INFO: Testing url https://en.wikipedia.org/wiki/Sweden iteration 5
[2018-09-17 11:09:40] INFO: Waiting on har-export-trigger to collect the HAR
[2018-09-17 11:09:51] INFO: BackEndTime: 367 DomInteractiveTime: 2299 DomContentLoadedTime: 2368 FirstPaint: 2002 PageLoadTime: 4185
[2018-09-17 11:09:51] INFO: VisualMetrics FirstVisualChange: 2066 SpeedIndex: 2181 PerceptualSpeedIndex: 3258 LastVisualChange: 4233
[2018-09-17 11:09:52] INFO: Testing url https://en.wikipedia.org/wiki/Sweden iteration 6
[2018-09-17 11:10:22] INFO: Waiting on har-export-trigger to collect the HAR
[2018-09-17 11:10:33] INFO: BackEndTime: 385 DomInteractiveTime: 2274 DomContentLoadedTime: 2402 FirstPaint: 2054 PageLoadTime: 4215
[2018-09-17 11:10:33] INFO: VisualMetrics FirstVisualChange: 3300 SpeedIndex: 3321 PerceptualSpeedIndex: 3519 LastVisualChange: 4166
[2018-09-17 11:10:33] INFO: Testing url https://en.wikipedia.org/wiki/Sweden iteration 7
[2018-09-17 11:11:03] INFO: Waiting on har-export-trigger to collect the HAR
[2018-09-17 11:11:13] INFO: BackEndTime: 364 DomInteractiveTime: 2180 DomContentLoadedTime: 2308 FirstPaint: 1869 PageLoadTime: 3711
[2018-09-17 11:11:13] INFO: VisualMetrics FirstVisualChange: 2900 SpeedIndex: 2967 PerceptualSpeedIndex: 3488 LastVisualChange: 4233
[2018-09-17 11:11:14] INFO: Testing url https://en.wikipedia.org/wiki/Sweden iteration 8
[2018-09-17 11:11:44] INFO: Waiting on har-export-trigger to collect the HAR
[2018-09-17 11:11:56] INFO: BackEndTime: 360 DomInteractiveTime: 2197 DomContentLoadedTime: 2323 FirstPaint: 1821 PageLoadTime: 4072
[2018-09-17 11:11:56] INFO: VisualMetrics FirstVisualChange: 1900 SpeedIndex: 2017 PerceptualSpeedIndex: 3088 LastVisualChange: 4200
[2018-09-17 11:11:56] INFO: Testing url https://en.wikipedia.org/wiki/Sweden iteration 9
[2018-09-17 11:12:26] INFO: Waiting on har-export-trigger to collect the HAR
[2018-09-17 11:12:37] INFO: BackEndTime: 376 DomInteractiveTime: 2062 DomContentLoadedTime: 2164 FirstPaint: 1872 PageLoadTime: 3984
[2018-09-17 11:12:37] INFO: VisualMetrics FirstVisualChange: 2566 SpeedIndex: 2655 PerceptualSpeedIndex: 3387 LastVisualChange: 4000
[2018-09-17 11:12:37] INFO: Testing url https://en.wikipedia.org/wiki/Sweden iteration 10
[2018-09-17 11:13:08] INFO: Waiting on har-export-trigger to collect the HAR
[2018-09-17 11:13:19] INFO: BackEndTime: 362 DomInteractiveTime: 2301 DomContentLoadedTime: 2418 FirstPaint: 1986 PageLoadTime: 4175
[2018-09-17 11:13:19] INFO: VisualMetrics FirstVisualChange: 2066 SpeedIndex: 2169 PerceptualSpeedIndex: 3298 LastVisualChange: 4266
[2018-09-17 11:13:19] INFO: Testing url https://en.wikipedia.org/wiki/Sweden iteration 11
[2018-09-17 11:13:50] INFO: Waiting on har-export-trigger to collect the HAR
[2018-09-17 11:14:01] INFO: BackEndTime: 374 DomInteractiveTime: 2229 DomContentLoadedTime: 2298 FirstPaint: 1968 PageLoadTime: 4114
[2018-09-17 11:14:01] INFO: VisualMetrics FirstVisualChange: 2066 SpeedIndex: 2153 PerceptualSpeedIndex: 3091 LastVisualChange: 4166
[2018-09-17 11:14:01] INFO: 148 requests, 2166.42 kb, backEndTime: 371ms (±2.76ms), firstPaint: 1.94s (±20.87ms), firstVisualChange: 2.49s (±140.29ms), DOMContentLoaded: 2.31s (±26.99ms), Load: 4.02s (±44.41ms), speedIndex: 2569 (±132.00), visualComplete85: 2.49s (±140.29ms), lastVisualChange: 4.12s (±33.72ms), rumSpeedIndex: 1091 (±11.45) (11 runs)

Hmm here we we see the same pattern as before.

c4.xlarge

Chrome

[2018-09-17 11:05:23] INFO: Running chrome for url: https://en.wikipedia.org/wiki/Barack_Obama
[2018-09-17 11:05:23] INFO: Changing network interfaces needs sudo rights.
[2018-09-17 11:05:23] INFO: Testing url https://en.wikipedia.org/wiki/Barack_Obama iteration 1
[2018-09-17 11:06:13] INFO: BackEndTime: 354 DomInteractiveTime: 1119 DomContentLoadedTime: 1120 FirstPaint: 955 PageLoadTime: 2296
[2018-09-17 11:06:13] INFO: VisualMetrics FirstVisualChange: 967 SpeedIndex: 1037 PerceptualSpeedIndex: 1737 LastVisualChange: 2300
[2018-09-17 11:06:13] INFO: Testing url https://en.wikipedia.org/wiki/Barack_Obama iteration 2
[2018-09-17 11:06:57] INFO: BackEndTime: 354 DomInteractiveTime: 1189 DomContentLoadedTime: 1189 FirstPaint: 909 PageLoadTime: 2289
[2018-09-17 11:06:57] INFO: VisualMetrics FirstVisualChange: 933 SpeedIndex: 1003 PerceptualSpeedIndex: 1727 LastVisualChange: 2300
[2018-09-17 11:06:57] INFO: Testing url https://en.wikipedia.org/wiki/Barack_Obama iteration 3
[2018-09-17 11:07:42] INFO: BackEndTime: 356 DomInteractiveTime: 1189 DomContentLoadedTime: 1190 FirstPaint: 936 PageLoadTime: 2322
[2018-09-17 11:07:42] INFO: VisualMetrics FirstVisualChange: 933 SpeedIndex: 1005 PerceptualSpeedIndex: 1753 LastVisualChange: 2333
[2018-09-17 11:07:42] INFO: Testing url https://en.wikipedia.org/wiki/Barack_Obama iteration 4
[2018-09-17 11:08:27] INFO: BackEndTime: 354 DomInteractiveTime: 1246 DomContentLoadedTime: 1247 FirstPaint: 771 PageLoadTime: 2271
[2018-09-17 11:08:27] INFO: VisualMetrics FirstVisualChange: 767 SpeedIndex: 850 PerceptualSpeedIndex: 1683 LastVisualChange: 2267
[2018-09-17 11:08:27] INFO: Testing url https://en.wikipedia.org/wiki/Barack_Obama iteration 5
[2018-09-17 11:09:12] INFO: BackEndTime: 352 DomInteractiveTime: 1207 DomContentLoadedTime: 1207 FirstPaint: 924 PageLoadTime: 2313
[2018-09-17 11:09:12] INFO: VisualMetrics FirstVisualChange: 933 SpeedIndex: 1005 PerceptualSpeedIndex: 1753 LastVisualChange: 2333
[2018-09-17 11:09:12] INFO: Testing url https://en.wikipedia.org/wiki/Barack_Obama iteration 6
[2018-09-17 11:09:55] INFO: BackEndTime: 353 DomInteractiveTime: 1214 DomContentLoadedTime: 1214 FirstPaint: 811 PageLoadTime: 2307
[2018-09-17 11:09:55] INFO: VisualMetrics FirstVisualChange: 833 SpeedIndex: 914 PerceptualSpeedIndex: 1721 LastVisualChange: 2300
[2018-09-17 11:09:55] INFO: Testing url https://en.wikipedia.org/wiki/Barack_Obama iteration 7
[2018-09-17 11:10:40] INFO: BackEndTime: 356 DomInteractiveTime: 1245 DomContentLoadedTime: 1246 FirstPaint: 801 PageLoadTime: 2292
[2018-09-17 11:10:40] INFO: VisualMetrics FirstVisualChange: 800 SpeedIndex: 884 PerceptualSpeedIndex: 1693 LastVisualChange: 2267
[2018-09-17 11:10:40] INFO: Testing url https://en.wikipedia.org/wiki/Barack_Obama iteration 8
[2018-09-17 11:11:29] INFO: BackEndTime: 355 DomInteractiveTime: 1228 DomContentLoadedTime: 1228 FirstPaint: 793 PageLoadTime: 2256
[2018-09-17 11:11:29] INFO: VisualMetrics FirstVisualChange: 800 SpeedIndex: 881 PerceptualSpeedIndex: 1690 LastVisualChange: 2267
[2018-09-17 11:11:29] INFO: Testing url https://en.wikipedia.org/wiki/Barack_Obama iteration 9
[2018-09-17 11:12:17] INFO: BackEndTime: 352 DomInteractiveTime: 1240 DomContentLoadedTime: 1240 FirstPaint: 792 PageLoadTime: 2316
[2018-09-17 11:12:17] INFO: VisualMetrics FirstVisualChange: 800 SpeedIndex: 883 PerceptualSpeedIndex: 1711 LastVisualChange: 2333
[2018-09-17 11:12:17] INFO: Testing url https://en.wikipedia.org/wiki/Barack_Obama iteration 10
[2018-09-17 11:13:02] INFO: BackEndTime: 356 DomInteractiveTime: 1170 DomContentLoadedTime: 1170 FirstPaint: 914 PageLoadTime: 2296
[2018-09-17 11:13:02] INFO: VisualMetrics FirstVisualChange: 933 SpeedIndex: 1001 PerceptualSpeedIndex: 1733 LastVisualChange: 2300
[2018-09-17 11:13:03] INFO: Testing url https://en.wikipedia.org/wiki/Barack_Obama iteration 11
[2018-09-17 11:13:50] INFO: BackEndTime: 351 DomInteractiveTime: 1194 DomContentLoadedTime: 1194 FirstPaint: 959 PageLoadTime: 2311
[2018-09-17 11:13:50] INFO: VisualMetrics FirstVisualChange: 967 SpeedIndex: 1034 PerceptualSpeedIndex: 1763 LastVisualChange: 2300
[2018-09-17 11:13:50] INFO: 112 requests, 1365.78 kb, backEndTime: 354ms (±0.51ms), firstPaint: 870ms (±21.50ms), firstVisualChange: 879ms (±22.37ms), DOMContentLoaded: 1.20s (±10.88ms), Load: 2.30s (±5.74ms), speedIndex: 954 (±20.49), visualComplete85: 879ms (±22.37ms), lastVisualChange: 2.30s (±7.35ms), rumSpeedIndex: 914 (±20.67) (11 runs)

Firefox

[2018-09-17 11:14:21] INFO: Running firefox for url: https://en.wikipedia.org/wiki/Barack_Obama
[2018-09-17 11:14:21] INFO: Changing network interfaces needs sudo rights.
[2018-09-17 11:14:21] INFO: Testing url https://en.wikipedia.org/wiki/Barack_Obama iteration 1
[2018-09-17 11:14:50] INFO: Waiting on har-export-trigger to collect the HAR
[2018-09-17 11:15:01] INFO: BackEndTime: 392 DomInteractiveTime: 1876 DomContentLoadedTime: 2089 FirstPaint: 1684 PageLoadTime: 4100
[2018-09-17 11:15:01] INFO: VisualMetrics FirstVisualChange: 1700 SpeedIndex: 1821 PerceptualSpeedIndex: 3062 LastVisualChange: 4000
[2018-09-17 11:15:01] INFO: Testing url https://en.wikipedia.org/wiki/Barack_Obama iteration 2
[2018-09-17 11:15:30] INFO: Waiting on har-export-trigger to collect the HAR
[2018-09-17 11:15:41] INFO: BackEndTime: 439 DomInteractiveTime: 1955 DomContentLoadedTime: 2192 FirstPaint: 1801 PageLoadTime: 3921
[2018-09-17 11:15:41] INFO: VisualMetrics FirstVisualChange: 1833 SpeedIndex: 1964 PerceptualSpeedIndex: 3361 LastVisualChange: 4366
[2018-09-17 11:15:41] INFO: Testing url https://en.wikipedia.org/wiki/Barack_Obama iteration 3
[2018-09-17 11:16:09] INFO: Waiting on har-export-trigger to collect the HAR
[2018-09-17 11:16:19] INFO: BackEndTime: 382 DomInteractiveTime: 1878 DomContentLoadedTime: 2198 FirstPaint: 1684 PageLoadTime: 3344
[2018-09-17 11:16:19] INFO: VisualMetrics FirstVisualChange: 1733 SpeedIndex: 1843 PerceptualSpeedIndex: 2967 LastVisualChange: 3800
[2018-09-17 11:16:20] INFO: Testing url https://en.wikipedia.org/wiki/Barack_Obama iteration 4
[2018-09-17 11:16:48] INFO: Waiting on har-export-trigger to collect the HAR
[2018-09-17 11:16:58] INFO: BackEndTime: 382 DomInteractiveTime: 1866 DomContentLoadedTime: 1932 FirstPaint: 1668 PageLoadTime: 3383
[2018-09-17 11:16:58] INFO: VisualMetrics FirstVisualChange: 1700 SpeedIndex: 1816 PerceptualSpeedIndex: 3014 LastVisualChange: 3866
[2018-09-17 11:16:58] INFO: Testing url https://en.wikipedia.org/wiki/Barack_Obama iteration 5
[2018-09-17 11:17:27] INFO: Waiting on har-export-trigger to collect the HAR
[2018-09-17 11:17:37] INFO: BackEndTime: 380 DomInteractiveTime: 1999 DomContentLoadedTime: 2058 FirstPaint: 1835 PageLoadTime: 3533
[2018-09-17 11:17:37] INFO: VisualMetrics FirstVisualChange: 1866 SpeedIndex: 1977 PerceptualSpeedIndex: 3138 LastVisualChange: 4000
[2018-09-17 11:17:37] INFO: Testing url https://en.wikipedia.org/wiki/Barack_Obama iteration 6
[2018-09-17 11:18:06] INFO: Waiting on har-export-trigger to collect the HAR
[2018-09-17 11:18:17] INFO: BackEndTime: 384 DomInteractiveTime: 1923 DomContentLoadedTime: 2137 FirstPaint: 1717 PageLoadTime: 3932
[2018-09-17 11:18:17] INFO: VisualMetrics FirstVisualChange: 1766 SpeedIndex: 1885 PerceptualSpeedIndex: 3124 LastVisualChange: 3933
[2018-09-17 11:18:17] INFO: Testing url https://en.wikipedia.org/wiki/Barack_Obama iteration 7
[2018-09-17 11:18:46] INFO: Waiting on har-export-trigger to collect the HAR
[2018-09-17 11:18:56] INFO: BackEndTime: 379 DomInteractiveTime: 1924 DomContentLoadedTime: 2316 FirstPaint: 1734 PageLoadTime: 3867
[2018-09-17 11:18:56] INFO: VisualMetrics FirstVisualChange: 1766 SpeedIndex: 1885 PerceptualSpeedIndex: 3129 LastVisualChange: 3900
[2018-09-17 11:18:56] INFO: Testing url https://en.wikipedia.org/wiki/Barack_Obama iteration 8
[2018-09-17 11:19:25] INFO: Waiting on har-export-trigger to collect the HAR
[2018-09-17 11:19:36] INFO: BackEndTime: 383 DomInteractiveTime: 1877 DomContentLoadedTime: 2104 FirstPaint: 1702 PageLoadTime: 4099
[2018-09-17 11:19:36] INFO: VisualMetrics FirstVisualChange: 1733 SpeedIndex: 1850 PerceptualSpeedIndex: 3073 LastVisualChange: 4000
[2018-09-17 11:19:36] INFO: Testing url https://en.wikipedia.org/wiki/Barack_Obama iteration 9
[2018-09-17 11:20:05] INFO: Waiting on har-export-trigger to collect the HAR
[2018-09-17 11:20:15] INFO: BackEndTime: 389 DomInteractiveTime: 1831 DomContentLoadedTime: 2102 FirstPaint: 1668 PageLoadTime: 3655
[2018-09-17 11:20:15] INFO: VisualMetrics FirstVisualChange: 1700 SpeedIndex: 1828 PerceptualSpeedIndex: 3166 LastVisualChange: 4100
[2018-09-17 11:20:15] INFO: Testing url https://en.wikipedia.org/wiki/Barack_Obama iteration 10
[2018-09-17 11:20:43] INFO: Waiting on har-export-trigger to collect the HAR
[2018-09-17 11:20:54] INFO: BackEndTime: 386 DomInteractiveTime: 1832 DomContentLoadedTime: 1884 FirstPaint: 1667 PageLoadTime: 3387
[2018-09-17 11:20:54] INFO: VisualMetrics FirstVisualChange: 1700 SpeedIndex: 1811 PerceptualSpeedIndex: 2972 LastVisualChange: 3833
[2018-09-17 11:20:54] INFO: Testing url https://en.wikipedia.org/wiki/Barack_Obama iteration 11
[2018-09-17 11:21:23] INFO: Waiting on har-export-trigger to collect the HAR
[2018-09-17 11:21:33] INFO: BackEndTime: 388 DomInteractiveTime: 1861 DomContentLoadedTime: 2077 FirstPaint: 1702 PageLoadTime: 3786
[2018-09-17 11:21:33] INFO: VisualMetrics FirstVisualChange: 1733 SpeedIndex: 1845 PerceptualSpeedIndex: 3026 LastVisualChange: 3833
[2018-09-17 11:21:33] INFO: 111 requests, 1426.52 kb, backEndTime: 389ms (±4.86ms), firstPaint: 1.72s (±16.04ms), firstVisualChange: 1.75s (±16.18ms), DOMContentLoaded: 2.10s (±34.43ms), Load: 3.73s (±81.50ms), speedIndex: 1866 (±16.48), visualComplete85: 1.75s (±16.18ms), lastVisualChange: 3.97s (±46.34ms), rumSpeedIndex: 882 (±5.88) (11 runs)

Add Sweden on Chrome

[2018-09-17 11:37:30] INFO: Running chrome for url: https://en.wikipedia.org/wiki/Sweden
[2018-09-17 11:37:30] INFO: Changing network interfaces needs sudo rights.
[2018-09-17 11:37:30] INFO: Testing url https://en.wikipedia.org/wiki/Sweden iteration 1
[2018-09-17 11:38:18] INFO: BackEndTime: 341 DomInteractiveTime: 1109 DomContentLoadedTime: 1110 FirstPaint: 861 PageLoadTime: 2660
[2018-09-17 11:38:18] INFO: VisualMetrics FirstVisualChange: 867 SpeedIndex: 939 PerceptualSpeedIndex: 1537 LastVisualChange: 2267
[2018-09-17 11:38:18] INFO: Testing url https://en.wikipedia.org/wiki/Sweden iteration 2
[2018-09-17 11:39:02] INFO: BackEndTime: 339 DomInteractiveTime: 1125 DomContentLoadedTime: 1126 FirstPaint: 856 PageLoadTime: 2663
[2018-09-17 11:39:02] INFO: VisualMetrics FirstVisualChange: 867 SpeedIndex: 941 PerceptualSpeedIndex: 1536 LastVisualChange: 2300
[2018-09-17 11:39:02] INFO: Testing url https://en.wikipedia.org/wiki/Sweden iteration 3
[2018-09-17 11:39:47] INFO: BackEndTime: 340 DomInteractiveTime: 1107 DomContentLoadedTime: 1107 FirstPaint: 966 PageLoadTime: 2692
[2018-09-17 11:39:47] INFO: VisualMetrics FirstVisualChange: 967 SpeedIndex: 1032 PerceptualSpeedIndex: 1557 LastVisualChange: 2233
[2018-09-17 11:39:47] INFO: Testing url https://en.wikipedia.org/wiki/Sweden iteration 4
[2018-09-17 11:40:31] INFO: BackEndTime: 340 DomInteractiveTime: 1041 DomContentLoadedTime: 1041 FirstPaint: 836 PageLoadTime: 2665
[2018-09-17 11:40:31] INFO: VisualMetrics FirstVisualChange: 833 SpeedIndex: 907 PerceptualSpeedIndex: 1519 LastVisualChange: 2233
[2018-09-17 11:40:31] INFO: Testing url https://en.wikipedia.org/wiki/Sweden iteration 5
[2018-09-17 11:41:16] INFO: BackEndTime: 338 DomInteractiveTime: 1084 DomContentLoadedTime: 1084 FirstPaint: 884 PageLoadTime: 2629
[2018-09-17 11:41:16] INFO: VisualMetrics FirstVisualChange: 900 SpeedIndex: 969 PerceptualSpeedIndex: 1538 LastVisualChange: 2267
[2018-09-17 11:41:16] INFO: Testing url https://en.wikipedia.org/wiki/Sweden iteration 6
[2018-09-17 11:42:04] INFO: BackEndTime: 338 DomInteractiveTime: 1100 DomContentLoadedTime: 1100 FirstPaint: 860 PageLoadTime: 2646
[2018-09-17 11:42:04] INFO: VisualMetrics FirstVisualChange: 867 SpeedIndex: 941 PerceptualSpeedIndex: 1528 LastVisualChange: 2233
[2018-09-17 11:42:04] INFO: Testing url https://en.wikipedia.org/wiki/Sweden iteration 7
[2018-09-17 11:42:51] INFO: BackEndTime: 342 DomInteractiveTime: 1081 DomContentLoadedTime: 1082 FirstPaint: 871 PageLoadTime: 2650
[2018-09-17 11:42:51] INFO: VisualMetrics FirstVisualChange: 867 SpeedIndex: 945 PerceptualSpeedIndex: 1525 LastVisualChange: 2200
[2018-09-17 11:42:51] INFO: Testing url https://en.wikipedia.org/wiki/Sweden iteration 8
[2018-09-17 11:43:35] INFO: BackEndTime: 339 DomInteractiveTime: 1084 DomContentLoadedTime: 1084 FirstPaint: 863 PageLoadTime: 2586
[2018-09-17 11:43:35] INFO: VisualMetrics FirstVisualChange: 867 SpeedIndex: 942 PerceptualSpeedIndex: 1529 LastVisualChange: 2233
[2018-09-17 11:43:35] INFO: Testing url https://en.wikipedia.org/wiki/Sweden iteration 9
[2018-09-17 11:44:19] INFO: BackEndTime: 339 DomInteractiveTime: 1085 DomContentLoadedTime: 1085 FirstPaint: 862 PageLoadTime: 2628
[2018-09-17 11:44:19] INFO: VisualMetrics FirstVisualChange: 867 SpeedIndex: 945 PerceptualSpeedIndex: 1511 LastVisualChange: 2200
[2018-09-17 11:44:19] INFO: Testing url https://en.wikipedia.org/wiki/Sweden iteration 10
[2018-09-17 11:45:06] INFO: BackEndTime: 343 DomInteractiveTime: 1066 DomContentLoadedTime: 1066 FirstPaint: 891 PageLoadTime: 2613
[2018-09-17 11:45:06] INFO: VisualMetrics FirstVisualChange: 900 SpeedIndex: 976 PerceptualSpeedIndex: 1527 LastVisualChange: 2200
[2018-09-17 11:45:06] INFO: Testing url https://en.wikipedia.org/wiki/Sweden iteration 11
[2018-09-17 11:45:51] INFO: BackEndTime: 342 DomInteractiveTime: 1125 DomContentLoadedTime: 1125 FirstPaint: 892 PageLoadTime: 2650
[2018-09-17 11:45:51] INFO: VisualMetrics FirstVisualChange: 900 SpeedIndex: 971 PerceptualSpeedIndex: 1544 LastVisualChange: 2267
[2018-09-17 11:45:51] INFO: 149 requests, 2084.55 kb, backEndTime: 340ms (±0.49ms), firstPaint: 877ms (±9.75ms), firstVisualChange: 882ms (±9.94ms), DOMContentLoaded: 1.09s (±7.28ms), Load: 2.64s (±8.25ms), speedIndex: 955 (±9.16), visualComplete85: 882ms (±9.94ms), lastVisualChange: 2.24s (±9.44ms), rumSpeedIndex: 985 (±9.01) (11 runs)

And Firefox

[2018-09-17 11:46:22] INFO: Running firefox for url: https://en.wikipedia.org/wiki/Sweden
[2018-09-17 11:46:22] INFO: Changing network interfaces needs sudo rights.
[2018-09-17 11:46:22] INFO: Testing url https://en.wikipedia.org/wiki/Sweden iteration 1
[2018-09-17 11:46:50] INFO: Waiting on har-export-trigger to collect the HAR
[2018-09-17 11:47:01] INFO: BackEndTime: 370 DomInteractiveTime: 1786 DomContentLoadedTime: 1858 FirstPaint: 1568 PageLoadTime: 3201
[2018-09-17 11:47:01] INFO: VisualMetrics FirstVisualChange: 1600 SpeedIndex: 1683 PerceptualSpeedIndex: 2505 LastVisualChange: 3233
[2018-09-17 11:47:01] INFO: Testing url https://en.wikipedia.org/wiki/Sweden iteration 2
[2018-09-17 11:47:28] INFO: Waiting on har-export-trigger to collect the HAR
[2018-09-17 11:47:38] INFO: BackEndTime: 373 DomInteractiveTime: 1761 DomContentLoadedTime: 1849 FirstPaint: 1601 PageLoadTime: 3208
[2018-09-17 11:47:38] INFO: VisualMetrics FirstVisualChange: 1666 SpeedIndex: 1742 PerceptualSpeedIndex: 2478 LastVisualChange: 3266
[2018-09-17 11:47:38] INFO: Testing url https://en.wikipedia.org/wiki/Sweden iteration 3
[2018-09-17 11:48:06] INFO: Waiting on har-export-trigger to collect the HAR
[2018-09-17 11:48:16] INFO: BackEndTime: 371 DomInteractiveTime: 1685 DomContentLoadedTime: 1802 FirstPaint: 1435 PageLoadTime: 3165
[2018-09-17 11:48:16] INFO: VisualMetrics FirstVisualChange: 1466 SpeedIndex: 1555 PerceptualSpeedIndex: 2452 LastVisualChange: 3200
[2018-09-17 11:48:16] INFO: Testing url https://en.wikipedia.org/wiki/Sweden iteration 4
[2018-09-17 11:48:44] INFO: Waiting on har-export-trigger to collect the HAR
[2018-09-17 11:48:54] INFO: BackEndTime: 363 DomInteractiveTime: 1743 DomContentLoadedTime: 1833 FirstPaint: 1503 PageLoadTime: 3181
[2018-09-17 11:48:54] INFO: VisualMetrics FirstVisualChange: 1533 SpeedIndex: 1620 PerceptualSpeedIndex: 2478 LastVisualChange: 3166
[2018-09-17 11:48:54] INFO: Testing url https://en.wikipedia.org/wiki/Sweden iteration 5
[2018-09-17 11:49:21] INFO: Waiting on har-export-trigger to collect the HAR
[2018-09-17 11:49:31] INFO: BackEndTime: 374 DomInteractiveTime: 1801 DomContentLoadedTime: 1893 FirstPaint: 1601 PageLoadTime: 3263
[2018-09-17 11:49:31] INFO: VisualMetrics FirstVisualChange: 1666 SpeedIndex: 1744 PerceptualSpeedIndex: 2496 LastVisualChange: 3266
[2018-09-17 11:49:32] INFO: Testing url https://en.wikipedia.org/wiki/Sweden iteration 6
[2018-09-17 11:49:59] INFO: Waiting on har-export-trigger to collect the HAR
[2018-09-17 11:50:09] INFO: BackEndTime: 368 DomInteractiveTime: 1747 DomContentLoadedTime: 1835 FirstPaint: 1535 PageLoadTime: 3165
[2018-09-17 11:50:09] INFO: VisualMetrics FirstVisualChange: 1566 SpeedIndex: 1666 PerceptualSpeedIndex: 2494 LastVisualChange: 3233
[2018-09-17 11:50:10] INFO: Testing url https://en.wikipedia.org/wiki/Sweden iteration 7
[2018-09-17 11:50:37] INFO: Waiting on har-export-trigger to collect the HAR
[2018-09-17 11:50:47] INFO: BackEndTime: 421 DomInteractiveTime: 1783 DomContentLoadedTime: 1894 FirstPaint: 1586 PageLoadTime: 2971
[2018-09-17 11:50:47] INFO: VisualMetrics FirstVisualChange: 1633 SpeedIndex: 1705 PerceptualSpeedIndex: 2376 LastVisualChange: 3333
[2018-09-17 11:50:47] INFO: Testing url https://en.wikipedia.org/wiki/Sweden iteration 8
[2018-09-17 11:51:15] INFO: Waiting on har-export-trigger to collect the HAR
[2018-09-17 11:51:25] INFO: BackEndTime: 373 DomInteractiveTime: 1688 DomContentLoadedTime: 1791 FirstPaint: 1451 PageLoadTime: 3270
[2018-09-17 11:51:25] INFO: VisualMetrics FirstVisualChange: 1500 SpeedIndex: 1592 PerceptualSpeedIndex: 2486 LastVisualChange: 3300
[2018-09-17 11:51:25] INFO: Testing url https://en.wikipedia.org/wiki/Sweden iteration 9
[2018-09-17 11:51:52] INFO: Waiting on har-export-trigger to collect the HAR
[2018-09-17 11:52:02] INFO: BackEndTime: 373 DomInteractiveTime: 1773 DomContentLoadedTime: 1840 FirstPaint: 1552 PageLoadTime: 3225
[2018-09-17 11:52:02] INFO: VisualMetrics FirstVisualChange: 1600 SpeedIndex: 1682 PerceptualSpeedIndex: 2489 LastVisualChange: 3266
[2018-09-17 11:52:02] INFO: Testing url https://en.wikipedia.org/wiki/Sweden iteration 10
[2018-09-17 11:52:30] INFO: Waiting on har-export-trigger to collect the HAR
[2018-09-17 11:52:40] INFO: BackEndTime: 372 DomInteractiveTime: 1761 DomContentLoadedTime: 1850 FirstPaint: 1501 PageLoadTime: 3129
[2018-09-17 11:52:40] INFO: VisualMetrics FirstVisualChange: 1533 SpeedIndex: 1619 PerceptualSpeedIndex: 2460 LastVisualChange: 3166
[2018-09-17 11:52:40] INFO: Testing url https://en.wikipedia.org/wiki/Sweden iteration 11
[2018-09-17 11:53:08] INFO: Waiting on har-export-trigger to collect the HAR
[2018-09-17 11:53:18] INFO: BackEndTime: 417 DomInteractiveTime: 1850 DomContentLoadedTime: 1976 FirstPaint: 1653 PageLoadTime: 3593
[2018-09-17 11:53:18] INFO: VisualMetrics FirstVisualChange: 1666 SpeedIndex: 1764 PerceptualSpeedIndex: 2700 LastVisualChange: 3566
[2018-09-17 11:53:19] INFO: 148 requests, 2166.44 kb, backEndTime: 380ms (±5.68ms), firstPaint: 1.54s (±19.27ms), firstVisualChange: 1.58s (±20.24ms), DOMContentLoaded: 1.86s (±14.56ms), Load: 3.22s (±42.80ms), speedIndex: 1670 (±19.38), visualComplete85: 1.58s (±20.24ms), lastVisualChange: 3.27s (±31.72ms), rumSpeedIndex: 958 (±17.49) (11 runs)

Yeah that was a lot of metrics :) Let me summarize: Different URLs/content behave differently (we know that before but we need to always think about that. It seems like going in a larger instance gives us more stable metrics for Firefox, maybe a c4.xlarge is the way to go, We could then maybe run only Firefox on that instance. But let us collect some more metrics first.

Peter added a comment.Sep 19 2018, 7:40 AM

I've been collecting metrics now for a while and concentrated on two URLs: https://en.wikipedia.org/wiki/Barack_Obama and https://en.wikipedia.org/wiki/Sweden

The timespan tested https://grafana.wikimedia.org/dashboard/db/webpagereplay-drilldown?orgId=1&from=1537272487594&to=1537340808984&var-wiki=enwiki&var-device=desktop&var-browser=chrome&var-latency=100&var-page=Barack_Obama

We run 11 runs each and take the median first visual change and median Speed Index. The runs on the c4.large is our normal tests. One thing to note is that it seems that instance has become more unstable the last month(s) see T204300.

I've been calculating the diffs like this: (largest median - smallest median) / smallest median

Instance typeBrowser Page First Visual Change diff  SpeedIndex
c4.largeChrome Sweden10 %6.8 %
c4.largeFirefox Sweden13 %11 %
c4.largeChrome Obama6.7 %5.6 %
c4.largeFirefox Obama15.3 %12.6 %
c4.xlargeChrome Sweden3.8 %6.2 %
c4.xlargeFirefox Sweden6.5 %5.9 %
c4.xlargeChrome Obama16.6 % (!)13.9 %
c4.xlargeFirefox Obama8.6 %5.3 %
c5.largeChrome Sweden7.6 %8.8 %
c5.largeFirefox Sweden42.6 % (!)37.7 %
c5.largeChrome Obama4.0 %3.8 %
c5.largeFirefox Obama8.4 %6.6 %

And then to compare also with the mobile version (for mobile we only do 5 runs but with higher frame rate 60 fps, on desktop we do 30 fps:

Instance typeBrowser Page First Visual Change diff  SpeedIndex
c4.largeChrome Sweden2.3 %1.5 %
c4.largeChrome Obama2.9 %0.5 %
Peter added a comment.Sep 27 2018, 7:20 AM

I've deployed Firefox tests permanently on a c4.xlarge T205246 and it looks good so far.

Peter closed this task as Resolved.Sep 28 2018, 12:19 PM

This is better now, let us close it.