Page MenuHomePhabricator

Collect HAR information for Firefox increase variance in Visual Metrics
Closed, ResolvedPublic

Description

I've been testing out https://github.com/devtools-html/har-export-trigger it seems it add extra noice to our metrics, I'll create an upstream bug when I have more metrics:
https://sv.wikipedia.org/wiki/Stockholm

With HAR turned on (11 runs):

FirstVisualChange:{"median":2266,"mean":2230,"mdev":80.3394,"min":1800,"p10":1866,"p90":2566,"p99":2700,"max":2700}

Without:

FirstVisualChange:{"median":1733,"mean":1633,"mdev":43.7079,"min":1300,"p10":1466,"p90":1733,"p99":1733,"max":1733},

And comparing with Chrome:

FirstVisualChange:{"median":1000,"mean":1021,"mdev":24.3429,"min":900,"p10":933,"p90":1133,"p99":1133,"max":1133}

So it makes Firefox slower (that is ok) but also adds extra variance (we don't want that).

Event Timeline

Peter triaged this task as Medium priority.Mar 13 2018, 7:46 AM

Adding some results. Running Firefox with 11 runs and the HAR plugin turned on:

Screen Shot 2018-03-14 at 9.02.50 AM.png (934×1 px, 136 KB)

Compared with Chrome at the same time running 5 runs:

Screen Shot 2018-03-14 at 9.02.57 AM.png (918×1 px, 137 KB)

The only way to fix this is to run on a larger instance. I tried it out on a larger machine on DO (one of the CPU Optimized Droplets) and then the metrics is more stable for Firefox. But DO has other problems over time.

Screen Shot 2018-08-29 at 2.04.01 PM.png (1×2 px, 225 KB)