Page MenuHomePhabricator

Investigate high variance in metrics for https://en.wikipedia.org/wiki/Sweden
Closed, ResolvedPublic

Description

We have one URL that stands out in our tests (both on WebPageTest and Browsertime/WebPageReplay) as more unstable in metrics than the rest of the URLs. This is most of a problem for the synthetic testing we are doing with Firefox.

Here we test the URL five runs with WebPageTest and the start render time varies between 3100 - 3900ms
http://wpt.wmftest.org/result/180328_WA_PB/
Running for example the Barack Obama page is between 3900-4000ms (http://wpt.wmftest.org/result/180328_V0_R6/).

We have the exact same situation with Browsertime/WebPageReplay even after we did the "trick" by enabling the MOZ log that makes FF slower and for other URLs makes the page more stable. With the log turned on the first visual change goes between 3166 - 3600 (even though we use WebPageReplay).

Checking the HAR (I'll enabling HARs again and add it to the issue) the waterfall looks pretty much the same, so we need to dig deeper. One thing I've seen with Firefox is that DOMInteractive seems to follow the same pattern (low DOMInteractive, low first visual change) but that is not true for Sweden.

Event Timeline

One main difference is that the Sweden article is considerably more image-heavy than the Barack Obama one. 128 images vs 80. And in terms of wikitext thumbnails, 64 vs 42. In terms of weight, 1587kb of images for Sweden and 800kb of images for Barack Obama.

Interestingly, the HTML weighed is much lighter for Sweden (171kb vs 242kb).

Decoding images can be quite processor intensive. If you look at the CPU graph for a Sweden and a Barack Obama load, you'll see that the CPU usage correlates quite well to all the images being loaded, with the CPU stopping to be maxed out only after all the images on the page have been loaded. For the Sweden article, this goes beyond start render.

IMHO this is probably an issue of critical mass of amount of images, where there are so many images to load that the loading and rendering of those images will overlap for a much longer period of the critical path lifecycle, introducing more randomness in the order of things happening, because the CPU is maxed out for longer while taking care of critical stuff.

Corrected my comment since I found no evidence that browsers leverage GPUs to decode JPG or PNG. They probably don't.

Comparing in terms of pixel surface:

var surface = 0; jQuery.each($('img'), function(idx, e) { surface += e.naturalWidth * e.naturalHeight; }); surface;

Sweden: 7636524
Barack Obama: 5640491

From that point of view Sweden "only" has 35% more image pixels to render.

Let me find a couple more image-rich articles to verify if this pattern holds.

Image-rich long articles:

https://en.wikipedia.org/wiki/Roman_Empire HTML 195kb IMG 2051kb
https://www.webpagetest.org/result/180405_MW_771fa15810bf323052d463ccf96026b4/
Start render 3900 - 4800 => 900ms variance

https://en.wikipedia.org/wiki/Bo_(parsha) HTML 208kb IMG 1397kb
https://www.webpagetest.org/result/180405_6Q_c88bec5cf3aeca64e191d2b6ed128f0d/
Start render 2500 - 3100 => 600ms variance

https://en.wikipedia.org/wiki/Fukushima_Daiichi_nuclear_disaster HTML 218kb IMG 561kb
https://www.webpagetest.org/result/180405_3C_b9cd45fc0f266f2b5d9a473eff4dddb1/
Start render 2100 - 2300 => 200ms variance

https://en.wikipedia.org/wiki/Native_Americans_in_the_United_States HTML 167kb IMG 1075kb
https://www.webpagetest.org/result/180405_12_00ec8a523b0033d5902243962699ae85/
Start render 2700-3300 => 600ms variance

Long articles with a small amount images:

https://en.wikipedia.org/wiki/List_of_Australian_treaties HTML 145kb IMG 38kb
https://www.webpagetest.org/result/180405_61_4f18b2d21b20f49390b5e14effe33017/
Start render 4100 - 4300 => 200ms variance

https://en.wikipedia.org/wiki/Food_Paradise HTML 225kb IMG 29kb
https://www.webpagetest.org/result/180405_MJ_5f7634a32d1498955b5b2063855b2b66/
Start render 1200 - 1300 => 100ms variance

https://en.wikipedia.org/wiki/School_shooting HTML 252kb IMG 86kb
https://www.webpagetest.org/result/180405_21_fcbc348e39f337fc5db5010384e382aa/
Start render 2700 - 3200 => 500ms variance

https://en.wikipedia.org/wiki/Mobile_country_code HTML 185kb IMG 38kb
https://www.webpagetest.org/result/180405_4H_b70bbc0dfe2cb2f65887b26cfa0822aa/
Start render 1800 - 2100 => 300ms variance

Seems like long articles can be affected, but less so? I'm not sure what's different between the first two that see little variance and the last two, though.

Small articles:

https://en.wikipedia.org/wiki/Manipulative_(mathematics_education) HTML 11kb IMG 85kb
https://www.webpagetest.org/result/180405_7F_809a3be9a6ec0e233e25ab6c006c4b7b/
Start render 600 - 800 => 200ms variance

https://en.wikipedia.org/wiki/Eight-Foot_High_Speed_Tunnel_(Hampton,_Virginia) HTML 15kb IMG 117kb
https://www.webpagetest.org/result/180405_1X_098bed65c26dc528218d0c024d74b049/
Start render 600 - 800 => 200ms variance

https://en.wikipedia.org/wiki/Riboflavin_synthase HTML 16kb IMG 120kb
https://www.webpagetest.org/result/180405_D6_0717f5c8602a7d349cc48d4d52a95117/
Start render 600 - 800 => 200ms variance

https://en.wikipedia.org/wiki/Allein_zu_dir,_Herr_Jesu_Christ,_BWV_33 HTML 32kb IMG 150kb
https://www.webpagetest.org/result/180405_EM_f2e99929f235180d333dfd0cc82b2554/
Start render 700 - 800 => 100ms variance

The patterns seems clear, if you have more than 1MB of images on the page, you'll get very high variance.

@Gilles great! Do you wanna contact Mozilla about it? We don't have see the same variance for Chrome.