Change 271144 had a related patch set uploaded (by Jforrester):
Speed trials: Add mobile and desktop versions with OOjs UI core loaded

https://gerrit.wikimedia.org/r/271144

Change 271144 merged by jenkins-bot:
Speed trials: Add mobile and desktop versions with OOjs UI core loaded

https://gerrit.wikimedia.org/r/271144

Hey Peter,

We now have four pages to compare:

• The desktop read page,
  • … as now, and
  • with OOUI loaded; and
• The mobile read page
  • … as now, and
  • with OOUI loaded.

Could you give us a steer on the performance implications, using Web Page Speed Testing? Not sure exactly what I'm asking for (cc @ori). :-)

Ok, I did a first check. I've tested doing 9 runs each, only looking at the first view, taking both the fastest SpeedIndex run and fastest start rendering (remember SpeedIndex is a way of calculating when the content within the viewport is ready (lower number is better) and start render is when the first time something happens on the screen).

All the tests are linked in the page if you want to look at the result yourself. To change how WebPageTest choose which test that are the median and what metric to base that on:
?medianRun=fastest&medianMetric=render
?medianRun=fastest&medianMetric=SpeedIndex
?medianRun=median&medianMetric=render
?medianRun=median&medianMetric=SpeedIndex

Let me do a summary tomorrow.

In T127125#2036890, @Peter wrote:

Ok, I did a first check. I've tested doing 9 runs each, only looking at the first view, taking both the fastest SpeedIndex run and fastest start rendering (remember SpeedIndex is a way of calculating when the content within the viewport is ready (lower number is better) and start render is when the first time something happens on the screen).

Page	How	Start render (fastest)	Start render (median)	SpeedIndex (fastest)	SpeedIndex (median)
now desktop	Chrome Cable	1.175 s	2.685 s	2230	2752
OOUI loaded desktop	Chrome Cable	1.772 s	2.986 s	2091	3200
now desktop	Firefox Cable	1.729 s	4.281 s	1700	4490
OOUI loaded desktop	Firefox Cable	1.397 s	2.281 s	1409	2309
now mobile	Chrome (emulated Iphone 6) 2G	13.985 s	14.275 s	14818	15157
OOUI loaded mobile	Chrome (emulated Iphone 6) 2G	12.986 s	15.474 s	14338	16478
now mobile	Moto G 3G	4.887 s	5.172 s	5562	5780
OOUI loaded mobile	Moto G 3G	4.900 s	5.176 s	5595	5717

All the tests are linked in the page if you want to look at the result yourself. To change how WebPageTest choose which test that are the median and what metric to base that on:
?medianRun=fastest&medianMetric=render
?medianRun=fastest&medianMetric=SpeedIndex
?medianRun=median&medianMetric=render
?medianRun=median&medianMetric=SpeedIndex

Let me do a summary tomorrow.

Could you add an 'X-Wikimedia-Debug: 1' header to the requests? That way, we'll know that any difference is not due to something silly like the OOjs UI variant not being in Varnish. Alternately, a cache-busting query param should do the trick too.

yep let me do that. will run on a private instance today so we can make more than 9 runs to see what happens with that median numbers.

Status update: I've been testing WPT Bulk tester all day but when I check the metrics for mobile it doesn't seems realistic on 2G connections (too fast). Testing now on European instance to see what happens. I'll share the script and result in a couple of hours.

Finally I got it running the way I want. I'll write some docs at WikiTech later today about what I needed todo to get it to work the way I wanted.

This is how I've done the test: Private instance (Ireland), 31 runs for each test and take the median run using SpeedIndex. I've tested desktop original & OOUI in Chrome & Firefox throttling as cable and mobile original OOUI using Chrome emulating mobile and running 3G and 2G. All requests has the 'X-Wikimedia-Debug: 1' header.

@ori & @Krinkle have I missed anything?

https://wikitech.wikimedia.org/wiki/WPTBulkTester

Ok, did a re-run again without the debug header. Again taking the median of 31 runs for SpeedIndex.

So…

These numbers are very surprising to me.

On desktop, Load time is significantly down, despite the number of bytes shipped being higher; Start render is improved but only on Chrome, and Visually complete and SpeedIndex are both improved too.

On mobile, Load time is significantly up (far more than I expected), but Start render and SpeedIndex are essentially flat, and Visually complete is much better(?) on 2G.

I'm lost. :-)

In T127125#2042281, @Peter wrote:

Ok, did a re-run again without the debug header. Again taking the median of 31 runs for SpeedIndex.

Taking the best run rather than the median run would be better, because the best run is the one that is least influenced by environmental noise.

You can take the fastest by adding ?medianRun=fastest&medianMetric=SpeedIndex to the run. So we sorted by the median of SpeedIndex so the other metrics are not the median, making difference in % isn't right. When we take the median for SpeedIndex, thats the only metric we should compare, sorry for including the rest.

Lets make list of the fastest start render & Speed Index. However I think using fastest is not right, some metrics are reported wrong checkout:
http://wpt.wmftest.org/result/160218_RX_BS/15/details/

Start rendering is at 0.125 s but checking the screenshots for the run it really happens at 3.7 s.

Did a new run since it so easy using WPT Bulk tester. Running 51 times on each URL, taking the median start rendering time, Making so many runs _should_ filter out the noise or?

One other thing that looks wrong in some tests is that the first byte is extremely fast, hitting 0.051s sometimes.

51 runs focusing on median SpeedIndex looks better

Lets tests it even more.

I've created an issue for WebPageTest https://github.com/WPO-Foundation/webpagetest/issues/566

Seems it happens only for Chrome & cable, but it happens so often that our values gets screwed :(

Ok, got it confirmed that it's not working correctly in Chrome/WPT:

"The issue is coming from Chrome's dev tools reporting of the timings and it looks like it is missing the start of the request (including socket and DNS) for some reason. The real fix is for me to move away from using dev tools for the timings for HTTPS requests and I'm really close to having that working (though then it will be like Firefox and SPDY will be an issue)."

https://github.com/WPO-Foundation/webpagetest/issues/566