URL without lazy loading: https://en.m.wikipedia.org/wiki/Barack_Obama

URL with lazy loading: https://en.m.wikipedia.org/wiki/Barack_Obama?mobileaction=beta

I started testing yesterday but run into a bug with VisualMetrics, when getting metrics, I'll fix that first and continue later today.

Let me summarize what I've done so far and what I plan to do before I'm finished:
I test in my Huawaei Y360, got a Quad-core 1.2 GHz Cortex-A7 and 512 mb RAM. It isn't perfect but it is at least ranked in the lower spec compared to other phones released at that time.

I've got ADB setup on my machine and use sitespeed.io and the chrometrace plugin https://github.com/betit/chrometrace-sitespeedio-plugin (that uses Paul Irish tools to make better outcome of the Chrome trace log).

I'm testing it with 21 runs and taking the median values, testing using my WIFI:

bin/sitespeed.js --plugins.add ../chrometrace-sitespeedio-plugin/ "https://en.m.wikipedia.org/wiki/Barack_Obama" -n 21 --browsertime.chrome.traceCategories "devtools.timeline" --browsertime.chrome.collectTracingEvents true --browsertime.chrome.android.package com.android.chrome

Without lazy loading

With lazy loading

You can see that the time spent parsing HTML, garbage collecting and recalculating styles is much smaller with lazy loading.

I'll try to get Visual Metrics working during the day too, so we can add first visual change/speed index to the plate too.

Fixed everything for Visual Metrics but since beta isn't cached we get a much higher TTFB for that URL, so comparing first visual change/speed index will not be fair. I can try to get it to work with WebPageReplay but then I need to root my device and install the fake certificate. I had a go at that a couple of weeks ago but didn't get it to work, but can try again.

Ok, I'm stuck on reversing the traffic on the device using adb reverse ..., my device run Android 4 and reverse was introduced in 5.

I wanna keep my Huawei in the original setup, so I bought another phone in the lower spectrum with 512 mb ram but running Android 6. Will not get it until next week though and then I'm off to the perf team meetup. I can do the rest of the testing the 21 but I think the metrics we have from the Chrome timeline is really good, showing us how the improvement.

@Jhernandez let me know if you wanna try it yourself and I can help you with the setup.

Very cool, thanks @Peter.

I've captured the results and sorted the activity numbers to compare, spreadsheet here https://docs.google.com/spreadsheets/d/1h5-TM4EEoyheUptX0jz_wy1UJfzT74ZtOCOOtd--6zc/edit?usp=sharing

It is public for viewing, and I sent you an invite for editing if you want to tweak something.

I think it is quite revealing, even if it is not totally accurate since beta has some other differences from stable, (a couple more JS features for example).

As you mentioned the diff on DOMGC and ParseHTML are quite significant, which probably helps the other metrics that come out better.

It would be great to have the process documented on the wiki somewhere.

Let me find an appropriate device to try this on and I'll ping you to set something up to help me reproduce it, and then add more data to that spreadsheet. I'll take notes to make a wiki page about this too.

Cool, I'll update https://wikitech.wikimedia.org/wiki/Measure_Performance with examples and how to:s, will start tomorrow morning. Today that only includes desktop testing.

If you run Ubuntu that makes it easier (because you can then mount your USB port to Docker) and then the only thing you need to install is Docker. Else you need to have ADB, NodejJS 8, and if you want video/SpeedIndex you need install ffmpeg/imagemagick etc (a lot more work).

I got the Android 6 phone late today and I've tested a couple of tests and I could actually make it work with WebPageReplay (that is actually pretty cool), I need to do a small hack but I'll be able to make the runs tomorrow to see if we can compare Speed Index and First Visual Change.

I was a little too fast on the trigger, it seems like mapping ports on my phone doesn't work so the traffic goes through the proxy, I need to look into but it could be quite much work to get it working.

I'll start first to collect data from that other phone so we have more metrics to compare, will update the Google Docs when I'm ready.

Made another sheet on docs @Jhernandez it mostly look the same, have look when you have time. Did 21 runs here too and took the median. I'm gonna try again with WebPageReplay and see if we can comparable first visual change.

Nice! Thanks @Peter!

Looks consistent with the other test 👍

Pics for reference:

• Huawei
  

• Alcatel
  

Ok I got the WebPageReplay server working with my Alcatel. One thing though, I haven't rooted my device and installed the fake certificate instead I'm using ignore-certificate-errors-spki-list so I get a warning in Chrome and some blocked time on the first domain (400-500ms), however the blocking time are the same for both test so it should probably not matter. But I should root it to make sure, I'll try to do that the week after our team meeting.

I've been running eleven runs on mobile and without lazy loading it looks like this:

And with lazy loading:

You can see that the metrics aren't as stable as on desktop (but more stable than without the replay proxy) and the difference isn't so big, but it seems like Speed Index is faster with lazy loading but I think I need to make more runs and try to fix that blocking issue.

If you check table you will also see that the browser built in metrics are reported wrong (fistPaint, rumSpeedIndex) and seems to correlate when you turn on the devtools timeline. I've seen it before and it's been reported both by me to WebPageTest (that has the same behavior and those working on SpeedCurve).

@Peter if it helps, we could enable the feature in stable on a single wiki (we did this before on Russian, Thai and Tagalog Wikipedia) https://www.mediawiki.org/wiki/Reading/Web/Projects/Performance/Lazy_loading_references
The feature is pretty stable.

@Jdlrobson ah cool, l'll do some more testing first see if I can get a better environment on my phone. For the wikis, it would be best if we could test the exact same article.

Fyi: I've been using the devtool-timeline project to get some better metrics from the Chrome trace log and I created an upstream question https://github.com/paulirish/devtools-timeline-model/issues/30 because in the metrics I used I got them by "EventName" and the API has another way of getting metrics, and when I tried that I can see some parts seems missing the current one. It wouldn't change how we compare metrics (that the lazy references is faster) but we could be missing out on some metrics, so it would be nice to have the questions answered so I know I need to rerun the tests.

SG9tZVBoYWJyaWNhdG9yCk5vIG1lc3NhZ2VzLiBObyBub3RpZmljYXRpb25zLgoKICAgIFNlYXJjaAoKQ3JlYXRlIFRhc2sKTWFuaXBoZXN0ClQxOTcyODEKRml4IGZhaWxpbmcgd2VicmVxdWVzdCBob3VycyAodXBsb2FkIGFuZCB0ZXh0IDIwMTgtMDYtMTQtMTEpCk9wZW4sIE5lZWRzIFRyaWFnZVB1YmxpYwoKICAgIEVkaXQgVGFzawogICAgRWRpdCBSZWxhdGVkIFRhc2tzLi4uCiAgICBFZGl0IFJlbGF0ZWQgT2JqZWN0cy4uLgogICAgUHJvdGVjdCBhcyBzZWN1cml0eSBpc3N1ZQoKICAgIE11dGUgTm90aWZpY2F0aW9ucwogICAgQXdhcmQgVG9rZW4KICAgIEZsYWcgRm9yIExhdGVyCgpFVzZSC3IERpc2NsYWltZXIgtyBDQy1CWS1TQSC3IEdQTApZb3VyIGJyb3dzZXIgdGltZXpvbmUgc2V0dGluZyBkaWZmZXJzIGZyb20gdGhlIHRpbWV6b25lIHNldHRpbmcgaW4geW91ciBwcm9maWxlLCBjbGljayB0byByZWNvbmNpbGUu