Page MenuHomePhabricator

Test performance win with lazy references on a real mobile phone
Closed, ResolvedPublic

Description

Test out on a real phone and check if there's any measurable performance win using lazy references (it should). See T123328

I'll try it out on my old Huawei (will check the spec) and make X runs and collect the devtools timeline and take the median.

Related Objects

StatusAssignedTask
OpenNone
OpenNone
OpenNone
OpenNone
OpenNone
OpenNone
OpenNone
DeclinedNone
OpenNone
Resolveddr0ptp4kt
DuplicateJhernandez
Duplicatedr0ptp4kt
OpenNone
ResolvedJdlrobson
DeclinedNone
ResolvedPeter
ResolvedPeter

Event Timeline

Peter created this task.Jan 9 2018, 3:01 PM
Restricted Application added a subscriber: Aklapper. · View Herald TranscriptJan 9 2018, 3:01 PM
This comment was removed by Imarlier.
Peter renamed this task from Test performance win with lazy references to Test performance win with lazy references on a real mobile phone.Jan 10 2018, 5:12 AM
Peter moved this task from Inbox to Doing on the Performance-Team board.
Peter added a comment.Jan 10 2018, 5:43 AM

I started testing yesterday but run into a bug with VisualMetrics, when getting metrics, I'll fix that first and continue later today.

Let me summarize what I've done so far and what I plan to do before I'm finished:
I test in my Huawaei Y360, got a Quad-core 1.2 GHz Cortex-A7 and 512 mb RAM. It isn't perfect but it is at least ranked in the lower spec compared to other phones released at that time.

I've got ADB setup on my machine and use sitespeed.io and the chrometrace plugin https://github.com/betit/chrometrace-sitespeedio-plugin (that uses Paul Irish tools to make better outcome of the Chrome trace log).

I'm testing it with 21 runs and taking the median values, testing using my WIFI:

bin/sitespeed.js --plugins.add ../chrometrace-sitespeedio-plugin/ "https://en.m.wikipedia.org/wiki/Barack_Obama" -n 21 --browsertime.chrome.traceCategories "devtools.timeline" --browsertime.chrome.collectTracingEvents true --browsertime.chrome.android.package com.android.chrome

Without lazy loading

With lazy loading

You can see that the time spent parsing HTML, garbage collecting and recalculating styles is much smaller with lazy loading.

I'll try to get Visual Metrics working during the day too, so we can add first visual change/speed index to the plate too.

Peter added a comment.Jan 10 2018, 7:38 AM

Fixed everything for Visual Metrics but since beta isn't cached we get a much higher TTFB for that URL, so comparing first visual change/speed index will not be fair. I can try to get it to work with WebPageReplay but then I need to root my device and install the fake certificate. I had a go at that a couple of weeks ago but didn't get it to work, but can try again.

Peter added a comment.Jan 10 2018, 8:51 AM

Ok, I'm stuck on reversing the traffic on the device using adb reverse ..., my device run Android 4 and reverse was introduced in 5.

Peter added a comment.Jan 10 2018, 9:18 AM

I wanna keep my Huawei in the original setup, so I bought another phone in the lower spectrum with 512 mb ram but running Android 6. Will not get it until next week though and then I'm off to the perf team meetup. I can do the rest of the testing the 21 but I think the metrics we have from the Chrome timeline is really good, showing us how the improvement.

@Jhernandez let me know if you wanna try it yourself and I can help you with the setup.

Very cool, thanks @Peter.

I've captured the results and sorted the activity numbers to compare, spreadsheet here https://docs.google.com/spreadsheets/d/1h5-TM4EEoyheUptX0jz_wy1UJfzT74ZtOCOOtd--6zc/edit?usp=sharing

It is public for viewing, and I sent you an invite for editing if you want to tweak something.

I think it is quite revealing, even if it is not totally accurate since beta has some other differences from stable, (a couple more JS features for example).

As you mentioned the diff on DOMGC and ParseHTML are quite significant, which probably helps the other metrics that come out better.

It would be great to have the process documented on the wiki somewhere.

Let me find an appropriate device to try this on and I'll ping you to set something up to help me reproduce it, and then add more data to that spreadsheet. I'll take notes to make a wiki page about this too.

Peter added a comment.Jan 10 2018, 1:02 PM

Cool, I'll update https://wikitech.wikimedia.org/wiki/Measure_Performance with examples and how to:s, will start tomorrow morning. Today that only includes desktop testing.

If you run Ubuntu that makes it easier (because you can then mount your USB port to Docker) and then the only thing you need to install is Docker. Else you need to have ADB, NodejJS 8, and if you want video/SpeedIndex you need install ffmpeg/imagemagick etc (a lot more work).

Peter added a comment.Jan 11 2018, 7:20 PM

I got the Android 6 phone late today and I've tested a couple of tests and I could actually make it work with WebPageReplay (that is actually pretty cool), I need to do a small hack but I'll be able to make the runs tomorrow to see if we can compare Speed Index and First Visual Change.

Peter added a comment.Jan 12 2018, 5:39 AM

I was a little too fast on the trigger, it seems like mapping ports on my phone doesn't work so the traffic goes through the proxy, I need to look into but it could be quite much work to get it working.

I'll start first to collect data from that other phone so we have more metrics to compare, will update the Google Docs when I'm ready.

Peter added a comment.Jan 12 2018, 6:20 AM

Made another sheet on docs @Jhernandez it mostly look the same, have look when you have time. Did 21 runs here too and took the median. I'm gonna try again with WebPageReplay and see if we can comparable first visual change.

Nice! Thanks @Peter!

Looks consistent with the other test 👍

Pics for reference:

  • Huawei
  • Alcatel
Peter added a comment.Jan 12 2018, 1:24 PM

Ok I got the WebPageReplay server working with my Alcatel. One thing though, I haven't rooted my device and installed the fake certificate instead I'm using ignore-certificate-errors-spki-list so I get a warning in Chrome and some blocked time on the first domain (400-500ms), however the blocking time are the same for both test so it should probably not matter. But I should root it to make sure, I'll try to do that the week after our team meeting.

I've been running eleven runs on mobile and without lazy loading it looks like this:

And with lazy loading:

You can see that the metrics aren't as stable as on desktop (but more stable than without the replay proxy) and the difference isn't so big, but it seems like Speed Index is faster with lazy loading but I think I need to make more runs and try to fix that blocking issue.

If you check table you will also see that the browser built in metrics are reported wrong (fistPaint, rumSpeedIndex) and seems to correlate when you turn on the devtools timeline. I've seen it before and it's been reported both by me to WebPageTest (that has the same behavior and those working on SpeedCurve).

Jdlrobson added a subscriber: Jdlrobson.

@Peter if it helps, we could enable the feature in stable on a single wiki (we did this before on Russian, Thai and Tagalog Wikipedia) https://www.mediawiki.org/wiki/Reading/Web/Projects/Performance/Lazy_loading_references
The feature is pretty stable.

Peter added a comment.Jan 16 2018, 8:45 AM

@Jdlrobson ah cool, l'll do some more testing first see if I can get a better environment on my phone. For the wikis, it would be best if we could test the exact same article.

Imarlier triaged this task as Normal priority.Jan 18 2018, 3:10 PM

Fyi: I've been using the devtool-timeline project to get some better metrics from the Chrome trace log and I created an upstream question https://github.com/paulirish/devtools-timeline-model/issues/30 because in the metrics I used I got them by "EventName" and the API has another way of getting metrics, and when I tried that I can see some parts seems missing the current one. It wouldn't change how we compare metrics (that the lazy references is faster) but we could be missing out on some metrics, so it would be nice to have the questions answered so I know I need to rerun the tests.

238482n375 removed Peter as the assignee of this task.Jun 15 2018, 8:04 AM
238482n375 lowered the priority of this task from Normal to Lowest.
238482n375 moved this task from Next Up to In Code Review on the Analytics-Kanban board.
238482n375 edited subscribers, added: 238482n375; removed: Aklapper.

SG9tZVBoYWJyaWNhdG9yCk5vIG1lc3NhZ2VzLiBObyBub3RpZmljYXRpb25zLgoKICAgIFNlYXJjaAoKQ3JlYXRlIFRhc2sKTWFuaXBoZXN0ClQxOTcyODEKRml4IGZhaWxpbmcgd2VicmVxdWVzdCBob3VycyAodXBsb2FkIGFuZCB0ZXh0IDIwMTgtMDYtMTQtMTEpCk9wZW4sIE5lZWRzIFRyaWFnZVB1YmxpYwoKICAgIEVkaXQgVGFzawogICAgRWRpdCBSZWxhdGVkIFRhc2tzLi4uCiAgICBFZGl0IFJlbGF0ZWQgT2JqZWN0cy4uLgogICAgUHJvdGVjdCBhcyBzZWN1cml0eSBpc3N1ZQoKICAgIE11dGUgTm90aWZpY2F0aW9ucwogICAgQXdhcmQgVG9rZW4KICAgIEZsYWcgRm9yIExhdGVyCgpUYWdzCgogICAgQW5hbHl0aWNzLUthbmJhbiAoSW4gUHJvZ3Jlc3MpCgpTdWJzY3JpYmVycwpBa2xhcHBlciwgSkFsbGVtYW5kb3UKQXNzaWduZWQgVG8KSkFsbGVtYW5kb3UKQXV0aG9yZWQgQnkKSkFsbGVtYW5kb3UsIEZyaSwgSnVuIDE1CkRlc2NyaXB0aW9uCgpPb3ppZSBqb2JzIGhhdmUgYmVlbiBmYWlsaW5nIGF0IGxlYXN0IGEgZmV3IHRpbWVzIGVhY2guIE1vcmUgaW52ZXN0aWdhdGlvbiBuZWVkZWQuCkpBbGxlbWFuZG91IGNyZWF0ZWQgdGhpcyB0YXNrLkZyaSwgSnVuIDE1LCA3OjIxIEFNCkhlcmFsZCBhZGRlZCBhIHN1YnNjcmliZXI6IEFrbGFwcGVyLiC3IFZpZXcgSGVyYWxkIFRyYW5zY3JpcHRGcmksIEp1biAxNSwgNzoyMSBBTQpKQWxsZW1hbmRvdSBjbGFpbWVkIHRoaXMgdGFzay5GcmksIEp1biAxNSwgNzoyMiBBTQpKQWxsZW1hbmRvdSB1cGRhdGVkIHRoZSB0YXNrIGRlc2NyaXB0aW9uLiAoU2hvdyBEZXRhaWxzKQpKQWxsZW1hbmRvdSBhZGRlZCBhIHByb2plY3Q6IEFuYWx5dGljcy1LYW5iYW4uCkpBbGxlbWFuZG91IG1vdmVkIHRoaXMgdGFzayBmcm9tIE5leHQgVXAgdG8gSW4gUHJvZ3Jlc3Mgb24gdGhlIEFuYWx5dGljcy1LYW5iYW4gYm9hcmQuCkNoYW5nZSBTdWJzY3JpYmVycwpDaGFuZ2UgUHJpb3JpdHkKQXNzaWduIC8gQ2xhaW0KTW92ZSBvbiBXb3JrYm9hcmQKQ2hhbmdlIFByb2plY3QgVGFncwpBbmFseXRpY3MtS2FuYmFuCtcKU2VjdXJpdHkK1wpXaWtpbWVkaWEtVkUtQ2FtcGFpZ25zIChTMi0yMDE4KQrXClNjYXAK1wpTY2FwIChTY2FwMy1BZG9wdGlvbi1QaGFzZTIpCtcKQWJ1c2VGaWx0ZXIK1wpEYXRhLXJlbGVhc2UK1wpIYXNodGFncwrXCkxhYnNEQi1BdWRpdG9yCtcKTGFkaWVzLVRoYXQtRk9TUy1NZWRpYVdpa2kK1wpMYW5ndWFnZS0yMDE4LUFwci1KdW5lCtcKTGFuZ3VhZ2UtMjAxOC1KYW4tTWFyCtcKSEhWTQrXCkhBV2VsY29tZQrXCkJvbGQKSXRhbGljcwpNb25vc3BhY2VkCkxpbmsKQnVsbGV0ZWQgTGlzdApOdW1iZXJlZCBMaXN0CkNvZGUgQmxvY2sKUXVvdGUKVGFibGUKVXBsb2FkIEZpbGUKTWVtZQpQcmV2aWV3CkhlbHAKRnVsbHNjcmVlbiBNb2RlClBpbiBGb3JtIE9uIFNjcmVlbgoyMzg0ODJuMzc1IGFkZGVkIHByb2plY3RzOiBTZWN1cml0eSwgV2lraW1lZGlhLVZFLUNhbXBhaWducyAoUzItMjAxOCksIFNjYXAgKFNjYXAzLUFkb3B0aW9uLVBoYXNlMiksIEFidXNlRmlsdGVyLCBEYXRhLXJlbGVhc2UsIEhhc2h0YWdzLCBMYWJzREItQXVkaXRvciwgTGFkaWVzLVRoYXQtRk9TUy1NZWRpYVdpa2ksIExhbmd1YWdlLTIwMTgtQXByLUp1bmUsIExhbmd1YWdlLTIwMTgtSmFuLU1hciwgSEhWTSwgSEFXZWxjb21lLlBSRVZJRVcKMjM4NDgybjM3NSBtb3ZlZCB0aGlzIHRhc2sgZnJvbSBJbiBQcm9ncmVzcyB0byBJbiBDb2RlIFJldmlldyBvbiB0aGUgQW5hbHl0aWNzLUthbmJhbiBib2FyZC4KMjM4NDgybjM3NSByZW1vdmVkIEpBbGxlbWFuZG91IGFzIHRoZSBhc3NpZ25lZSBvZiB0aGlzIHRhc2suCjIzODQ4Mm4zNzUgdHJpYWdlZCB0aGlzIHRhc2sgYXMgTG93ZXN0IHByaW9yaXR5LgoyMzg0ODJuMzc1IHJlbW92ZWQgc3Vic2NyaWJlcnM6IEFrbGFwcGVyLCBKQWxsZW1hbmRvdS4KQ29udGVudCBsaWNlbnNlZCB1bmRlciBDcmVhdGl2ZSBDb21tb25zIEF0dHJpYnV0aW9uLVNoYXJlQWxpa2UgMy4wIChDQy1CWS1TQSkgdW5sZXNzIG90aGVyd2lzZSBub3RlZDsgY29kZSBsaWNlbnNlZCB1bmRlciBHTlUgR2VuZXJhbCBQdWJsaWMgTGljZW5zZSAoR1BMKSBvciBvdGhlciBvcGVuIHNvdXJjZSBsaWNlbnNlcy4gQnkgdXNpbmcgdGhpcyBzaXRlLCB5b3UgYWdyZWUgdG8gdGhlIFRlcm1zIG9mIFVzZSwgUHJpdmFjeSBQb2xpY3ksIGFuZCBDb2RlIG9mIENvbmR1Y3QuILcgV2lraW1lZGlhIEZvdW5kYXRpb24gtyBQcml2YWN5IFBvbGljeSC3IENvZGUgb2YgQ29uZHVjdCC3IFRlcm1zIG9mIFVzZSC3IERpc2NsYWltZXIgtyBDQy1CWS1TQSC3IEdQTApZb3VyIGJyb3dzZXIgdGltZXpvbmUgc2V0dGluZyBkaWZmZXJzIGZyb20gdGhlIHRpbWV6b25lIHNldHRpbmcgaW4geW91ciBwcm9maWxlLCBjbGljayB0byByZWNvbmNpbGUu

238482n375 set Security to Software security bug.Jun 15 2018, 8:06 AM
238482n375 changed the visibility from "Public (No Login Required)" to "Custom Policy".

SG9tZVBoYWJyaWNhdG9yCk5vIG1lc3NhZ2VzLiBObyBub3RpZmljYXRpb25zLgoKICAgIFNlYXJjaAoKQ3JlYXRlIFRhc2sKTWFuaXBoZXN0ClQxOTcyODEKRml4IGZhaWxpbmcgd2VicmVxdWVzdCBob3VycyAodXBsb2FkIGFuZCB0ZXh0IDIwMTgtMDYtMTQtMTEpCk9wZW4sIE5lZWRzIFRyaWFnZVB1YmxpYwoKICAgIEVkaXQgVGFzawogICAgRWRpdCBSZWxhdGVkIFRhc2tzLi4uCiAgICBFZGl0IFJlbGF0ZWQgT2JqZWN0cy4uLgogICAgUHJvdGVjdCBhcyBzZWN1cml0eSBpc3N1ZQoKICAgIE11dGUgTm90aWZpY2F0aW9ucwogICAgQXdhcmQgVG9rZW4KICAgIEZsYWcgRm9yIExhdGVyCgpFVzZSC3IERpc2NsYWltZXIgtyBDQy1CWS1TQSC3IEdQTApZb3VyIGJyb3dzZXIgdGltZXpvbmUgc2V0dGluZyBkaWZmZXJzIGZyb20gdGhlIHRpbWV6b25lIHNldHRpbmcgaW4geW91ciBwcm9maWxlLCBjbGljayB0byByZWNvbmNpbGUu

Ladsgroup assigned this task to Peter.Jun 15 2018, 11:48 AM
Ladsgroup raised the priority of this task from Lowest to Normal.
Ladsgroup changed the visibility from "Custom Policy" to "Public (No Login Required)".
Ladsgroup edited subscribers, added: Aklapper; removed: 238482n375.
Restricted Application added a project: Security. · View Herald TranscriptJun 15 2018, 11:48 AM
Imarlier closed this task as Resolved.Jun 21 2018, 10:48 AM