Page MenuHomePhabricator

Element Timing for Images origin trial
Closed, ResolvedPublic

Description

Running until September 4, Chrome 73 to 76: https://github.com/npm1/Element-Timing

Details

Related Gerrit Patches:
mediawiki/vagrant : masterRenew origin trial tokens
operations/mediawiki-config : masterRenew origin trial tokens for ruwiki
operations/mediawiki-config : masterEnable Priority Hints and Element Timing on eswiki
operations/mediawiki-config : masterElement Timing for Images and Layout Stability on ruwiki
mediawiki/vagrant : masterUpdate origin trials tokens
mediawiki/extensions/NavigationTiming : masterAdd Element Timing support
mediawiki/core : masterAdd support for Element Timing API

Event Timeline

Gilles removed Gilles as the assignee of this task.Feb 20 2019, 8:54 AM
Gilles triaged this task as Medium priority.
Gilles created this task.
Peter added a subscriber: Peter.Feb 20 2019, 9:26 PM
Gilles claimed this task.Feb 25 2019, 8:52 PM

Filed a github issue about the spec: https://github.com/WICG/element-timing/issues/1

Emailed Nicolas asking if performance timeline support that's available in Chrome 75 could be backported to 73.

Change 497468 had a related patch set uploaded (by Gilles; owner: Gilles):
[mediawiki/core@master] Add support for Element Timing API

https://gerrit.wikimedia.org/r/497468

Change 497470 had a related patch set uploaded (by Gilles; owner: Gilles):
[mediawiki/extensions/NavigationTiming@master] Add Element Timing support

https://gerrit.wikimedia.org/r/497470

Change 497468 merged by jenkins-bot:
[mediawiki/core@master] Add support for Element Timing API

https://gerrit.wikimedia.org/r/497468

Change 497470 merged by jenkins-bot:
[mediawiki/extensions/NavigationTiming@master] Add Element Timing support

https://gerrit.wikimedia.org/r/497470

Change 499152 had a related patch set uploaded (by Gilles; owner: Gilles):
[operations/mediawiki-config@master] Element Timing for Images and Layout Stability on ruwiki

https://gerrit.wikimedia.org/r/499152

Change 499710 had a related patch set uploaded (by Gilles; owner: Gilles):
[mediawiki/vagrant@master] Update origin trials tokens

https://gerrit.wikimedia.org/r/499710

Change 499710 merged by jenkins-bot:
[mediawiki/vagrant@master] Update origin trials tokens

https://gerrit.wikimedia.org/r/499710

Change 499152 merged by jenkins-bot:
[operations/mediawiki-config@master] Element Timing for Images and Layout Stability on ruwiki

https://gerrit.wikimedia.org/r/499152

Mentioned in SAL (#wikimedia-operations) [2019-03-29T07:01:07Z] <gilles@deploy1001> Synchronized wmf-config/InitialiseSettings.php: T216598 T216594 Element Timing for Images and Layout Stability on ruwiki (duration: 00m 51s)

The first events have made it to the EL schema according to Grafana. Not seeing them in Hive yet.

Gilles added a comment.Apr 2 2019, 8:56 PM

Looking at the initial data... Focusing on thumbnails marked as "top article image", I see that the difference between the element startTime and firstPaint is usual small, but can sometimes be large. It's identical 75% of the time.

Some of the most extreme example (eg. 11 seconds difference) are for images that weren't initially in the viewport. Which suggests some lazy-loading done by the browser, perhaps? This test is desktop-only.

The fact that it differs 25% of the time makes me hopeful that this could be a good feature.

Gilles added a comment.Apr 2 2019, 9:16 PM

At this point we have 419 non-neutral survey responses with a record of the thumbnail-high event timing. Looking only at this subset of the data, I find that the Pearson correlation coefficient is slightly better for the event timing than for firstPaint (0.134 vs 0.120). Meanwhile, still on that subset of data, loadEventEnd gets a terrible coefficient (0.03).

Could it be that thumbnail-high is indicative of the likelihood that there will be more thumbnails outside of the initial viewport that will make loadEventEnd more disconnected to the perception than on an article that might lack images altogether? In last year's study we didn't really consider different classes of articles based on the presence or absence of thumbnails on the page.

Further investigation is required, but this initial finding suggests that when a top image is present in the initial viewport, the element timing of that image is a better visual metric than firstPaint.

Gilles added a comment.Apr 3 2019, 8:32 PM

@Peter I think it would be worth turning your initial tests in WPT into a blog post of its own. Basically about validating the accuracy of Element Timing using synthetic testing. What do you think?

Gilles added a comment.Apr 3 2019, 9:09 PM

With an extra day of data, the correlation coefficients are already changing significantly:

SELECT CORR(nt.event.loadeventend, CASE qr.event.surveyresponsevalue WHEN 'ext-quicksurveys-example-internal-survey-answer-positive' THEN -1 ELSE 1 END) AS loadeventend_correlation, CORR(et.event.startTime, CASE qr.event.surveyresponsevalue WHEN 'ext-quicksurveys-example-internal-survey-answer-positive' THEN -1 ELSE 1 END) AS elementtiming_correlation, CORR(nt.event.firstPaint, CASE qr.event.surveyresponsevalue WHEN 'ext-quicksurveys-example-internal-survey-answer-positive' THEN -1 ELSE 1 END) AS firstpaint_correlation, COUNT(*) AS count FROM event.elementtiming et JOIN event.navigationtiming nt ON et.event.pageviewToken = nt.event.pageviewToken JOIN event.quicksurveysresponses qr ON et.event.pageviewToken = qr.event.pageviewToken WHERE nt.year = 2019 AND et.year = 2019 AND et.event.name = 'thumbnail-high' AND et.event.x > 0 AND qr.year = 2019 AND qr.event.surveyresponsevalue IN ('ext-quicksurveys-example-internal-survey-answer-positive', 'ext-quicksurveys-example-internal-survey-answer-negative');
loadeventend_correlationelementtiming_correlationfirstpaint_correlationcount
0.0424752711142605750.198741867097204230.19478855853319674579
Peter added a comment.Apr 9 2019, 8:12 AM

Let do the blog post tomorrow.

Gilles added a comment.Apr 9 2019, 8:14 AM

Update:

SELECT et.event.name AS name, CORR(nt.event.loadeventend, CASE qr.event.surveyresponsevalue WHEN 'ext-quicksurveys-example-internal-survey-answer-positive' THEN -1 ELSE 1 END) AS loadeventend_correlation, CORR(et.event.startTime, CASE qr.event.surveyresponsevalue WHEN 'ext-quicksurveys-example-internal-survey-answer-positive' THEN -1 ELSE 1 END) AS elementtiming_correlation, CORR(nt.event.firstPaint, CASE qr.event.surveyresponsevalue WHEN 'ext-quicksurveys-example-internal-survey-answer-positive' THEN -1 ELSE 1 END) AS firstpaint_correlation, COUNT(*) AS count FROM event.elementtiming et JOIN event.navigationtiming nt ON et.event.pageviewToken = nt.event.pageviewToken JOIN event.quicksurveysresponses qr ON et.event.pageviewToken = qr.event.pageviewToken WHERE nt.year = 2019 AND et.year = 2019 AND nt.month = 4 AND et.month = 4 AND et.day > 3 AND nt.day > 3 AND et.event.name IN ('thumbnail-high', 'thumbnail-top') AND et.event.x > 0 AND qr.year = 2019 AND qr.event.surveyresponsevalue IN ('ext-quicksurveys-example-internal-survey-answer-positive', 'ext-quicksurveys-example-internal-survey-answer-negative') GROUP BY et.event.name;
nameloadeventend_correlationelementtiming_correlationfirstpaint_correlationcount
thumbnail-high0.115554647753079250.120313073816922480.118450429224469671168
thumbnail-top0.066783228950514330.090122325568207570.10593221800782102970

Probably still not enough data, the correlation factor varies a lot as we get more samples.

Update:

nameloadeventend_correlationelementtiming_correlationfirstpaint_correlationcount
thumbnail-high0.14928254718682540.14608746153146420.11845042922446973322
thumbnail-top0.153047166582761070.158344370193867430.106026122383987873107

Change 505237 had a related patch set uploaded (by Gilles; owner: Gilles):
[operations/mediawiki-config@master] Enable Priority Hints and Element Timing on eswiki

https://gerrit.wikimedia.org/r/505237

Change 505237 merged by jenkins-bot:
[operations/mediawiki-config@master] Enable Priority Hints and Element Timing on eswiki

https://gerrit.wikimedia.org/r/505237

Mentioned in SAL (#wikimedia-operations) [2019-04-19T12:59:38Z] <gilles@deploy1001> Synchronized wmf-config/InitialiseSettings.php: T216499 T216598 Enable Priority Hints and Element Timing on eswiki (duration: 00m 56s)

Change 507545 had a related patch set uploaded (by Gilles; owner: Gilles):
[operations/mediawiki-config@master] Renew origin trial tokens for ruwiki

https://gerrit.wikimedia.org/r/507545

Change 507545 merged by jenkins-bot:
[operations/mediawiki-config@master] Renew origin trial tokens for ruwiki

https://gerrit.wikimedia.org/r/507545

Mentioned in SAL (#wikimedia-operations) [2019-05-01T11:22:05Z] <gilles@deploy1001> Synchronized wmf-config/InitialiseSettings.php: T216499 T216598 T216594 Renew origin trial tokens for ruwiki (duration: 01m 14s)

Mentioned in SAL (#wikimedia-operations) [2019-05-01T11:27:29Z] <gilles> T216499 Y216594 T216598 mwscript purgeList.php ruwiki --all --verbose

Change 509630 had a related patch set uploaded (by Gilles; owner: Gilles):
[mediawiki/vagrant@master] Renew origin trial tokens

https://gerrit.wikimedia.org/r/509630

Change 509630 merged by jenkins-bot:
[mediawiki/vagrant@master] Renew origin trial tokens

https://gerrit.wikimedia.org/r/509630

Gilles raised the priority of this task from Medium to High.May 27 2019, 5:04 AM
Gilles closed this task as Resolved.Jun 18 2019, 9:16 AM

I've done all the analysis that I could think of for this one. It works, it's a pretty good metric, but for us it's not as good as FP/FCP and domInteractive in terms of correlation to user perception.