Page MenuHomePhabricator

Priority Hints origin trial
Closed, ResolvedPublic

Description

Priority Hints: https://github.com/WICG/priority-hints/blob/master/EXPLAINER.md

The origin trial is ongoing until May 29.

By targeting ruwiki, we could ensure that we stay below the usage limit (0.5% of all Chrome pageloads worldwide, which we might break if we launched to all wikis). And we would get performance perception survey results, which would inform us of whether or not users felt the difference.
I generated an origin trial token for ru.wikipedia.org:

Aq3kBZfnEiSKSGftuvXgxz83ex6pKdjY5/g9LeDm+YINUgfZ7PliX7AY6qm5e9q+MDlZ5PhIohflorFFqmdWLg8AAABqeyJvcmlnaW4iOiJodHRwczovL3J1Lndpa2lwZWRpYS5vcmc6NDQzIiwiZmVhdHVyZSI6IlByaW9yaXR5SGludHMiLCJleHBpcnkiOjE1NTQyMDY4OTMsImlzU3ViZG9tYWluIjp0cnVlfQ==

It is set to expire on April 2. To renew it, we need to provide feedback about the trial.

I imagine that since the mobile site is technically a different subdomain, we would need to generate a separate token for it.

Chrome 73 graduates to stable on March 12.

Details

Show related patches Customize query in gerrit

Event Timeline

There are a very large number of changes, so older changes are hidden. Show Older Changes

Mentioned in SAL (#wikimedia-operations) [2019-03-04T08:45:32Z] <gilles@deploy1001> Synchronized wmf-config/InitialiseSettings.php: T216499 Undo enabling Priority Hints origin trial on ruwiki (duration: 00m 49s)

Change 494175 had a related patch set uploaded (by Gilles; owner: Gilles):
[mediawiki/core@master] Fix typo in Priority Hints static variable name

https://gerrit.wikimedia.org/r/494175

Change 494175 merged by jenkins-bot:
[mediawiki/core@master] Fix typo in Priority Hints static variable name

https://gerrit.wikimedia.org/r/494175

Change 494921 had a related patch set uploaded (by Gilles; owner: Gilles):
[operations/mediawiki-config@master] Enable Priority Hints origin trial on ruwiki

https://gerrit.wikimedia.org/r/494921

Looks like @ReleaseTaggerBot is wrong, the typo fix patch did make it to the 1.33.0-wmf.20 branch.

Change 494921 merged by jenkins-bot:
[operations/mediawiki-config@master] Enable Priority Hints origin trial on ruwiki

https://gerrit.wikimedia.org/r/494921

Mentioned in SAL (#wikimedia-operations) [2019-03-07T19:33:20Z] <gilles@deploy1001> Synchronized wmf-config/InitialiseSettings.php: T216499 Enable Priority Hints origin trial on ruwiki (duration: 00m 48s)

Change 495668 had a related patch set uploaded (by Phedenskog; owner: Phedenskog):
[performance/WebPageTest@master] Add tests for WebPageTest for Russian wiki to support Priority Hints.

https://gerrit.wikimedia.org/r/495668

Change 495668 merged by jenkins-bot:
[performance/WebPageTest@master] Add tests for WebPageTest for Russian wiki to support Priority Hints.

https://gerrit.wikimedia.org/r/495668

The ramp up's pace is increasing, we're now past the point of Chrome 73 being more prevalent than 72 as of today. Let's take a look at the difference between 72 and 73 from Mar 25 until now, considering medians for the metrics.

On ruwiki (with Priority Hint on 73):

browser_majorloadeventendfirstpaintcount
731236.0975.049484
721283.0997.040348

On frwiki (no Priority Hint)

browser_majorloadeventendfirstpaintcount
731231.0950.03843
721257.0967.03177

Not sure this is conclusive, the gap looks a bit wider on ruwiki but that's about it. There definitely seems to be another effect at hand as well. Either 73 itself is faster, or people with faster internet tend to get the update first.

To be certain of the effect of Priority Hints, I think I need to A/B test within the same corpus (ruwiki+73) and not have all Chrome 73 ruwiki users have it. I wasn't sure how to do that until now, because there's no way to introspect from JS whether the origin trial token was present as a header.

I think I have an idea that will work out to be equivalent: only apply the hint to half the pages, based on page id. As well as a way to tell that the current pageload had any hint applied to it (looking for imgs with the attribute in the DOM is probably the simplest).

Change 499709 had a related patch set uploaded (by Gilles; owner: Gilles):
[mediawiki/vagrant@master] Enable various origin trials

https://gerrit.wikimedia.org/r/499709

Change 499710 had a related patch set uploaded (by Gilles; owner: Gilles):
[mediawiki/vagrant@master] Update origin trials tokens

https://gerrit.wikimedia.org/r/499710

Change 499709 merged by jenkins-bot:
[mediawiki/vagrant@master] Enable various origin trials

https://gerrit.wikimedia.org/r/499709

Change 499710 merged by jenkins-bot:
[mediawiki/vagrant@master] Update origin trials tokens

https://gerrit.wikimedia.org/r/499710

Change 499767 had a related patch set uploaded (by Gilles; owner: Gilles):
[mediawiki/core@master] Only apply high priority hint half the time

https://gerrit.wikimedia.org/r/499767

Change 499909 had a related patch set uploaded (by Krinkle; owner: Gilles):
[mediawiki/core@wmf/1.33.0-wmf.23] Only apply high priority hint half the time

https://gerrit.wikimedia.org/r/499909

Change 499909 merged by jenkins-bot:
[mediawiki/core@wmf/1.33.0-wmf.23] Only apply high priority hint half the time

https://gerrit.wikimedia.org/r/499909

Mentioned in SAL (#wikimedia-operations) [2019-03-29T07:58:42Z] <gilles@deploy1001> Synchronized php-1.33.0-wmf.23/includes/media/MediaTransformOutput.php: T216499 Only apply high priority half the time (duration: 00m 50s)

Change 500537 had a related patch set uploaded (by Gilles; owner: Gilles):
[operations/mediawiki-config@master] Renew Priority Hints origin trial token

https://gerrit.wikimedia.org/r/500537

Change 500537 merged by jenkins-bot:
[operations/mediawiki-config@master] Renew Priority Hints origin trial token

https://gerrit.wikimedia.org/r/500537

Mentioned in SAL (#wikimedia-operations) [2019-04-01T19:48:56Z] <gilles@deploy1001> Synchronized wmf-config/InitialiseSettings.php: T216499 Renew Priority Hints origin trial token (duration: 00m 54s)

Change 500896 had a related patch set uploaded (by Gilles; owner: Gilles):
[mediawiki/core@wmf/1.33.0-wmf.24] Only apply high priority hint half the time

https://gerrit.wikimedia.org/r/500896

Change 500896 merged by jenkins-bot:
[mediawiki/core@wmf/1.33.0-wmf.24] Only apply high priority hint half the time

https://gerrit.wikimedia.org/r/500896

Mentioned in SAL (#wikimedia-operations) [2019-04-03T07:53:26Z] <gilles@deploy1001> Synchronized php-1.33.0-wmf.24/includes/media/ThumbnailImage.php: T216499 Only apply high priority hint half the time (duration: 00m 58s)

Change 500955 had a related patch set uploaded (by Gilles; owner: Gilles):
[mediawiki/core@wmf/1.33.0-wmf.23] Identify images that should have had high importance

https://gerrit.wikimedia.org/r/500955

Change 500955 merged by jenkins-bot:
[mediawiki/core@wmf/1.33.0-wmf.23] Identify images that should have had high importance

https://gerrit.wikimedia.org/r/500955

Mentioned in SAL (#wikimedia-operations) [2019-04-03T13:44:07Z] <gilles@deploy1001> Synchronized php-1.33.0-wmf.23/includes/media/MediaTransformOutput.php: T216499 Identify images that should have had high importance (duration: 00m 59s)

Very early results, to be re-done once there's more data:

SELECT PERCENTILE(nt.event.firstpaint, 0.5) AS median_firstpaint, PERCENTILE(et.event.starttime, 0.5) AS median_elementtime, PERCENTILE(nt.event.transfersize, 0.5) AS median_transfersize, COUNT(*) AS count, et.event.name AS type FROM event.elementtiming et JOIN event.navigationtiming nt ON et.event.pageviewToken = nt.event.pageviewToken WHERE nt.year = 2019 AND et.year = 2019 AND et.event.x > 0 GROUP BY et.event.name;
median_firstpaintmedian_elementtimemedian_transfersizecounttype
975.01021.039006.047thumbnail-top
1027.01071.035224.017671thumbnail-high

At this stage it would suggest that the high priority is counter-productive, but 47 samples is way too small to draw any conclusion. Just posting here to have the query available to run again in a few days.

Update:

SELECT PERCENTILE(nt.event.firstpaint, 0.5) AS median_firstpaint, PERCENTILE(et.event.starttime, 0.5) AS median_elementtime, PERCENTILE(nt.event.transfersize, 0.5) AS median_transfersize, COUNT(*) AS count, et.event.name AS type FROM event.elementtiming et JOIN event.navigationtiming nt ON et.event.pageviewToken = nt.event.pageviewToken WHERE nt.year = 2019 AND et.year = 2019 AND et.event.x > 0 AND et.month = 4 AND nt.month = 4 AND et.day > 3 AND nt.day > 3 AND et.event.name IN ('thumbnail-high', 'thumbnail-top') GROUP BY et.event.name;
median_firstpaintmedian_elementtimemedian_transfersizecounttype
1021.01065.031987.027747thumbnail-top
1005.01043.031659.535046thumbnail-high
-1.56%-2.06%-1.02%

So far this would suggest that giving high priority to the top article image is slightly beneficial, both for the element timing of that image and for firstPaint. I'm not surprised that the effect is small, because we already have good default priorities for everything else. We will be able to confirm this further once we have weeks of data and the gap in median transferSize due to randomness reduces even more.

Change 502809 had a related patch set uploaded (by Gilles; owner: Gilles):
[mediawiki/core@wmf/1.33.0-wmf.25] Only apply high priority hint half the time

https://gerrit.wikimedia.org/r/502809

Change 502809 merged by jenkins-bot:
[mediawiki/core@wmf/1.33.0-wmf.25] Only apply high priority hint half the time

https://gerrit.wikimedia.org/r/502809

Mentioned in SAL (#wikimedia-operations) [2019-04-10T15:28:11Z] <gilles@deploy1001> Synchronized php-1.33.0-wmf.25/includes/media/ThumbnailImage.php: T216499 Only apply high priority hint half the time (duration: 00m 59s)

Update:

median_firstpaintavg_firstpaintmedian_elementtimeavg_elementtimemedian_transfersizeavg_transfersizecounttype
1005.01610.71017.01667.630758.047511.390731thumbnail-high
1020.01655.31027.01685.930796.047454.582725thumbnail-top
-1.47%-2.7%-0.97%-1.08%-0.12%+0.12%

Now that we have a lot of data and the median and average transferSize are really close, we can see without a doubt that the high priority of the top image brings firstPaint gain of 1-2%.

Note that these results desktop-only for now.

I'm going to wait until we collect enough data on Spanish Wikipedia's mobile site before publishing results: T221065: Set up origin trials on Spanish Wikipedia mobile site There's more traffic there, it should be even faster to collect a sufficient amount of samples.

As for looking at the two populations and their perception survey results, it seems like we don't have enough data yet to draw conclusions:

SELECT et.event.name AS type, qr.event.surveyResponseValue AS response, COUNT(*) AS count, AVG(nt.event.transferSize) AS avg_transfersize, PERCENTILE(nt.event.transferSize, 0.5) AS median_transferSize 
FROM event.elementtiming et JOIN event.quicksurveysresponses qr ON et.event.pageviewToken = qr.event.pageviewToken JOIN event.navigationtiming nt ON et.event.pageviewToken = nt.event.pageviewToken WHERE qr.year = 2019 AND et.year = 2019 AND nt.year = 2019 AND et.month = 4 AND qr.month = 4 AND nt.month = 4 AND et.day > 3 AND qr.day > 3 AND nt.day > 3 AND et.wiki = 'ruwiki' AND et.event.name IN ('thumbnail-high', 'thumbnail-top') GROUP BY et.event.name, qr.event.surveyResponseValue;
 typeresponsecountavg_transfersizemedian_transfersize
 thumbnail-highext-quicksurveys-example-internal-survey-answer-negative28650692.931665.0
 thumbnail-highext-quicksurveys-example-internal-survey-answer-neutral20838878.828234.5
 thumbnail-highext-quicksurveys-example-internal-survey-answer-positive312246227.130019.5
 thumbnail-topext-quicksurveys-example-internal-survey-answer-negative25049497.731848.0
 thumbnail-topext-quicksurveys-example-internal-survey-answer-neutral18139715.428948.0
 thumbnail-topext-quicksurveys-example-internal-survey-answer-positive286446936.830719.0

I don't know if we'll have collected enough survey data by the end of the origin trial, though, since it ends at the end of May.

Change 505237 had a related patch set uploaded (by Gilles; owner: Gilles):
[operations/mediawiki-config@master] Enable Priority Hints and Element Timing on eswiki

https://gerrit.wikimedia.org/r/505237

Change 505237 merged by jenkins-bot:
[operations/mediawiki-config@master] Enable Priority Hints and Element Timing on eswiki

https://gerrit.wikimedia.org/r/505237

Mentioned in SAL (#wikimedia-operations) [2019-04-19T12:59:38Z] <gilles@deploy1001> Synchronized wmf-config/InitialiseSettings.php: T216499 T216598 Enable Priority Hints and Element Timing on eswiki (duration: 00m 56s)

Change 505650 had a related patch set uploaded (by Gilles; owner: Gilles):
[mediawiki/core@wmf/1.34.0-wmf.1] Only apply high priority hint half the time

https://gerrit.wikimedia.org/r/505650

Change 505650 merged by Gilles:
[mediawiki/core@wmf/1.34.0-wmf.1] Only apply high priority hint half the time

https://gerrit.wikimedia.org/r/505650

Mentioned in SAL (#wikimedia-operations) [2019-04-22T18:46:52Z] <gilles@deploy1001> Synchronized php-1.34.0-wmf.1/includes/media/ThumbnailImage.php: T216499 Only apply high priority hint half the time (duration: 00m 53s)

Change 499767 merged by jenkins-bot:
[mediawiki/core@master] Make the ratio of requests getting Priority Hints configurable

https://gerrit.wikimedia.org/r/499767

Change 505701 had a related patch set uploaded (by Gilles; owner: Gilles):
[operations/mediawiki-config@master] Set wgPriorityHintsRatio

https://gerrit.wikimedia.org/r/505701

Change 505701 merged by Gilles:
[operations/mediawiki-config@master] Set wgPriorityHintsRatio

https://gerrit.wikimedia.org/r/505701

Mentioned in SAL (#wikimedia-operations) [2019-04-23T07:27:36Z] <gilles@deploy1001> Synchronized wmf-config/InitialiseSettings.php: T216499 Set wgPriorityHintsRatio (duration: 00m 52s)

Change 507545 had a related patch set uploaded (by Gilles; owner: Gilles):
[operations/mediawiki-config@master] Renew origin trial tokens for ruwiki

https://gerrit.wikimedia.org/r/507545

Change 507545 merged by jenkins-bot:
[operations/mediawiki-config@master] Renew origin trial tokens for ruwiki

https://gerrit.wikimedia.org/r/507545

Mentioned in SAL (#wikimedia-operations) [2019-05-01T11:22:05Z] <gilles@deploy1001> Synchronized wmf-config/InitialiseSettings.php: T216499 T216598 T216594 Renew origin trial tokens for ruwiki (duration: 01m 14s)

Mentioned in SAL (#wikimedia-operations) [2019-05-01T11:27:29Z] <gilles> T216499 Y216594 T216598 mwscript purgeList.php ruwiki --all --verbose

Change 509630 had a related patch set uploaded (by Gilles; owner: Gilles):
[mediawiki/vagrant@master] Renew origin trial tokens

https://gerrit.wikimedia.org/r/509630

Change 509630 merged by jenkins-bot:
[mediawiki/vagrant@master] Renew origin trial tokens

https://gerrit.wikimedia.org/r/509630

Looking at the newest data, there might be a slight preference towards high priority top image on mobile (86.45% satisfaction vs 85.5%, sample size 39,880). On desktop there's no visible difference.

Let's run the numbers again after the summer when we have more data, to see if the trend is confirmed. At this stage it seems worth keeping that priority hint long term.

Change 527837 had a related patch set uploaded (by Gilles; owner: Gilles):
[operations/mediawiki-config@master] Renew origin trial tokens

https://gerrit.wikimedia.org/r/527837

Change 527837 merged by jenkins-bot:
[operations/mediawiki-config@master] Renew origin trial tokens

https://gerrit.wikimedia.org/r/527837

Mentioned in SAL (#wikimedia-operations) [2019-08-03T09:35:58Z] <gilles@deploy1001> Synchronized wmf-config/InitialiseSettings.php: T216499 T216594 Renew origin trial tokens (duration: 00m 48s)

There actually isn't that much data, because of the data retention (90 days). For some reason I can't join on the pageviewtoken on the event_sanitized table. Anyway, that means you we have data from May 20 until now to look at. That should be a lot more than before, but it seems like lower summer traffic got in the way of that. Anyway, here's what i'm seeing:

eswiki thumbnail-high ext-quicksurveys-example-internal-survey-answer-negative 818 25052.977995110024 18082.5
eswiki thumbnail-high ext-quicksurveys-example-internal-survey-answer-neutral 784 24955.473214285714 17570.5
eswiki thumbnail-high ext-quicksurveys-example-internal-survey-answer-positive 4787 24210.479005640274 17354.0

eswiki thumbnail-top ext-quicksurveys-example-internal-survey-answer-negative 833 24677.51980792317 17192.0
eswiki thumbnail-top ext-quicksurveys-example-internal-survey-answer-neutral 751 24357.324900133157 17264.0
eswiki thumbnail-top ext-quicksurveys-example-internal-survey-answer-positive 4701 24756.132099553288 17498.0

ruwiki thumbnail-high ext-quicksurveys-example-internal-survey-answer-negative 211 51622.336492890994 33263.0
ruwiki thumbnail-high ext-quicksurveys-example-internal-survey-answer-neutral 140 53801.7 33375.0
ruwiki thumbnail-high ext-quicksurveys-example-internal-survey-answer-positive 1961 53501.382967873535 29726.0

ruwiki thumbnail-top ext-quicksurveys-example-internal-survey-answer-negative 201 42507.18905472637 29597.0
ruwiki thumbnail-top ext-quicksurveys-example-internal-survey-answer-neutral 147 73738.06802721089 39637.0
ruwiki thumbnail-top ext-quicksurveys-example-internal-survey-answer-positive 2067 50837.6076439284 29129.0

This means that for eswiki the satisfaction ratio is 85.4% with priority hint and 84.9% without. For ruwiki 90.3% with priority hint and 91.1% without. Ruwiki probably has insufficient sampling, however, given the significant differences in median transfer size.

This suggests that giving high priority to the first big article thumbnail is slightly beneficial for perceived performance on mobile. At the very least, it's not detrimental. On desktop we can't make that claim, due to insufficient data.