Page MenuHomePhabricator
Create Task
Maniphest T280621

Calibrate connectivity for synthetic tests to match FCP and LCP in Chrome User Experience Report
Closed, ResolvedPublic
Actions

Assigned To
Peter
Authored By
Peter
Apr 20 2021, 6:48 AM
Tags
Referenced Files
F34413939: Screenshot 2021-04-23 at 11.04.32.png
Apr 23 2021, 9:06 AM
F34413940: Screenshot 2021-04-23 at 11.04.12.png
Apr 23 2021, 9:06 AM
F34413941: Screenshot 2021-04-23 at 11.04.22.png
Apr 23 2021, 9:06 AM
F34413936: Screenshot 2021-04-23 at 11.04.40.png
Apr 23 2021, 9:06 AM
F34413937: Screenshot 2021-04-23 at 11.04.47.png
Apr 23 2021, 9:06 AM
F34413934: Screenshot 2021-04-23 at 11.05.08.png
Apr 23 2021, 9:06 AM
F34413938: Screenshot 2021-04-23 at 11.04.54.png
Apr 23 2021, 9:06 AM
F34413935: Screenshot 2021-04-23 at 11.05.01.png
Apr 23 2021, 9:06 AM
Subscribers
Jdlrobson
Peter

Description

Let me fine tune the connectivity so that we match the 75 percentile metrics we get out of the Chrome User Experience report. That way we can be in front of changes that will happen and do not need to wait on crux to pick it up.

  1. Upgrade to latest Chrome/Firefox first
  2. Update the dashboard to more clearly show Google Web Vitals
  3. Update this issue with the values we have today and compare them with data from Crux
  4. Make sure we get the individual metrics per URL in Crux for the most tested URLs on English Wikipedia
  5. Lets tune the connectivity so the metrics match as much as possible.

Details

SubjectRepoBranchLines +/-
Switch to 4g to better match Crux 75 p.performance/synthetic-monitoring-testsmaster+1 -1
Increase emulate mobile latency even more to match crux.performance/synthetic-monitoring-testsmaster+2 -0
Moved from 160 -> 180 latency for WPR.performance/synthetic-monitoring-testsmaster+1 -1
Set lower latency for Firefox using WebPageReplay.performance/synthetic-monitoring-testsmaster+7 -1
Increase latency even more to meet Crux p75.performance/synthetic-monitoring-testsmaster+1 -1
Increase latency from 100 -> 120 to match CrUx.performance/synthetic-monitoring-testsmaster+1 -1
Update to Chrome 90 and Firefox 88.performance/synthetic-monitoring-testsmaster+1 -1
Customize query in gerrit

Related Objects

Event Timeline

Peter created this task.Apr 20 2021, 6:48 AM
gerritbot added a comment.Apr 20 2021, 7:49 AM

Change 681294 had a related patch set uploaded (by Phedenskog; author: Phedenskog):

[performance/synthetic-monitoring-tests@master] Update to Chrome 90 and Firefox 88.

https://gerrit.wikimedia.org/r/681294

gerritbot added a project: Patch-For-Review.Apr 20 2021, 7:49 AM

Change 681294 merged by jenkins-bot:

[performance/synthetic-monitoring-tests@master] Update to Chrome 90 and Firefox 88.

https://gerrit.wikimedia.org/r/681294

Maintenance_bot removed a project: Patch-For-Review.Apr 20 2021, 8:10 AM
Peter moved this task from Inbox, needs triage to Doing: Goals on the Performance-Team board.Apr 20 2021, 5:45 PM
Peter added a comment.Apr 23 2021, 9:02 AM

I created a dashboard where we have the metric for all tools: https://grafana.wikimedia.org/d/N-K4xrXGk/synthetic-testing-calibration?orgId=1

Peter added a comment.Apr 23 2021, 9:03 AM

Let me tune the settings on Monday. Lets start with WebPageTest and WebPageReplay.

Peter added a comment.Apr 23 2021, 9:06 AM

These are the current metrics:

Screenshot 2021-04-23 at 11.05.08.png (1×1 px, 580 KB)

Screenshot 2021-04-23 at 11.05.01.png (992×1 px, 585 KB)

Screenshot 2021-04-23 at 11.04.40.png (982×2 px, 807 KB)

Screenshot 2021-04-23 at 11.04.54.png (988×2 px, 803 KB)

Screenshot 2021-04-23 at 11.04.47.png (986×2 px, 807 KB)

Screenshot 2021-04-23 at 11.04.32.png (986×2 px, 818 KB)

Screenshot 2021-04-23 at 11.04.12.png (994×2 px, 1019 KB)

Screenshot 2021-04-23 at 11.04.22.png (984×2 px, 1 MB)

Peter mentioned this in T281558: Investigate possible performance improvements for Main_Page on mobile.Apr 30 2021, 9:56 AM
gerritbot added a comment.May 10 2021, 12:28 PM

Change 688266 had a related patch set uploaded (by Phedenskog; author: Phedenskog):

[performance/synthetic-monitoring-tests@master] Increase latency from 100 -> 120 to match CrUx.

https://gerrit.wikimedia.org/r/688266

gerritbot added a project: Patch-For-Review.May 10 2021, 12:28 PM
gerritbot added a comment.May 10 2021, 12:33 PM

Change 688266 merged by jenkins-bot:

[performance/synthetic-monitoring-tests@master] Increase latency from 100 -> 120 to match CrUx.

https://gerrit.wikimedia.org/r/688266

Maintenance_bot removed a project: Patch-For-Review.May 10 2021, 1:10 PM
Peter added a subscriber: Jdlrobson.May 10 2021, 1:37 PM

@Jdlrobson wanted to give you a heads up that I want to change the connectivity setting for the emulated mobile tests that we run on WebPageTest (today its 3gfast). My main plan is to try to calibrate the metrics that we get from Chrome UX Report. The idea is that if they better match, we could (best case) see changes that will affect us earlier in our tooling, instead of the Crux report and in SEO-ranking.

When I look at our metrics today, the WebPageTest metrics are much higher than the 75 percentile reported in Crux. When I look specific at the Obama page the 75p for first contentful paint is 1.28 seconds and in WebPageTest its 2.22 seconds (almost one second higher!). I want to change so we test it with 4g settings instead, it will not perfectly match but it will be much closer. Let me know if you feel ok with the change.

gerritbot added a comment.May 11 2021, 7:18 AM

Change 688884 had a related patch set uploaded (by Phedenskog; author: Phedenskog):

[performance/synthetic-monitoring-tests@master] Increase latency even more to meet Crux p75.

https://gerrit.wikimedia.org/r/688884

gerritbot added a project: Patch-For-Review.May 11 2021, 7:18 AM
Peter mentioned this in T282517: Firefox WebPageReplay tests stopped working when latency was increased.May 11 2021, 8:23 AM
gerritbot added a comment.May 11 2021, 8:25 AM

Change 688884 merged by jenkins-bot:

[performance/synthetic-monitoring-tests@master] Increase latency even more to meet Crux p75.

https://gerrit.wikimedia.org/r/688884

gerritbot added a comment.May 11 2021, 8:32 AM

Change 688914 had a related patch set uploaded (by Phedenskog; author: Phedenskog):

[performance/synthetic-monitoring-tests@master] Set lower latency for Firefox using WebPageReplay.

https://gerrit.wikimedia.org/r/688914

gerritbot added a comment.May 11 2021, 8:34 AM

Change 688914 merged by jenkins-bot:

[performance/synthetic-monitoring-tests@master] Set lower latency for Firefox using WebPageReplay.

https://gerrit.wikimedia.org/r/688914

gerritbot added a comment.May 11 2021, 12:07 PM

Change 689004 had a related patch set uploaded (by Phedenskog; author: Phedenskog):

[performance/synthetic-monitoring-tests@master] Moved from 160 -> 180 latency for WPR.

https://gerrit.wikimedia.org/r/689004

gerritbot added a comment.May 11 2021, 12:09 PM

Change 689004 merged by jenkins-bot:

[performance/synthetic-monitoring-tests@master] Moved from 160 -> 180 latency for WPR.

https://gerrit.wikimedia.org/r/689004

gerritbot added a comment.May 11 2021, 12:15 PM

Change 689012 had a related patch set uploaded (by Phedenskog; author: Phedenskog):

[performance/synthetic-monitoring-tests@master] Increase emulate mobile latency even more to match crux.

https://gerrit.wikimedia.org/r/689012

gerritbot added a comment.May 11 2021, 12:17 PM

Change 689012 merged by jenkins-bot:

[performance/synthetic-monitoring-tests@master] Increase emulate mobile latency even more to match crux.

https://gerrit.wikimedia.org/r/689012

gerritbot added a comment.May 14 2021, 6:57 AM

Change 691041 had a related patch set uploaded (by Phedenskog; author: Phedenskog):

[performance/synthetic-monitoring-tests@master] Switch to 4g to better match Crux 75 p.

https://gerrit.wikimedia.org/r/691041

gerritbot added a comment.May 14 2021, 7:01 AM

Change 691041 merged by jenkins-bot:

[performance/synthetic-monitoring-tests@master] Switch to 4g to better match Crux 75 p.

https://gerrit.wikimedia.org/r/691041

Peter closed this task as Resolved.Sep 20 2021, 6:24 PM
Peter mentioned this in rPSMTd0e94251363f: Update to Chrome 90 and Firefox 88..Apr 8 2024, 3:14 AM
Peter mentioned this in rPSMTc089e2b51aa6: Increase latency from 100 -> 120 to match CrUx..
Peter mentioned this in rPSMTcb65ab2d959c: Increase latency even more to meet Crux p75..
Peter mentioned this in rPSMT723526fa5c34: Set lower latency for Firefox using WebPageReplay..
Peter mentioned this in rPSMT94a71eaa10b9: Increase emulate mobile latency even more to match crux..
Peter mentioned this in rPSMT24d662db9b3c: Moved from 160 -> 180 latency for WPR..
Peter mentioned this in rPSMTe89e0aa7b48f: Switch to 4g to better match Crux 75 p..
Maintenance_bot removed a project: Patch-For-Review.Apr 8 2024, 3:30 AM
Content licensed under Creative Commons Attribution-ShareAlike (CC BY-SA) 4.0 unless otherwise noted; code licensed under GNU General Public License (GPL) 2.0 or later and other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL