Page MenuHomePhabricator
Create Task
Maniphest T307984

Sync TTFB/CPU benchmark for synthetic tests with data from the Chrome User Experience report and our own reporting
Open, Needs TriagePublic
Actions

Description

A year ago I tried to use First Contentful Paint to sync the speed our synthetic tests but its easier and makes more sense to use time to first byte now when it's available in the Chrome User Experience report. That way our synthetic tests try to match our 75 percentile (of Chrome users).

On mobile English Wikipedia TTFB is 638 ms and desktop 467 ms (https://grafana.wikimedia.org/d/t_bhsNGMk/chrome-user-experience-report?orgId=1).

In our testing today the TTFB for emulated mobile 450 ms is and desktop 190 ms for first view (warm cache is faster).

Event Timeline

Peter created this task.May 10 2022, 8:29 AM
Restricted Application added a subscriber: Aklapper. · View Herald TranscriptMay 10 2022, 8:29 AM
Peter moved this task from Inbox, needs triage to Backlog: Maintenance, non-prioritized on the Performance-Team board.May 16 2022, 6:38 PM
Peter added a comment.Aug 16 2022, 7:50 AM

This is perfect to do when we move to bare metal and our real mobile phone tests.

Peter renamed this task from Sync TTFB for synthetic tests with data from the Chrome User Experience report to Sync TTFB/CPU benchmark for synthetic tests with data from the Chrome User Experience report and our own reporting.Mar 21 2023, 8:25 AM

I looked at the CPU benchmark for India. The 75 percentile for mobile is 285 ms (span 259 -352) and 95 percentile 466 (span 394 - 544). For desktop in India the mean 75 p is 147 ms and the 95 p is 349 ms.

Peter added a comment.Mar 21 2023, 8:30 AM

Looking at the CPU benchmark in our testing infrastructure:

Samsung A51: 187 ms
Moto G5: 375 ms
AWS: 61 ms
Bare metal: 204 ms

We can see there that the Samsung A51 is too fast to match 75 p in India. Those phones run the CPU pinned to max, let me either change so we can run half the speed or we can set the speed.

Krinkle edited projects, added NavigationTiming, Synthetic-Performance-Testing; removed Performance-Team.Mar 10 2024, 2:40 AM
Content licensed under Creative Commons Attribution-ShareAlike (CC BY-SA) 4.0 unless otherwise noted; code licensed under GNU General Public License (GPL) 2.0 or later and other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL