Page MenuHomePhabricator

CPU benchmark are really unstable on emulated mobile
Closed, DeclinedPublic

Description

When we run tests as emulated mobile we artificially slow down the CPU using the Chrome built in slow down. The problem is that it makes the CPU benchmark score unstable:

Screenshot 2022-02-02 at 14.39.37.png (758×2 px, 669 KB)

And that gives us unstable metrics.

Comparing with running on desktop without the slow down:

Screenshot 2022-02-02 at 14.47.31.png (752×2 px, 510 KB)

That said, running on another machine, the desktop tests looks like this:

Screenshot 2022-02-02 at 14.48.27.png (746×2 px, 631 KB)

The last two tests test Barack Obama on different servers, one against WebPageReplay and one directly against Wikipedia. That should not matter but let me verify by just switching the tests for one of the machines tomorrow.

When I look at the metrics for running tests on a dedicated Mac mini (against Wikipedia) the metric are stable:

Screenshot 2022-02-02 at 14.52.04.png (1×2 px, 554 KB)

It would be interesting to check the CPU benchmark over time on Moto G5 and compare it to the emulated mobile tests. Let me do that too.

Event Timeline

I pushed some tests on a Mac mini to run emulated mobile tests with and without CPU throttling., That way we can see if there's any difference between that and the AWS instance.

With the tests on bare metal in T203060 we could see that the CPU metrics was really stable, so when we do the switch to run tests on bare metal instead this will work fine, so there's nothing to do about it right now.