Page MenuHomePhabricator

Investigate and understand difference in webpagetest setups
Closed, DeclinedPublic

Description

Running identical tests on webpagetest suggests fully loaded time on a 2G connection is currently 38s:

However on our instance it is suggested that it is 22s:

Which is closer to reality and why are these results so different?

Event Timeline

Restricted Application added a subscriber: Aklapper. · View Herald TranscriptApr 5 2016, 11:45 AM
Peter added a subscriber: Peter.Apr 7 2016, 7:01 AM
Peter added a comment.Apr 7 2016, 7:45 AM

I think thinking of WebPageTest and other synthetic testing as the "real" time it takes for the user is not the best way to think because it varies so much in the real world. What's important when we do the testing is that we get consistent numbers when we run from the same instance, so that we can measure if the changes we do increase or decrease the metrics.

About the difference in fully loaded:

  • We test with different connection types. On our instance we used: 35000/32000 Kbps, 1300ms Latency, For WebPageTest we used 2G, when I check the config in the dropdown it looks like that it is 28000/25600/1300.
  • For WebPageTest we used a real phone and our instance we tested with emulated Chrome.
  • We tested different URLs (I didn't check if it is different content, I only checked that the URLs is different).

I think if we try with the same setting we will have metrics that looks more the same. However still think it's more important that our metrics are consistent from the same device :)

FYI: one annoying thing with WebPageTest is that it default picks the median run of all the runs for page load time. You can change that by adding a couple of parameters to the URL. This will choose the median run for using FullyLoaded as metric:
?medianRun=median&medianMetric=FullyLoaded

So it will looks like this:
http://www.webpagetest.org/result/160329_QB_157K/?medianRun=median&medianMetric=FullyLoaded

I think thinking of WebPageTest and other synthetic testing as the "real" time it takes for the user is not the best way to think because it varies so much in the real world. What's important when we do the testing is that we get consistent numbers when we run from the same instance, so that we can measure if the changes we do increase or decrease the metrics.

Agreed.. but ideally I want this value to be as close to the slow connections our users are exposed to in the real world. We have the consistent numbers part more or less covered, but I am a little concerned at the 22s to 38s difference - as our changes have a much greater impact on the latter and we're not predicting them well.

FYI: one annoying thing with WebPageTest is that it default picks the median run of all the runs for page load time. You can change that by adding a couple of parameters to the URL. This will choose the median run for using FullyLoaded as metric:
?medianRun=median&medianMetric=FullyLoaded

Thanks didn't know about that.

  • We test with different connection types. On our instance we used: 35000/32000 Kbps, 1300ms Latency, For WebPageTest we used 2G, when I check the config in the dropdown it looks like that it is 28000/25600/1300.
  • We tested different URLs (I didn't check if it is different content, I only checked that the URLs is different).

Whoops. Timo updated and I neglected that. They shouldn't vary too much but I've run again with the same URL and speed just to be sure.
http://wpt.wmftest.org/result/160408_V6_65/
http://www.webpagetest.org/result/160408_K2_ZKT/?medianRun=median&medianMetric=FullyLoaded
but I'm still hitting that inconsistency.

I think if we try with the same setting we will have metrics that looks more the same. However still think it's more important that our metrics are consistent from the same device :)

Yes, but I think we do need to cover a REAL mobile device rather than an emulated device if this is the big difference. From what I've read up on this subject this really matters when testing mobile performance as real devices have different memory/processing power... all sorts of things? Is this something we can make happen - if the webpagetest.org values are more realistic of what's in the wild, I'd like to setup performance tests that cover this our side to avoid people making statements such as "X page loads in Ys on mobile" or "X page loads Z faster on mobile now".

  • For WebPageTest we used a real phone and our instance we tested with emulated Chrome.

I think this is the key point here. We should definitely use a real phone if we can as mobile devices differ greatly from emulated ones. We have lots of idle phones in the office we could make use of for this purpose if necessary.... how do we make that happen?

Jdlrobson closed this task as Declined.Jun 15 2017, 10:14 PM

Not working on this.