Evaluate number of runs for WebPageTest to get more stable metrics
Closed, ResolvedPublic
Actions

Assigned To

Authored By

	Peter
	Mar 8 2017, 11:33 AM

Description

@Krinkle added the change for increasing the number of runs to see if we can get more stable metrics for WebPageTest. We haven't done any testing since the beginning when we setup WebPageTest and we have changed the size of the instance, changes URLs and added timeline and other data collection.

I've added some test: This morning I've changed number runs from 5 to 11 on my test instance and then also crated a new one that runs 21 runs per URL (yes twentyone). I'll add the graphs here tonight and we can keep it running for a couple of days just to see how it's doing and then after that take decision.

Related Objects

Mentioned In: T160315: Remove mwLoadStart/mediaWikiLoadStart and track mwLoadEnd relative to fetchStart
Mentioned Here: T168526: Set up a server running mitmproxy + mahimahi to measure key synthetic testing over time

Event Timeline

Peter created this task.Mar 8 2017, 11:33 AM

Restricted Application added a subscriber: Aklapper. · View Herald TranscriptMar 8 2017, 11:33 AM

Ok I got some numbers. I've compared only SpeedIndex and calculated standard deviation for 30-35 runs running over a little more than a day however it isn't perfect since i used sitespeed.io and Digital Ocean for 21 runs and 5 our regular WebPageTest instance.

Runs	URL	AVG	Stdev	Relative stdev
21	Obama	1037	20	1,9%
21	Facebook	1114	28	2,5%
21	Sweden	1162	52	4,4%
5	Obama	1739	116	6,5%
5	Facebook	1664	44	2,6%
5	Sweden	1726	68	3,9%

In these runs we only see a real win for the Obama page. To test with WebPageTest it's more work, but I think we should do it so we get some numbers and compare apples with apples.

Maybe nothing perfectly match for this issue but it seems like we started to have real big diff in mwLoadStart (1 second) ... going back in time I can see that we have had it for a month at least, maybe longer.

Screen Shot 2017-03-09 at 8.03.45 PM.png (606×2 px, 117 KB)

Peter claimed this task.Mar 9 2017, 9:46 PM

Peter moved this task from Inbox, needs triage to To-do: Goals prioritized current Quarter on the Performance-Team board.

Peter mentioned this in T160315: Remove mwLoadStart/mediaWikiLoadStart and track mwLoadEnd relative to fetchStart.Mar 13 2017, 7:52 AM

Before doing anymore testing for find the optimal runs, I think we should investigate the large difference in mwLoadStart.

Firefox (the difference is 2 seconds):

Screen Shot 2017-03-13 at 8.41.33 AM.png (620×2 px, 155 KB)

Chrome (the difference is 1 second):

Screen Shot 2017-03-13 at 8.42.26 AM.png (616×2 px, 170 KB)

Internet Explorer (using HTTP1 the difference is less than 100 ms):

Screen Shot 2017-03-13 at 8.42.48 AM.png (610×2 px, 133 KB)

I think T168526 is the thing we need

Peter moved this task from To-do: Goals prioritized current Quarter to Backlog: Maintenance, non-prioritized on the Performance-Team board.Jul 4 2017, 12:01 PM

We should look into https://github.com/catapult-project/catapult/blob/master/web_page_replay_go/README.md

Moving to proxy in the future.

	F6516057: Screen Shot 2017-03-13 at 8.42.26 AM.png
	May 30 2017, 9:40 AM

	F6516074: Screen Shot 2017-03-13 at 8.42.48 AM.png
	May 30 2017, 9:40 AM

	F6515957: Screen Shot 2017-03-13 at 8.41.33 AM.png
	May 30 2017, 9:40 AM

	F6378432: Screen Shot 2017-03-09 at 8.03.45 PM.png
	Mar 9 2017, 7:08 PM

Evaluate number of runs for WebPageTest to get more stable metricsClosed, ResolvedPublicActions

Description

Related Objects

Event Timeline

Evaluate number of runs for WebPageTest to get more stable metrics
Closed, ResolvedPublic
Actions