This should be quite easy to add, so we give a latency and then add a small variance for each run. That can help us so we don't hit an exact latency that can cause us problems.
Description
Description
Details
Details
Customize query in gerrit
Related Objects
Related Objects
- Mentioned In
- rPSMTae44aa836844: Remove variance latency 2% tests since I'm finished.
rPSMTa78257443537: Rename config file to be correctly picked up.
rPSMT9cee5f89776f: Try out 2% variance in latency when replaying.
rPSMTffe9f37a34b0: Remove desktop latency variance tests.
rPSMT2d2bdb528f97: Run tests with 2% RTT variance.
Event Timeline
Comment Actions
I've started to implement this. Trying to make as simple as possible: you add a percentage of variance and then it is set for every run. I've started to just change the latency, that's what we use for WebPageReplay.
To test it out we need to run it for a couple of days, would be cool if I could roll it out tomorrow and then it can run during next week when I',m on vacation.
Comment Actions
I got this to work, but I haven't been able to push this somewhere yet. It's hard to see the outcome on my local machine. I'll push it to Browsertime, but I think its best to add a test case when I get back from vacation.
At the moment it works like this: you add a variance in % and that it will be used for every run. Say that we run with cable connectivity (using RTT 28) and run with a variance of 2%, then the RTT will be 28 or 29 for each run.
Comment Actions
Change 595461 had a related patch set uploaded (by Phedenskog; owner: Phedenskog):
[performance/synthetic-monitoring-tests@master] Run tests with 2% RTT variance.
Comment Actions
Change 595461 merged by jenkins-bot:
[performance/synthetic-monitoring-tests@master] Run tests with 2% RTT variance.
Comment Actions
I've tested without WPR for a while and no direct difference.
Variance:
Without:
The idea is that we could hit certain thresholds where just a small diff in latency and running a small variance could make use see that in the metrics, but no luck.
Comment Actions
Change 598438 had a related patch set uploaded (by Phedenskog; owner: Phedenskog):
[performance/synthetic-monitoring-tests@master] Remove desktop latency variance tests.
Comment Actions
Change 598438 merged by jenkins-bot:
[performance/synthetic-monitoring-tests@master] Remove desktop latency variance tests.
Comment Actions
Change 598440 had a related patch set uploaded (by Phedenskog; owner: Phedenskog):
[performance/synthetic-monitoring-tests@master] Try out 2% variance in latency when replaying.
Comment Actions
Change 598440 merged by jenkins-bot:
[performance/synthetic-monitoring-tests@master] Try out 2% variance in latency when replaying.
Comment Actions
Change 598809 had a related patch set uploaded (by Phedenskog; owner: Phedenskog):
[performance/synthetic-monitoring-tests@master] Rename config file to be correctly picked up.
Comment Actions
Change 598809 merged by jenkins-bot:
[performance/synthetic-monitoring-tests@master] Rename config file to be correctly picked up.
Comment Actions
I tested with WebPageReplay on emulated mobile variance of 2% of 100 latency. I could't see any difference in median TTFB (but I could see that the variance was set) so my conclusion is that 2% doesn't gives us any difference. Maybe we should try another tactic in the future but for now this experience is done.
Facebook with WPR:
Facebook with WPR and 2% variance:
And Obama WPR:
Obama WPR and 2% variance:
Comment Actions
Change 599772 had a related patch set uploaded (by Phedenskog; owner: Phedenskog):
[performance/synthetic-monitoring-tests@master] Remove variance latency 2% tests since I'm finished.
Comment Actions
Change 599772 merged by jenkins-bot:
[performance/synthetic-monitoring-tests@master] Remove variance latency 2% tests since I'm finished.