| Peter | |
| Feb 18 2021, 10:43 AM |
| F34459891: Screenshot 2021-05-19 at 13.45.24.png | |
| May 19 2021, 11:48 AM |
| F34459890: Screenshot 2021-05-19 at 13.45.32.png | |
| May 19 2021, 11:48 AM |
| F34459884: Screenshot 2021-05-19 at 13.40.44.png | |
| May 19 2021, 11:48 AM |
| F34459885: Screenshot 2021-05-19 at 13.40.35.png | |
| May 19 2021, 11:48 AM |
| F34118659: Screenshot 2021-02-22 at 13.48.35.png | |
| Feb 22 2021, 12:50 PM |
| F34118660: Screenshot 2021-02-22 at 13.48.24.png | |
| Feb 22 2021, 12:50 PM |
| F34118652: Screenshot 2021-02-22 at 13.45.30.png | |
| Feb 22 2021, 12:50 PM |
| F34118651: Screenshot 2021-02-22 at 13.45.41.png | |
| Feb 22 2021, 12:50 PM |
One of the key issues for us with the synthetic testing is to get stable TTFB between test runs. If we have a stable TTFB it will be easier to find front end regressions. This is extra important for our mobile testing since we have had problems getting stable metrics.
Let me document what kind of variation we have today in our different setup as a start and maybe we can make a blog post of it in the future.
I'll go through the different tests we run, focusing on the Barack Obama page since we test that page on all different setups. Where we have 7 days of data I'll use that and look at min/median/max TTFB during that time and also the standard derivation.
We run a couple of test using browsertime/sitespeed.io testing the first view of Wikipedia. So far I mainly been using them to verify that the WebPageReplay tests is ok. We run the tests inside Docker and the throttling use Throttle.
The median TTFB through 7 days are 175-184 ms. I would rank that as good, but we can also see that the max value and standard deviation sometimes have spikes.
We run tests where we set Chrome in emulation mode and try to emulate running on an Android phone. We run the tests inside Docker and the throttling use Throttle.
I've also been testing running desktop tests on a Mac mini at Mac Stadium. There we do not run inside Docker but we still use Throttle to throttle the connection.
I also use that instance to test the iOS simulator, running tests as the latest iPhone.
We also run tests using WebPageReplay as a replay proxy. When we do that we add 100 ms latency on localhost.
We also run tests using WebPageReplay as a replay proxy. When we do that we add 100 ms latency on localhost.
WebPageTest do their own throttling but its based on the same idea. Today we only have the median TTFB however its not the same as in the other tests, since with WebPageReplay we pick the median Speed Index run and use that run and take the TTFB from that run, so we cannot compare with the rest. The standard deviation is measured the same way though:
We also run tests where emulate Chrome and use the 3gfast setting. It's the same thing here with the median: it is picked from the run with the median Speed Index so we cannot compare that directly with the rest of our tools.
We also run tests on Android phones. Today we use Kobiton as host and we have had some problems with the connectivity setup.
We also run tests on an Android phone against a WebPageReplay replay server.
I'm gonna add some data from Bitbar too but want to give them chance to tweak their settings first. What's also missing is using gnirehtet and I have a hard time having a device up and running at home for 7 days.
I went through add calculated the difference in TTFB % over time by the formula: (max - min) /min
| Type | Difference |
| Desktop simulate cable connection on AWS | 5,1 % |
| Emulated mobile simulate 3g connection on AWS | 25% |
| Desktop simulate cable connection on a Mac mini M1 | 24% |
| Simulate iOS mobile 3g connection on a Mac mini M1 | 1,6 % |
| Desktop WebPageReplay on AWS | 0,7% |
| Emulated mobile WebPageReplay on AWS | 1,0 % |
| WebPageTest desktop on AWS using cable | 318% |
| WebPageTest emulated mobile on AWS using 3gfast | 6,3% |
| Android tests using "3g"-wifi Kobiton | 193% |
| Android tests using WebPageReplay and 100 ms latency | 1,25% (edited) |
Something is seriously broken on the WebPageReplay tests on Android, I think something going on with the hosting, I'll get back to Kobiton about that.
Also WebPageTest tests on Desktop stands out, I wonder if we have a bad instance and should just replace it. Would be interesting to also compare with what we get when we move the server inhouse.
@dpifke interested in your feedback on this. Also it would be great to have your eyes on the commands that actually do the throttling https://github.com/sitespeedio/throttle - the actual tc/pfctl commands that runs since you started to implement that for the wifi throttling, I haven't had any eyes on that for a while and maybe there are room for improvements that can give us more stable metrics.
There where something strange going on on the Mac mini/Kobiton side when we collected the metrics for Android/WebPageReplay setup. Checking a new timespan it looks like this.
The difference in TTFB is then 1,25%. That almost match the emulated mobile running on AWS which is pretty cool.
I'm adding the Bitbar numbers here from the first test:
And median variation: 35%.
And median variation: 22%.
The Mac mini has been moved and the phones use netropy device for wifi. It's been setup with 9000 kb/s up and down + 40 ms delay in both directions. Also when the mini been moved the metrics looks much better for WebPageReplay.
The difference is 0.1% for median ttfb (really good).
The difference is 25%.