Page MenuHomePhabricator

Test TTFB stability using gnirehtet with an Android device
Closed, ResolvedPublic

Description

Lets run a test at least for a day using gnirehtet to rout the traffic back from a Android device to a throttled Mac connection. That way we have baseline of what kind of stability we can get for Android.

Event Timeline

I got "ERROR Router: Cannot create route, dropping packet: Too many open files (os error 24)" from gnirehtet when I tried doing 11 runs. However, it seems that gnirehtet adds some overhead, so I need to check that too. For first test TTFB looks ok (my Mac is using a wifi though so I should test with a better connection). Running on 4g connection make the TTFB go between 2.38-2.42 but we need to test for a couple of hours to have better metrics.

The too many files is fixed with https://krypted.com/mac-os-x/maximum-files-in-mac-os-x/ and setting the number of files open per shell ulimit -S -n XXX.

It seems that we get ok TTFB. This is a good run:

[2021-02-10 09:34:47] INFO: Versions OS: darwin 20.3.0 nodejs: v14.15.4 sitespeed.io: 16.8.0 browsertime: 11.4.0 coach: 6.0.1
[2021-02-10 09:34:47] INFO: Changing network interfaces needs sudo rights.
[2021-02-10 09:34:53] INFO: Run tests on Moto G (5) [ZY322GXR4B] using Android version 7.0
[2021-02-10 09:34:53] INFO: Running tests using Chrome - 11 iteration(s)
[2021-02-10 09:35:09] INFO: Testing url https://en.wikipedia.org/wiki/Barack_Obama iteration 1
[2021-02-10 09:35:47] INFO: https://en.wikipedia.org/wiki/Barack_Obama TTFB: 2.38s DOMContentLoaded: 4.06s firstPaint: 3.91s FCP: 3.91s LCP: 3.91s Load: 6.41s TBT: 620ms CLS:0.0681
[2021-02-10 09:36:03] INFO: Testing url https://en.wikipedia.org/wiki/Barack_Obama iteration 2
[2021-02-10 09:36:39] INFO: https://en.wikipedia.org/wiki/Barack_Obama TTFB: 2.40s DOMContentLoaded: 4.08s firstPaint: 3.98s FCP: 3.98s LCP: 3.98s Load: 5.24s TBT: 660ms CLS:0.0681
[2021-02-10 09:36:55] INFO: Testing url https://en.wikipedia.org/wiki/Barack_Obama iteration 3
[2021-02-10 09:37:31] INFO: https://en.wikipedia.org/wiki/Barack_Obama TTFB: 2.36s DOMContentLoaded: 3.86s firstPaint: 3.91s FCP: 3.91s LCP: 3.91s Load: 4.91s TBT: 643ms CLS:0.0681
[2021-02-10 09:37:47] INFO: Testing url https://en.wikipedia.org/wiki/Barack_Obama iteration 4
[2021-02-10 09:38:23] INFO: https://en.wikipedia.org/wiki/Barack_Obama TTFB: 2.38s DOMContentLoaded: 3.87s firstPaint: 4.12s FCP: 4.12s LCP: 4.12s Load: 5.04s TBT: 654ms CLS:0.0681
[2021-02-10 09:38:39] INFO: Testing url https://en.wikipedia.org/wiki/Barack_Obama iteration 5
[2021-02-10 09:39:15] INFO: https://en.wikipedia.org/wiki/Barack_Obama TTFB: 2.37s DOMContentLoaded: 3.87s firstPaint: 3.88s FCP: 3.88s LCP: 3.88s Load: 5.00s TBT: 667ms CLS:0.0681
[2021-02-10 09:39:31] INFO: Testing url https://en.wikipedia.org/wiki/Barack_Obama iteration 6
[2021-02-10 09:40:07] INFO: https://en.wikipedia.org/wiki/Barack_Obama TTFB: 2.41s DOMContentLoaded: 3.95s firstPaint: 3.95s FCP: 3.95s LCP: 3.95s Load: 4.90s TBT: 662ms CLS:0.0681
[2021-02-10 09:40:23] INFO: Testing url https://en.wikipedia.org/wiki/Barack_Obama iteration 7
[2021-02-10 09:40:59] INFO: https://en.wikipedia.org/wiki/Barack_Obama TTFB: 2.38s DOMContentLoaded: 3.87s firstPaint: 4.14s FCP: 4.14s LCP: 4.14s Load: 5.00s TBT: 677ms CLS:0.0681
[2021-02-10 09:41:15] INFO: Testing url https://en.wikipedia.org/wiki/Barack_Obama iteration 8
[2021-02-10 09:41:51] INFO: https://en.wikipedia.org/wiki/Barack_Obama TTFB: 2.39s DOMContentLoaded: 3.91s firstPaint: 3.85s FCP: 3.85s LCP: 3.85s Load: 5.00s TBT: 666ms CLS:0.0681
[2021-02-10 09:42:07] INFO: Testing url https://en.wikipedia.org/wiki/Barack_Obama iteration 9
[2021-02-10 09:42:43] INFO: https://en.wikipedia.org/wiki/Barack_Obama TTFB: 2.36s DOMContentLoaded: 3.90s firstPaint: 4.09s FCP: 4.09s LCP: 4.09s Load: 4.97s TBT: 654ms CLS:0.0681
[2021-02-10 09:43:00] INFO: Testing url https://en.wikipedia.org/wiki/Barack_Obama iteration 10
[2021-02-10 09:43:36] INFO: https://en.wikipedia.org/wiki/Barack_Obama TTFB: 2.40s DOMContentLoaded: 3.96s firstPaint: 3.98s FCP: 3.98s LCP: 3.98s Load: 4.95s TBT: 666ms CLS:0.0681
[2021-02-10 09:43:52] INFO: Testing url https://en.wikipedia.org/wiki/Barack_Obama iteration 11
[2021-02-10 09:44:28] INFO: https://en.wikipedia.org/wiki/Barack_Obama TTFB: 2.39s DOMContentLoaded: 3.94s firstPaint: 3.87s FCP: 3.87s LCP: 3.87s Load: 4.96s TBT: 646ms CLS:0.0681
[2021-02-10 09:44:28] INFO: https://en.wikipedia.org/wiki/Barack_Obama 28 requests, TTFB: 2.38s (±4.49ms), firstPaint: 3.97s (±29.61ms), FCP: 3.97s (±29.61ms), DOMContentLoaded: 3.94s (±21.93ms), LCP: 3.97s (±29.61ms), CLS: 0.0681 (±0.00), TBT: 656ms (±4.44ms), Load: 5.13s (±125.27ms) (11 runs)

But we sometimes we get +20ms runs. I've simulated 4g since it seems using gnirehtet adds overhead. I'm gonna run it for half a day or something and send the metrics to Graphite to be sure its stable over time,

I haven't been able to get any stability on the tests, after a while the phone loses it connection to internet. I cannot see anything in the logs from gnirehtet and its hard to say if its the client on the phone or desktop. However if I restart the client on the phone its starts to work.

I've been trying gnirehtet on a Mac for a couple of days now and the stability in metrics is in general better than using a wifi. However sometimes the pages stop to load and I haven't been able to nail down the problem. I can see that when it happens it never recovers until I restart both gnirehtet on the phone and on the computer. Restarting only on the phone do not solve the problem. However when Chrome can't access the page, I could use curl on the phone to access the same URL.

I've been running 15 iterations. It happened I can do that 4-5 times without any problem and then suddenly it starts to fail. I will add this to the runs at Bitbar to see how it works there.