Today we have a way of getting stable metrics though Moto G5, but we haven't had it running for a time. We should try to collect data for a couple of days/week and see that the metric are stable.
Description
Description
| Status | Subtype | Assigned | Task | ||
|---|---|---|---|---|---|
| Resolved | Peter | T197847 Collect performance metrics from a mobile over time | |||
| Declined | Peter | T209025 Drive a Moto G5 Chrome from a Raspberry Pi | |||
| Resolved | Peter | T212485 Automatically match phone with bus/devnum | |||
| Resolved | Peter | T213261 Investigate difference in firstPaint/firstVisualChange using WebPageReplay compared to not using it | |||
| Resolved | Peter | T213676 Update Chrome using ADB | |||
| Declined | • dpifke | T214026 Create a list of what's needed for getting the mobile phone performance lab running | |||
| Resolved | Peter | T214027 Collect mobile phone statistics and send them to Graphite | |||
| Resolved | Peter | T217729 Try out gnirehtet for reverse tethering |
Event Timeline
Comment Actions
Checked our current runs with WPT (https://www.webpagetest.org/results.php?test=180629_3X_4f6991429e1eef066370d00dfe773740&medianRun=fastest&medianMetric=SpeedIndex)
Between runs we have a diff of 400 ms (or 15%). So we should test run the proxy locally for a couple of days and see what kind of metrics we get.
Comment Actions
I started this again today. I got a new computer and I can use my old running Ubuntu for a couple of days. My plan is to get it up and running early next week.
When I started to prepare and tested it out with a couple of runs, the setup was broken and also there was a lot of variation between metrics and a big difference between first paint and first visual change (the same as we have seen on WPT). I've made a couple of changes + changed so the video is 60 fps (instead of 30fps) and it looks like that decreased the diff between first paint and first visual paint. But I know for sure when I have everything up and running.
Comment Actions
Argh I need to write down the fix for this since I spent many hours to try to solve it:
I've been working on to minimize and understand the difference in firstPaint and firstVisualChange when running on Android. Since we use WebPageReplay and I want it to all "just work" I use the Docker container, and that only works on Linux. I run Ubuntu 16 in Parallels on my other laptop and to be able to do changes locally on Ubuntu, I rebuild the container ...
docker build -t XXX ..
But on my machine I couldn't build because it couldn't find any dependency for the container from archive.ubuntu.com. After some searching I found the fix. Change /etc/default/docker and enable the row:
# DOCKER_OPTS=“--dns 8.8.8.8 --dns 8.8.4.4”
I did that and I could build the container. Hooray I could build the container. But then when running, I started to get error when recording the URL with WebPageReplay (but I didn't understand that it was because of that change). Running the container and testing on desktop worked fine, running with WebPageReplay worked fine, but using WebPageReplay from my Android phone I got an error loading the URL. I made the mistake at the beginning and didn't look so much at the error.
The error I got was uri:chrome-error://chromewebdata/ so after a lot of try and error I started to search for it and I finally found that someone got that error when they had problems with certificates in Chrome on Android. I went through the changes I done through the day and rollbacked the DOCKER_OPTS change (since it was the only network change I've done), restarted Docker and it worked again!
Now I'm back on track and will try to verify the first visual change.
Comment Actions
I got this up and running today. I need to fine-tune the start script a little but more (removing Docker data etc after each run) but I've been able to have it up and running for a couple of hours at least. At the moment we do 5 runs (the blur vertical line is when I turned on devtools.timeline):
The visual metrics isn't super stable so far, I try more runs and try to tune it the coming days.
The CPU metrics looks good (I'm using a Moto G5):
But WAIT! Look at the time spent scripting (this is the Obama page but it is the same for Facebook and Sweden). Its 1700 ms. Then checkout what time we have on AWS running emulate mobile:
290-340 ms! Much much faster than on a real phone.
I'll try to keep it running through the night to check that the phone still works and will try to get it running from my Raspberry Pi.
Comment Actions
I had a go trying out to drive the phone from my Raspberry but got stuck on installing Chromedriver (that you need to drive chrome). Chromedriver doesn't support ARM, Docker containers needs to be built upon ARM base containers. I tried that too but then the container can't install the driver.
Comment Actions
One of the problems was that Browsertime automatically tries to download Chromedriver, but since there's no pre-built on ARM it fails. I've added so it just skips installing on ARM and then I'll try to see if I can get a matching version to work.
Comment Actions
I had it up and running for some time yesterday until my battery run out of power. The next I will try more runs to see if we can get more stable metrics. There was two things, checkout the spike in First Visual Change that correlates to really low time spent in JS:
The next time I'll make sure we store the result data, so I can get back and have look. There's also an increase in all three URLs in rendering time between 17-18 but I haven't stored the result so I don't know the cause.
Comment Actions
This is my setup: On my local machine I run Graphite/Grafana with docker-compose and then I use Ubuntu in Parallels running a simple bash script:
#!/bin/bash RUNS=5 while true do URL=https://en.m.wikipedia.org/wiki/Barack_Obama docker run --privileged -v /dev/bus/usb:/dev/bus/usb -e START_ADB_SERVER=true --cap-add=NET_ADMIN --shm-size=1g --rm -v "$(pwd)":/browsertime -e REPLAY=true -e LATENCY=100 sitespeedio/browsertime-autobuild:navigationScript $URL --resultDir result --chrome.android.package com.android.chrome --xvfb false --chrome.args ignore-certificate-errors-spki-list=PhrPvGIaAMmd29hj8BCZOq096yj7uMpRNHpn5PDxI6I= -n $RUNS --chrome.args user-data-dir=/data/tmp/chrome --chrome.timeline true echo "browsertime.android.obama.SpeedIndex.median" $(cat result/browsertime.json | jq .statistics.visualMetrics.SpeedIndex.median) "`date +%s`" | nc -q0 10.0.1.9 2003 echo "browsertime.android.obama.FirstVisualChange.median" $(cat result/browsertime.json | jq .statistics.visualMetrics.FirstVisualChange.median) "`date +%s`" | nc -q0 10.0.1.9 2003 echo "browsertime.android.obama.cpu.categories.Scripting.median" $(cat result/browsertime.json | jq .statistics.cpu.categories.Scripting.median) "`date +%s`" | nc -q0 10.0.1.9 2003 echo "browsertime.android.obama.cpu.categories.Loading.median" $(cat result/browsertime.json | jq .statistics.cpu.categories.Loading.median) "`date +%s`" | nc -q0 10.0.1.9 2003 echo "browsertime.android.obama.cpu.categories.Rendering.median" $(cat result/browsertime.json | jq .statistics.cpu.categories.Rendering.median) "`date +%s`" | nc -q0 10.0.1.9 2003 echo "browsertime.android.obama.cpu.categories.Painting.median" $(cat result/browsertime.json | jq .statistics.cpu.categories.Painting.median) "`date +%s`" | nc -q0 10.0.1.9 2003 sudo rm -fR result URL=https://en.m.wikipedia.org/wiki/Facebook docker run --privileged -v /dev/bus/usb:/dev/bus/usb -e START_ADB_SERVER=true --cap-add=NET_ADMIN --shm-size=1g --rm -v "$(pwd)":/browsertime -e REPLAY=true -e LATENCY=100 sitespeedio/browsertime-autobuild:navigationScript $URL --resultDir result --chrome.android.package com.android.chrome --xvfb false --chrome.args ignore-certificate-errors-spki-list=PhrPvGIaAMmd29hj8BCZOq096yj7uMpRNHpn5PDxI6I= -n $RUNS --chrome.args user-data-dir=/data/tmp/chrome --chrome.timeline true echo "browsertime.android.facebook.SpeedIndex.median" $(cat result/browsertime.json | jq .statistics.visualMetrics.SpeedIndex.median) "`date +%s`" | nc -q0 10.0.1.9 2003 echo "browsertime.android.facebook.FirstVisualChange.median" $(cat result/browsertime.json | jq .statistics.visualMetrics.FirstVisualChange.median) "`date +%s`" | nc -q0 10.0.1.9 2003 echo "browsertime.android.facebook.cpu.categories.Scripting.median" $(cat result/browsertime.json | jq .statistics.cpu.categories.Scripting.median) "`date +%s`" | nc -q0 10.0.1.9 2003 echo "browsertime.android.facebook.cpu.categories.Loading.median" $(cat result/browsertime.json | jq .statistics.cpu.categories.Loading.median) "`date +%s`" | nc -q0 10.0.1.9 2003 echo "browsertime.android.facebook.cpu.categories.Rendering.median" $(cat result/browsertime.json | jq .statistics.cpu.categories.Rendering.median) "`date +%s`" | nc -q0 10.0.1.9 2003 echo "browsertime.android.facebook.cpu.categories.Painting.median" $(cat result/browsertime.json | jq .statistics.cpu.categories.Painting.median) "`date +%s`" | nc -q0 10.0.1.9 2003 sudo rm -fR result URL=https://en.m.wikipedia.org/wiki/Sweden docker run --privileged -v /dev/bus/usb:/dev/bus/usb -e START_ADB_SERVER=true --cap-add=NET_ADMIN --shm-size=1g --rm -v "$(pwd)":/browsertime -e REPLAY=true -e LATENCY=100 sitespeedio/browsertime-autobuild:navigationScript $URL --resultDir result --chrome.android.package com.android.chrome --xvfb false --chrome.args ignore-certificate-errors-spki-list=PhrPvGIaAMmd29hj8BCZOq096yj7uMpRNHpn5PDxI6I= -n $RUNS --chrome.args user-data-dir=/data/tmp/chrome --chrome.timeline true echo "browsertime.android.sweden.SpeedIndex.median" $(cat result/browsertime.json | jq .statistics.visualMetrics.SpeedIndex.median) "`date +%s`" | nc -q0 10.0.1.9 2003 echo "browsertime.android.sweden.FirstVisualChange.median" $(cat result/browsertime.json | jq .statistics.visualMetrics.FirstVisualChange.median) "`date +%s`" | nc -q0 10.0.1.9 2003 echo "browsertime.android.sweden.cpu.categories.Scripting.median" $(cat result/browsertime.json | jq .statistics.cpu.categories.Scripting.median) "`date +%s`" | nc -q0 10.0.1.9 2003 echo "browsertime.android.sweden.cpu.categories.Loading.median" $(cat result/browsertime.json | jq .statistics.cpu.categories.Loading.median) "`date +%s`" | nc -q0 10.0.1.9 2003 echo "browsertime.android.sweden.cpu.categories.Rendering.median" $(cat result/browsertime.json | jq .statistics.cpu.categories.Rendering.median) "`date +%s`" | nc -q0 10.0.1.9 2003 echo "browsertime.android.sweden.cpu.categories.Painting.median" $(cat result/browsertime.json | jq .statistics.cpu.categories.Painting.median) "`date +%s`" | nc -q0 10.0.1.9 2003 sudo rm -fR result sleep 120 done