Pat Meenan told me he could setup a Moto G4 for our private use to test out, that would be a good idea I think.

Let me summarize a couple of different alternatives:

• We could use WebPageTest and host the phones ourselves. We will still have the hosting problem, but we have control. It should handle both Android and iphone, but the iphone is a little limited (but at least we have it).
• We could re-work the test I did on the Reading offsite a couple of years ago: https://github.com/soulgalore/vs-mobile - that have never got any attention. Drawbacks is that we still host the phones and it will not work on iphone (until you can use the webdriver to run it). And it will need work to update and keep updated. However we will have full control.
• Wallmarts labs has used Sauce labs for performance testing, see https://medium.com/@abhaguptatweet/run-sitespeed-io-on-android-devices-4777b405faaf but I'm not 100% the metrics will be stable enough.

@Imarlier has an idea of other providers we can test out, so let us test them.

It will be really interesting to try out how "stable" metrics we can get. How many runs do we need to do? Can we set the connectivity? Can we use the WebPageReplay proxy (that should work on Android but you need to root the device to install the certificate, but I haven't tried that yet).

Kobiton (which @Gilles already checked out) was one that I was aware of. The other one that I know people using is Sauce Labs (https://saucelabs.com)

Let me try Saucelabs later this week (as I mentioned earliers Wallmarts Labs use SauceLabs but I have never seen any metrics but they say that it is ok).

We should have the device lab link here: http://buildingadevicelab.com/

There are actually only one alternative and that is hosting the mobile phones ourselves. I'll list pro and cons and then we can discuss this on the next offsite.

Hosting our own phones

Hosting the phones means either the office or one of us need to host X phones with a server that is reachable from the internet.

Pros

• We can have super stable metrics as long as we test on an Android phone together with WebPageReplay. We will be able to find small regressions the same way as we can do today on Desktop.
• We can build our own tool on top of what we have today that we could open source => making the mobile performance testing world a little better.
• We can get videos with Visual Metrics out of the box.
• It will be a lot of fun :D

Cons

• Someone needs to take care of the devices (restart if something fails, install new versions of Chrome).
• There are a learning curve setup the devices correctly for testing but http://buildingadevicelab.com/ can be a good resource.
• We need to build a tool (a service) on top of the test tool where we can schedule tests.
• Testing on multiple phones at the same time are untested so far so we don't know how many phones we can have per server.
• How do we scale for testing per commit? How many phones do we need for the current "desktop" testing?

Using third party vendor

We could use third party vendor however there are no vendors that are focused on performance tests. The best we have tested so far is SauceLabs: best in the way that we can use most of Chrome hacks to get performance logs etc. SauceLabs misses a way to set connectivity.

Pros

• Someone else will take of the phones

Cons

• We will not have control of what's installed on the phone (unstable metrics).
• We cannot set connectivity (unstable metrics).
• We are not sure if the videos are usable (can we collect Visual Metrics?).
• The stability in other environments are untested. See the DNS problem I had testing out SauceLabs. Testing with WebPageReplay and a local phone are the safest way to get stable metrics.

I've been testing out to run multiple devices using Docker but from Mac today following the Appium setup: https://github.com/appium/appium-docker-android

I got it to work with one phone so far, need to make some config changes to run ADB on different ports. But the solution does't feel so robust: You need to first create a docker machine, then manually configure it to have access the phones/usb port, log into the machine and from there run the Docker containers ... I'm having a hard time seeing how that would work when we want to automate it.

I will have a go at running in Ubuntu and running ADB on different ports (per instance) and see if that will work.

I tried this some more but I couldn't get that to work, so this is something we need to address if we want to implement something like doing mobile performance tests on commits.