One area that we need to improve is to start test performance on real devices. Today we have some tests running on the free WebPageTest instance (testing on Android and iOS) but the problem there is that we can only run a couple of tests and we hit different mobile phones different times. In the long run we want to test commits on real devices, meaning test PR + master on the same device and check for performance changes.
There's is a short term goal and a long term. Lets start with the long term:
- Every code change should automatically be tested on a real device.
Short term goal:
- Make it easy to for a engineer to test if specific changes have an impact on performance on real devices.
There are some interesting reads on how Facebook does it and a video.
What it means is that we need to have a device lab in house (= devices that we as engineers can use automatically). Or could this be something that we buy?
There's a couple of different ways to do it:
- WebPageTest, it support real devices but do we need to run Windows?
- Selendroid that is used by Ebay and Facebook.
- Appium
- Could STF be something for us?
- vs-mobile - I've made a POC during the reading offsite.
The plan is to postpone this for now (2016) and see if there's a off the shelf solution that we can use in the future. When we have time we can start looking into testing only on Android (since it is much simpler, see https://github.com/soulgalore/vs-mobile).
See also: