Page MenuHomePhabricator

Test performance on real mobile devices
Open, LowPublic

Description

One area that we need to improve is to start test performance on real devices. Today we have some tests running on the free WebPageTest instance (testing on Android and iOS) but the problem there is that we can only run a couple of tests and we hit different mobile phones different times. In the long run we want to test commits on real devices, meaning test PR + master on the same device and check for performance changes.

There's is a short term goal and a long term. Lets start with the long term:

  • Every code change should automatically be tested on a real device.

Short term goal:

  • Make it easy to for a engineer to test if specific changes have an impact on performance on real devices.

There are some interesting reads on how Facebook does it and a video.

What it means is that we need to have a device lab in house (= devices that we as engineers can use automatically). Or could this be something that we buy?

There's a couple of different ways to do it:

  • WebPageTest, it support real devices but do we need to run Windows?
  • Selendroid that is used by Ebay and Facebook.
  • Appium
  • Could STF be something for us?
  • vs-mobile - I've made a POC during the reading offsite.

The plan is to postpone this for now (2016) and see if there's a off the shelf solution that we can use in the future. When we have time we can start looking into testing only on Android (since it is much simpler, see https://github.com/soulgalore/vs-mobile).

See also:

Event Timeline

Peter created this task.Jun 2 2016, 5:33 PM
Restricted Application added subscribers: Zppix, Aklapper. · View Herald TranscriptJun 2 2016, 5:33 PM
Krinkle added a subscriber: Krinkle.Jun 2 2016, 7:24 PM
ori moved this task from Inbox to Radar on the Performance-Team board.Jun 6 2016, 5:59 PM
ori moved this task from Radar to Backlog: Small & Maintenance on the Performance-Team board.
Peter updated the task description. (Show Details)Jun 7 2016, 10:54 AM
Peter updated the task description. (Show Details)Jun 9 2016, 10:33 AM

@Peter This is done now, right?

Gilles triaged this task as Low priority.Dec 7 2016, 5:41 PM
Peter added a comment.Dec 13 2016, 7:52 AM

@Krinkle only in WebPageTest, the instances we get for free this is more setting up our own infrastructure.

Peter updated the task description. (Show Details)Dec 13 2016, 8:21 AM
Peter added a comment.Jun 14 2017, 7:28 PM

I would like to have higher prio on this one. Using the free ones on WPT doesn't give us anything. Either we should do it ourselves or find a service we can use,