Upgrade CI emulator to API 25
Closed, ResolvedPublic

Description

Bumping the SDK platform variable in the plugin config to android-25 should Just Work but actually doesn't because of a change in the system image naming conventions beginning in API 25 that isn't accounted for in the plugin. We have have a fork of the upstream repo now, with a patch submitted for review with the needed change. Maybe we can look at getting some needed changes like this merged upstream.

N.B. Reference screenshots may also need updating as a result of the SDK version change.

status update

The apps-android-wikipedia-periodic-test job is now configured to use android-25 to take snapshots. @Mholloway proposed two patches to update the Android Plugin which I have deployed them on the Jenkins master.

https://gerrit.wikimedia.org/r/#/c/366253/ to support a change in the system name introduced by android-25 > Github PR #69

https://gerrit.wikimedia.org/r/#/c/366965/ is the related CI config change to switch to android-25 and change the ABI from x86 to google_apis/x86 (fix found by @Mholloway

https://gerrit.wikimedia.org/r/#/c/366484/ get rid of the -engine classic which is QEMU1. There are a couple side effects:

  • the instance has four CPU and the build time went from ~45 minutes to ~11 minutes.

Github PR #70

That finally address T158014: Investigate how to improve Android CI performance and stability !

The job currently fails due to screenshot differences. That is apparently not related to the platform or QEMU change.

Restricted Application added a subscriber: Aklapper. · View Herald TranscriptNov 13 2016, 10:08 PM
Mholloway raised the priority of this task from Low to Normal.Jul 17 2017, 9:16 PM
Mholloway added a subscriber: Mholloway.

Change 366253 had a related patch set uploaded (by Mholloway; owner: Mholloway):
[integration/jenkinsci/android-emulator-plugin@master] Update system image component name construction for APIs >= 25

https://gerrit.wikimedia.org/r/366253

Mholloway updated the task description. (Show Details)Jul 19 2017, 1:50 PM

T152455 would be handy here.

Mentioned in SAL (#wikimedia-releng) [2017-07-19T14:43:04Z] <hashar> Jenkins: uploaded a patched android-emulator plugin for T150623 and restarting Jenkins

gdb wait-for-devices complained with:

Your emulator is out of date, please update by launching Android Studio:
 - Start Android Studio
 - Select menu "Tools > Android > SDK Manager"
 - Click "SDK Tools" tab
 - Check "Android SDK Tools" checkbox
 - Click "OK"

I have upgraded it with:

$ ssh jenkins-deploy@integration-slave-jessie-android
$ sudo su - jenkins-deploy
$ cd /srv/jenkins-workspace/tools/android-sdk
$ ./tools/android update sdk --no-ui  --filter tool,platform-tool
<snip license>
Installing Archives:
  Preparing to install archives
  Downloading Android SDK Platform-tools, revision 26
  Installing Android SDK Platform-tools, revision 26
    Stopping ADB server succeeded.
    Installed Android SDK Platform-tools, revision 2691%)
    Stopping ADB server succeeded.
    Starting ADB server succeeded.
  Done. 1 package installed.
$

That error is gone.


There is some issue to bring up the emulator though :(

tools/emulator -engine classic -ports 5794,5795 -report-console tcp:5855,max=60 -prop persist.sys.language=en -prop persist.sys.country=US -avd hudson_en-US_240_WVGA_android-25_google_apis-x86 -no-snapshot-load -no-snapshot-save -no-window -no-audio -no-boot-anim
emulator: WARNING: Requested adb port (5795) is outside the recommended range [5555,5586]. ADB may not function properly for the emulator. See -help-port for details.
emulator: Listening for console connections on port: 5794
emulator: Serial number of this emulator (for ADB): emulator-5794
[android] Emulator reported that the console is available on port 5,794
[android] Waiting for emulator to finish booting...
$ /srv/jenkins-workspace/tools/android-sdk/platform-tools/adb -s emulator-5794 wait-for-device shell getprop init.svc.bootanim
ERROR: Timeout after 45000 milliseconds
$ /srv/jenkins-workspace/tools/android-sdk/platform-tools/adb -s emulator-5794 wait-for-device shell getprop init.svc.bootanim
ERROR: Timeout after 45000 milliseconds
...
Mholloway added a comment.EditedJul 19 2017, 4:26 PM

And with the API 25 image we're back to boot timeouts, it seems. That's an unpleasant surprise. Our longstanding timeout issues booting the API 24 emulator were fixed by adding the -no-window command line option to the launch command, but this one doesn't boot even with that flag set.

I tried unchecking "Show emulator window" in the web configuration UI and re-running, but that didn't help.

Change 366908 had a related patch set uploaded (by Hashar; owner: Hashar):
[integration/config@master] android: drop -no-audio

https://gerrit.wikimedia.org/r/366908

Change 366965 had a related patch set uploaded (by Hashar; owner: Hashar):
[integration/config@master] Bump Android periodic test to android-25

https://gerrit.wikimedia.org/r/366965

status update

The apps-android-wikipedia-periodic-test job is now configured to use android-25 to take snapshots. @Mholloway proposed two patches to update the Android Plugin which I have deployed them on the Jenkins master.

https://gerrit.wikimedia.org/r/#/c/366253/ to support a change in the system name introduced by android-25

https://gerrit.wikimedia.org/r/#/c/366965/ is the related CI config change to switch to android-25 and change the ABI from x86 to google_apis/x86 (fix found by @Mholloway

https://gerrit.wikimedia.org/r/#/c/366484/ get rid of the -engine classic which is QEMU1. There are a couple side effects:

  • the instance has four CPU and the build time went from ~45 minutes to ~11 minutes.

That finally address T158014: Investigate how to improve Android CI performance and stability !

The job currently fails due to screenshot differences. That is apparently not related to the platform or QEMU change.

Fix for screenshot differences (some legit, some questionable and needing review, some due to platform discrepancies) is here: https://gerrit.wikimedia.org/r/#/c/366991/.

hashar updated the task description. (Show Details)Jul 26 2017, 11:06 AM

Screenshots need updating again as a result of 31aedab. I'll get to that sometime today.

Change 366253 merged by jenkins-bot:
[integration/jenkinsci/android-emulator-plugin@master] Update system image component name construction for APIs >= 25

https://gerrit.wikimedia.org/r/366253

The job is now running with android-25 and QEMU 2 which make it run tasks in parallel.

We had to resort to forking the Android plugin. The patch are in our Gerrit and merged.

Co-Fixed with @Mholloway \O/

hashar closed this task as Resolved.Jul 26 2017, 3:41 PM

Change 366908 merged by jenkins-bot:
[integration/config@master] android: drop -no-audio

https://gerrit.wikimedia.org/r/366908

Change 366965 merged by jenkins-bot:
[integration/config@master] Bump Android periodic test to android-25

https://gerrit.wikimedia.org/r/366965

Screenshots need updating again as a result of 31aedab. I'll get to that sometime today.

Once the screenshot tests are fast enough, I guess we will run them on every proposed patchsets and they will never regress :-]