Page MenuHomePhabricator

Jenkins should run tests for the Wikipedia app before merge
Closed, ResolvedPublic

Description

The maven setup for the Wikipedia app was just finished up (https://gerrit.wikimedia.org/r/#/c/110676/). Now we should setup a jenkins build and test server that does a build and runs the unit tests on it. There's an Android Emulator Plugin for Jenkins (https://wiki.jenkins-ci.org/display/JENKINS/Android+Emulator+Plugin) that seems to do the biggest bit of work.

The same is probably need for the iOS version of the app.

Details

Reference
bz60720

Event Timeline

There are a very large number of changes, so older changes are hidden. Show Older Changes

Proposed configuration of the Android Emulator Jenkins plugin:

Running on headless build machines

Selecting an emulator

  • Run an Android emulator during build
    • Run emulator with properties: Android OS version: 4.4 Screen density: 240 Screen resolution: WVGA Device locale: de_DE

Multi-configuration (matrix) job

This is for later:
Let's skip that for patch set builds. Maybe we can add something like that later, to run once a day.
(Would be nice to cover different Android versions (but for now avoid 2.3 since the emulator has issues there with our WebView.) 3.0 and up to 5.1 should be fine: 3.2 4.0 4.1 4.2 4.3 5.0
Multiple languages would be nice too: es_ES fr_FR zh_CN zh_TW iw_IL ar_001

Installing project prerequisites

Add "Install Android project prerequisites" build step.
This build step requires no configuration — just add it before the build step that compiles your Android projects.

Creating project build files

These instructions seem very old (referring to Ant // we're using Gradle). I think this is probably not needed since this runs after our build is done. (Should we add a Jenkins dependency?)

Installing and uninstalling APKs

Under the "Build" section of your job configuration, select "Add build step" and choose "Install Android package".

**/build/outputs/apk/*-debug.apk

(just pick one, there's usually a wikipedia-alpha-debug.apk and wikipedia-alpha-debug-unaligned.apk)

  • Uninstall existing APK first

Running the Android monkey tool

Might be nice for a once a day build job.

Artifacts

This will be archived automatically as a build artifact, with the name logcat.txt

Nothing to configure, I suppose :)

On Mon, Jun 15, 2015 at 2:33 PM, @hashar wrote:

Hello Bernd,
I will pair with Zeljkof for the Jenkins Android emulator plugin. Maybe we will even craft a small job as a proof of concept.
Writing the JJB plugin is straightforward, someone else even proposed a patch: https://review.openstack.org/#/c/146990/
So in short, let us give it a shot this week and we can setup a quick checkin end of week for a short presentation.

@hashar That's great progress!. Let me know if you need any info from me (in addition to what I specified in T62720#1366119) to get the Jenkins task configured.

@bearND I have followed your detailed instructions somewhere above and crafted a basic Jenkins job: https://integration.wikimedia.org/ci/job/test-T62720-android-emulator/

It is not a matrix job, so it only test a single configuration set, but that is good enough for experimentation.

The job is tied to run on a specific Ubuntu Trusty slave in wmflabs ( integration-slave-trusty-1015 ), this way the job does not roam between instances and thus does not install Android everywhere :-D

I had to set the ABI explicitly to x86. Running the job (build #4) yields an error:

$ tools/emulator -ports 7654,7655 -prop persist.sys.language=de \
 -prop persist.sys.country=DE \
 -avd hudson_de-DE_240_WVGA_android-19_x86 \
 -no-snapshot-load -no-snapshot-save -no-window
emulator: ERROR: x86 emulation currently requires hardware acceleration!
Please ensure KVM is properly installed and usable.
CPU acceleration status: This user doesn't have permissions to use KVM (/dev/kvm).
[android] Emulator did not appear to start; giving up
[android] Stopping Android emulator

That is all for today :-(

Using your wmf labs account, feel free to continue tweaking the job https://integration.wikimedia.org/ci/job/test-T62720-android-emulator/configure but I guess we need some labs work :-(

Thank you @hashar. @Niedzielski is going to take a look.

I gave it a try setting the ABI to armeabi but Build #8 fails as well:

Error: Ignoring unknown package filter 'sys-img-armeabi-android-19'
Warning: The package filter removed all packages. There is nothing to install.
         Please consider trying to update again without a package filter.
[android] Using Android SDK: /mnt/jenkins-workspace/tools/android-sdk
[android] Creating Android AVD: /mnt/home/jenkins-deploy/.android/avd/hudson_de-DE_240_WVGA_android-19_armeabi.avd
[android] /mnt/jenkins-workspace/tools/android-sdk/tools/android create avd -f -a -s WVGA800 -n hudson_de-DE_240_WVGA_android-19_armeabi -t android-19 --abi armeabi
    Error: Invalid --abi armeabi for the selected target.
[android] Could not create Android emulator: Failed to run AVD creation command

Build #9 is with ABI set to armeabi-v7a. Whine about not being able to open audio and the emulator can't be reached so I aborted it.

Build #10 comes with extra options passed to emulator: -verbose -no-audio -no-boot-anim. Cycles with:

00:00:28.237 $ /mnt/jenkins-workspace/tools/android-sdk/platform-tools/adb -s localhost:9511 shell getprop init.svc.bootanim
00:00:28.276 error: device offline
00:00:28.293 $ /mnt/jenkins-workspace/tools/android-sdk/platform-tools/adb connect localhost:9511

netstat shows the emulator64-arm listening on ports 9510 and 9511. There is a bunch of connections established to 127.0.0.1 port 9511 but nothing happens :(

@hashar, hey there! Still reading through the notes but I think we would strongly prefer x86 or x86_64 emulator instances as they're night and day faster than the ARM* variants.

emulator: ERROR: x86 emulation currently requires hardware acceleration!
Please ensure KVM is properly installed and usable.

I believe you're seeing the above error because a QEMU installation is required and / or you're attempting virtualization within a virtual machine, which may not be supported. The second point is a leap as I'm assuming the whole build process in a VM and that the software doesn't support spawning another VM within it.

@hashar, would you mind giving my account modify permissions? When I visit the job configuration, it says "sniedzielski is missing the Job/ExtendedRead permission."

@hashar, would you mind giving my account modify permissions? When I visit the job configuration, it says "sniedzielski is missing the Job/ExtendedRead permission."

I gave you a bunch of permissions, that should be good now. Please be careful :-} Poke me 'hashar' on IRC anytime.

@hashar, thanks! For some reason, I'm still getting the same error, "sniedzielski is missing the Job/ExtendedRead permission" :/

Is that your labs account ? https://integration.wikimedia.org/ci/user/Sniedzielski/ Sniedzielski

Since you are a WMF staff your labs account should be in the ldap groups 'wmf' and 'nda' , the 'wmf' group would grant you full admin rights on jenkins :}

Please fill a request for SRE-Access-Requests so Operations can the groups in LDAP. That should give the appropriate rights.

On https://integration.wikimedia.org/ci/user/Sniedzielski/ , I saved your user configuration, that might have populated a missing field inside Jenkins.

Dzahn added a subscriber: Dzahn.Jun 19 2015, 10:56 PM

Hey,

some mistake happened here in the past. When looking at LDAP i saw your non-WMF user was in the WMF group but the WMF user was not. I fixed that and switched it around as it should have been in the first place.

So now the user with the wikimedia.org email is in WMF but the one with your private email is not anymore.

  • Daniel

@Sniedzielski you should now be able to connect to integration-slave-trusty-1015.eqiad.wmflabs where the Jenkins job is tied.

Workspace is /mnt/jenkins-workspace/workspace/test-T62720-android-emulator

Hey @Dzahn, thanks for hooking me up! It doesn't seem to quite be working yet:

ssh sniedzielski@integration-slave-trusty-1015.eqiad.wmflabs
Permission denied (publickey).

I also tried the "niedzielski" account:

ssh niedzielski@integration-slave-trusty-1015.eqiad.wmflabs
Connection closed by UNKNOWN

Sorry, the previous comment is more appropriate on T103192. Copying...

Wikipedia Zero tests fixed in https://gerrit.wikimedia.org/r/#/c/220326/. This task is now blocked pending T103192.

Hey @hashar! I don't know anything about Puppet but I took a stab at adding the following to the configuration:

  • apt-get install qemu
  • adduser jenkins-deploy kvm

The proposed changes are in this draft patch[0]. Your guidance is much appreciated :)

[0] https://gerrit.wikimedia.org/r/#/c/226237/

The emulator is quite slow to start, seems it is polling usb over and over for a minute or so:

[pid 18444] openat(AT_FDCWD, "/dev/bus/usb", O_RDONLY|O_NONBLOCK|O_LARGEFILE|O_DIRECTORY|O_CLOEXEC) = 9
[pid 18444] getdents64(9, /* 3 entries */, 32768) = 72
[pid 18444] openat(AT_FDCWD, "/dev/bus/usb/001", O_RDONLY|O_NONBLOCK|O_LARGEFILE|O_DIRECTORY|O_CLOEXEC) = 12
[pid 18444] getdents64(12, /* 4 entries */, 32768) = 96
[pid 18444] open("/dev/bus/usb/001/002", O_RDONLY|O_LARGEFILE|O_CLOEXEC) = 16
[pid 18444] read(16, "\22\1\0\2\0\0\0\10'\6\1\0\0\0\1\3\5\1\t\2\"\0\1\1\7\2402\t\4\0\0\1"..., 4096) = 52
[pid 18444] close(16)                   = 0
[pid 18444] open("/dev/bus/usb/001/001", O_RDONLY|O_LARGEFILE|O_CLOEXEC) = 16
[pid 18444] read(16, "\22\1\20\1\t\0\0@k\35\1\0\23\3\3\2\1\1\t\2\31\0\1\1\0\340\0\t\4\0\0\1"..., 4096) = 43
[pid 18444] close(16)                   = 0
[pid 18444] getdents64(12, /* 0 entries */, 32768) = 0
[pid 18444] close(12)                   = 0
[pid 18444] getdents64(9, /* 0 entries */, 32768) = 0
[pid 18444] close(9)                    = 0
[pid 18444] rt_sigprocmask(SIG_BLOCK, [CHLD], [QUIT], 8) = 0
[pid 18444] rt_sigaction(SIGCHLD, NULL, {SIG_DFL, [], 0}, 8) = 0
[pid 18444] rt_sigprocmask(SIG_SETMASK, [QUIT], NULL, 8) = 0
[pid 18444] nanosleep({1, 0}, 0xf73a2ecc) = 0

I have manually applied https://gerrit.wikimedia.org/r/#/c/226237/ on integration-slave-trusty1015.eqiad.wmflabs. ie installed qemu package and added the user jenkins-deploy to kvm.

https://integration.wikimedia.org/ci/job/test-T62720-android-emulator/22/console managed to boot the emulator, but fails with a new error :-)

[android] Emulator is ready for use (took 74 seconds)
[android] Searching for Android projects...
[android] No Android projects found; wont install any dependencies.
[android] Could not find APK file '/mnt/jenkins-workspace/workspace/test-T62720-android-emulator/**/build/outputs/apk/*-debug.apk' to be installed
Build step 'Install Android package' marked build as failure

I checked the workspace, and there is no apk available. In the Jenkins job configuration, I thus added a new build step before the emulator that executes:

./gradlew -q clean assembleAlphaDebug

https://integration.wikimedia.org/ci/job/test-T62720-android-emulator/24/console has a stacktrace because it can't find some softs:

+ ./gradlew -q clean assembleAlphaDebug
/mnt/home/jenkins-deploy/.sign/signing.properties not found

FAILURE: Build failed with an exception.

* What went wrong:
A problem occurred configuring project ':wikipedia'.
> Could not resolve all dependencies for configuration ':wikipedia:_releasesprodDebugCompile'.
   > Could not find com.android.support:appcompat-v7:22.2.1.
     Searched in the following locations:
         file:/mnt/jenkins-workspace/tools/android-sdk/extras/google/m2repository/com/android/support/appcompat-v7/22.2.1/appcompat-v7-22.2.1.pom
         file:/mnt/jenkins-workspace/tools/android-sdk/extras/google/m2repository/com/android/support/appcompat-v7/22.2.1/appcompat-v7-22.2.1.jar
     Required by:
         test-T62720-android-emulator:wikipedia:unspecified
   > Could not find com.android.support:design:22.2.1.
     Searched in the following locations:
         file:/mnt/jenkins-workspace/tools/android-sdk/extras/google/m2repository/com/android/support/design/22.2.1/design-22.2.1.pom
         file:/mnt/jenkins-workspace/tools/android-sdk/extras/google/m2repository/com/android/support/design/22.2.1/design-22.2.1.jar
     Required by:
         test-T62720-android-emulator:wikipedia:unspecified

The emulator has Android OS version: android-22. Might be related?

It takes about 20 seconds to launch headless on my laptop. I confirm it took about a minute to launch on the server but it's difficult to analyze because there are other jobs maxing the cores out according to htop. What was the load on the system when you tested?

ADB will check for devices on USB but AFAIK the emulator shouldn't, although it does use QEMU under the covers. Here's the full output of strace /mnt/jenkins-workspace/tools/android-sdk/tools/emulator @test -no-skin -no-audio -no-window|&grep -i usb on the server for a full boot:

read(4, " ja ottaa k\303\244ytt\303\266\303\266n MTP USB-pr"..., 32768) = 32768
write(3, " ja ottaa k\303\244ytt\303\266\303\266n MTP USB-pr"..., 32768) = 32768
read(4, "/statusbar/StatusBarIconList;\0000L"..., 32768) = 32768
write(3, "/statusbar/StatusBarIconList;\0000L"..., 32768) = 32768
read(3, "UT_HDMI\0AUDIO_DEVICE_OUT_USB_DEV"..., 512) = 512
read(3, "VICE_IN_USB_DEVICE\0AUDIO_DEVICE_"..., 512) = 512
read(3, "evice_claim_interface\0ioctl\0usb_"..., 512) = 512
read(3, "dev/bus/usb\0/dev/bus/usb/%03d\0/d"..., 512) = 512
read(3, "tat_sys_tether_usb.png\0\0\0PK\1\2\n\0\n"..., 512) = 512

And a similar run of strace emulator @22g-nexus-5 -no-skin -no-audio -no-window|&grep -i usb on my system for a full boot:

read(5, "p\303\245 kortet.\0\10\10Formater\0\35\35USB-fej"..., 32768) = 32768
write(4, "p\303\245 kortet.\0\10\10Formater\0\35\35USB-fej"..., 32768) = 32768
read(5, "\340\271\202\340\270\225\340\270\204\340\270\255\340\270\245 MTP USB\0\16*\340\270\227\340\270\224"..., 32768) = 32768
write(4, "\340\271\202\340\270\225\340\270\204\340\270\255\340\270\245 MTP USB\0\16*\340\270\227\340\270\224"..., 32768) = 32768
read(5, "om/android/systemui/statusbar/ph"..., 32768) = 32768
write(4, "om/android/systemui/statusbar/ph"..., 32768) = 32768
read(4, "UT_HDMI\0AUDIO_DEVICE_OUT_USB_DEV"..., 512) = 512
read(4, "VICE_IN_USB_DEVICE\0AUDIO_DEVICE_"..., 512) = 512
read(4, "evice_claim_interface\0ioctl\0usb_"..., 512) = 512
read(4, "dev/bus/usb\0/dev/bus/usb/%03d\0/d"..., 512) = 512
read(4, "t_sys_tether_usb.png\0\0\0PK\1\2\n\0\n\0\0"..., 512) = 512
read(4, "(Landroid/hardware/usb/UsbDevice"..., 512) = 512
read(5, "ceStatusBarFromKeyguard\0\31mForced"..., 512) = 512
read(5, "l;\0\36Landroid/app/StatusBarManage"..., 512) = 512
read(5, "$Landroid/hardware/usb/UsbEndpoi"..., 512) = 512
read(5, "om/android/internal/statusbar/St"..., 512) = 512
read(5, "oid.hardware.usb.IUsbManager\0\36an"..., 512) = 512
read(5, "HORT_TEXT\0$USB_AUDIO_AUTOMATIC_R"..., 512) = 512
read(5, "YPE_MASK\0\26USB_ENDPOINT_XFER_BULK"..., 512) = 512
read(5, "\20UsbEndpoint.java\0\25UsbEndpoint[m"..., 512) = 512
read(5, "sBarBackground\0\16statusBarColor\0\32"..., 512) = 512
read(5, "tic_routing_disabled\0#usb_cd_ins"..., 512) = 512
read(5, "IN_USB_DEVICE_NAME\0\24DEVICE_IN_VO"..., 512) = 512
read(4, "roid.hardware.usb.UsbDevice\nandr"..., 512) = 512
read(4, "un\0/dev/mtp_usb\0android/mtp/MtpS"..., 512) = 512
read(4, "iled for %s\0usb_device_new faile"..., 512) = 512
read(4, "statusBarColor\0\22\22navigationBarCo"..., 512) = 512
read(4, "rmlab_manageUsb\0\22\22permdesc_manag"..., 512) = 512
read(4, "fication_title\0%%usb_storage_sto"..., 512) = 512
read(5, "ndroid.systemui.usb.UsbAccessory"..., 512) = 512
read(5, "r\0\32enforceStatusBarPermission\0\27e"..., 512) = 512
read(5, "d/server/usb/UsbAudioManager;\0@L"..., 512) = 512
read(5, "oid/server/usb/UsbHostManager;\0%"..., 512) = 512
read(5, "evices/virtual/android_usb/andro"..., 512) = 512
read(5, "\0'Landroid/hardware/usb/IUsbMana"..., 512) = 512
read(5, "t thread\0\17UsbService.java\0\22UsbSe"..., 512) = 512
read(5, "mation\0\20mHasUsbAccessory\0\17mHaveC"..., 512) = 512
read(5, "Bar\0\21mStatusBarManager\0\21mStatusB"..., 512) = 512
read(5, "rStatusBar\0%mUndrawnActivitiesBe"..., 512) = 512
read(5, "eUsbNotification\0\16updateUsbState"..., 512) = 512
read(5, "\0\20ODD_PRIME_NUMBER\0\22OEM USB over"..., 512) = 512
read(4, "I)V\0addUsbInterface\0Can't find a"..., 512) = 512
read(4, "e_get_fd\0dup\0usb_device_close\0us"..., 512) = 512
read(5, "\v\0\0\0\"\0\264\tpUSB C\32\1y\203q q\n\1\0\21\0\0\0\n\0\3\0"..., 512) = 512
read(5, "ndroid/hardware/usb/IUsbManager;"..., 512) = 512
read(4, "B-masselager\0\21\21USB er tilsluttet"..., 512) = 512
read(4, "Sl\303\245 USB-lager fra\0svDer opstod "..., 512) = 512
read(4, "p\303\245 kortet.\0\10\10Formater\0\35\35USB-fej"..., 512) = 512
read(4, "Karte bzw. dem USB-Speicher zu s"..., 512) = 512
read(4, "\232\204 SD \345\215\241/USB \345\255\230\345\202\250\350\256\276\345\244\207\344\270\255\357\274\214"..., 512) = 512
read(4, "un error amb la connexi\303\263 USB. A"..., 512) = 512
read(4, "c_usb\0\32\32battery_info_power_unkno"..., 512) = 512
read(4, "ing_turnon_subtext\0\35\35usb_tetheri"..., 512) = 512
read(4, "ser\0\r\rUSB debugging\0  Debug mode"..., 512) = 512
read(4, "magatzematge massiu USB\0\30\30No hi "..., 512) = 512
read(4, "\27\27Error d'ancoratge d'USB\0-/Comp"..., 512) = 512
read(4, "\32\32Fejl ved netdeling via USB\0$$D"..., 512) = 512
read(5, "ettingFromNone\0\3uri\0\3url\0\30usb_ma"..., 512) = 512
read(4, "ra este dispositivo USB.\00077Se us"..., 512) = 512
read(4, "\0')O carregamento por USB n\303\243o \303"..., 512) = 512
read(4, "\250\347\250\213\345\274\217\345\255\230\345\217\226 USB \351\205\215\344\273\266\345\227\216\357\274\237\0\26"..., 512) = 512
read(4, "p\0$$TextAppearance.StatusBar.Sys"..., 512) = 512
read(4, "g_message\0\24\24usb_debugging_always"..., 512) = 512
read(4, ".phone.PhoneStatusBar\0IIwifi,bt,"..., 512) = 512
read(4, " USB\00000Utilitza de manera predet"..., 512) = 512
read(4, "\23\23SD-Karte ausblenden\0\23\25Dateigr\303"..., 512) = 512
read(4, "ng USB?\00000Permanenteng tanggalin"..., 512) = 512
read(4, "orage ng USB\0//Walang SD card na"..., 512) = 512
read(4, "\352\262\240\354\212\265\353\213\210\352\271\214?\0 DUSB \354\240\200\354\236\245\354\206\214\354\227\220"..., 512) = 512
read(4, "\206\330\247\331\203 \331\210\330\255\330\257\330\251 \330\252\330\256\330\262\331\212\331\206 USB\0 7"..., 512) = 512
read(4, "gre\305\241ke u USB memoriji.\0\17\17Nepozn"..., 512) = 512
read(4, "USB bellekten kal\304\261c\304\261 olarak si"..., 512) = 512
read(4, "s z \303\272lo\305\276i\305\241t\304\233 USB?\0%%Trvale v"..., 512) = 512
read(4, "amiento USB\342\200\246\0\r\rReproduciendo\0\17"..., 512) = 512
read(4, "almacenamiento USB.\0\37!La repetic"..., 512) = 512
read(4, "ida.\0\"&J\305\253s\305\263 USB atmintin\304\227je \304"..., 512) = 512
read(4, "\26\31USB-t\303\241rol\303\263 keres\303\251se...\0\r\16Mo"..., 512) = 512
read(4, "\305\221 el\0\30\33Az USB-t\303\241r nem \303\251rhet\305\221"..., 512) = 512
read(4, "\200\320\265\320\272 \"%s\" \321\201 USB-\320\275\320\260\320\272\320\276\320\277\320\270\321\202"..., 512) = 512
read(4, "\272\320\260 USB-\320\275\320\260\320\272\320\276\320\277\320\270\321\202\320\265\320\273\321\217.\0\27-"..., 512) = 512
read(4, "\346\255\243\345\234\250\346\211\253\346\217\217 USB \345\255\230\345\202\250\350\256\276\345\244\207..."..., 512) = 512
read(4, "D \345\215\241\351\224\231\350\257\257\0\n\26USB \345\255\230\345\202\250\350\256\276\345\244\207\351\224"..., 512) = 512
read(4, "cenamiento USB.\0\33\34Tu tarjeta SD "..., 512) = 512
read(4, "a mem\303\263ria de armazenamento USB?"..., 512) = 512
read(4, "\241\340\270\271\340\270\245 USB \340\271\203\340\270\212\340\271\211\340\270\207\340\270\262\340\270\231\340\271"..., 512) = 512
read(4, "\244\225\340\245\207 \340\244\253\340\244\274\340\245\213\340\244\250 \340\244\256\340\245\207\340\244\202 USB"..., 512) = 512
read(4, "ettingFromNone\0\3uri\0\3url\0\30usb_ma"..., 512) = 512
read(5, "entState\0\21allowUsbDebugging\0\7all"..., 512) = 512
read(5, "ED\0002android.hardware.usb.action."..., 512) = 512
read(5, "guardStatusBarIn\0\33animateKeyguar"..., 512) = 512
read(5, "n\0\25handleUsbStateChanged\0\25handle"..., 512) = 512
read(5, "StatusBarScrimView\0\17mStatusBarSt"..., 512) = 512
read(5, "tusbar/ActivatableNotificationVi"..., 512) = 512
read(5, "/statusbar/BaseStatusBar$12;\0000Lc"..., 512) = 512
read(5, "statusbar/BaseStatusBar$5;\0000Lcom"..., 512) = 512
read(5, "usBar$NotificationClicker;\0CLcom"..., 512) = 512
read(5, "ystemui/statusbar/DragDownHelper"..., 512) = 512
read(5, "Lcom/android/systemui/statusbar/"..., 512) = 512
read(5, "d/systemui/statusbar/KeyguardAff"..., 512) = 512
read(5, "stemui/statusbar/NotificationOve"..., 512) = 512
read(5, "/statusbar/ScrimView$2;\0*Lcom/an"..., 512) = 512
read(5, "systemui/statusbar/SignalCluster"..., 512) = 512
read(5, "Lcom/android/systemui/statusbar/"..., 512) = 512
read(5, "tatusbar/phone/KeyguardAffordanc"..., 512) = 512
read(5, "\0?Lcom/android/systemui/statusba"..., 512) = 512
read(5, "usbar/phone/MultiUserSwitch;\0ALc"..., 512) = 512
read(5, "8Lcom/android/systemui/statusbar"..., 512) = 512
read(5, "statusbar/phone/PanelView$2$1;\0002"..., 512) = 512
read(5, "ystemui/statusbar/phone/PhoneSta"..., 512) = 512
read(5, "om/android/systemui/statusbar/ph"..., 512) = 512
read(5, "eStatusBar$35;\0:Lcom/android/sys"..., 512) = 512
read(5, "ar/phone/PhoneStatusBar$42;\0007Lco"..., 512) = 512
read(5, ";\0GLcom/android/systemui/statusb"..., 512) = 512
read(5, "m/android/systemui/statusbar/pho"..., 512) = 512
read(5, "sbar/phone/StatusBarHeaderView$1"..., 512) = 512
read(5, "hone/StatusBarHeaderView$4;\0GLco"..., 512) = 512
read(5, "emui/statusbar/policy/BluetoothC"..., 512) = 512
read(5, "tusbar/policy/BluetoothUtil;\0DLc"..., 512) = 512
read(5, "mui/statusbar/policy/HeadsUpNoti"..., 512) = 512
read(5, "ndroid/systemui/statusbar/policy"..., 512) = 512
read(5, "roid/systemui/statusbar/policy/L"..., 512) = 512
read(5, "droid/systemui/statusbar/stack/S"..., 512) = 512
read(5, "atusbar/policy/UserSwitcherContr"..., 512) = 512
read(5, "id/systemui/statusbar/policy/Net"..., 512) = 512
read(5, "nfirmActivity\0\27UsbConfirmActivit"..., 512) = 512
read(5, "/statusbar/StatusBarIconList;\0000L"..., 512) = 512
read(5, "reshAllStatusBarIcons\0\17refreshAl"..., 512) = 512
read(5, "phoneOn\0\6mState\0\21mStatusBarManag"..., 512) = 512
read(4, "\346\213\251\346\223\215\344\275\234\0\f\36\344\270\272USB\350\256\276\345\244\207\351\200\211\346\213\251\344\270"..., 512) = 512
read(4, "\216\351\207\215\350\257\225\343\200\202\0001\200\207\345\246\202\346\236\234\346\202\250\346\211\223\345\274\200USB"..., 512) = 512
read(4, "ara el dispositivo USB.\0./Lament"..., 512) = 512
read(4, "el almacenamiento USB\0LLSe produ"..., 512) = 512
read(5, "Percent\0\16usagereporting\0\3usb\0\nus"..., 512) = 512
read(5, "i/StatusBarNotifier$Notification"..., 512) = 512
read(5, "tatus\0\24mStatusBarBackground\0\22mSt"..., 512) = 512
read(5, "cessory Mode\0\17USB Normal Mode\0\22U"..., 512) = 512
read(5, "s\0\nmStatusBar\0\24mStatusBarBackgro"..., 512) = 512
read(5, "\0\17setUsbTethering\0\23setUseImmersi"..., 512) = 512
read(5, "usBarBackground\0\rgetDefaultUri\0\24"..., 512) = 512
read(5, "\0\21showSyncStatusBar\0\fshowToHeade"..., 512) = 512

@hashar, thank you again! I feel we're very close.

I played with the job configuration some after you and it seems to functioning properly. However, the Jenkins installation allows for limitless executors which, I believe, is causing commands to time out. For example[0], a simple adb shell whoami command, which should complete in well under a second, instead took 38 seconds. The emulator alone will eat up one core. Is it possible to run these tests on a dedicated Jenkins installation or drop the number of executors down to one?

[0] https://integration.wikimedia.org/ci/job/test-T62720-android-emulator/40/console

Change 226237 merged by Yuvipanda:
Add Android emulation prerequisites

https://gerrit.wikimedia.org/r/226237

Change 226696 had a related patch set uploaded (by Hashar):
contint: fully qualify unix command

https://gerrit.wikimedia.org/r/226696

Change 226698 had a related patch set uploaded (by Hashar):
contint: drop require to a user in LDAP

https://gerrit.wikimedia.org/r/226698

Change 226696 merged by Giuseppe Lavagetto:
contint: fully qualify unix command

https://gerrit.wikimedia.org/r/226696

Change 226698 merged by Giuseppe Lavagetto:
contint: drop require to a user in LDAP

https://gerrit.wikimedia.org/r/226698

It takes about 20 seconds to launch headless on my laptop. I confirm it took about a minute to launch on the server but it's difficult to analyze because there are other jobs maxing the cores out according to htop. What was the load on the system when you tested?
ADB will check for devices on USB but AFAIK the emulator shouldn't, although it does use QEMU under the covers. Here's the full output of strace /mnt/jenkins-workspace/tools/android-sdk/tools/emulator @test -no-skin -no-audio -no-window|&grep -i usb on the server for a full boot:

<snip>

At least it ends up booting. I mentioned the potential USB related slowdown so it can be acted upon eventually :-}

Niedzielski added a comment.EditedJul 24 2015, 4:29 PM

@hashar, right! Booting is good! Slow boot is undesirable but something we can look into optimizing later. The real problem is described in my earlier comment:

I played with the job configuration some after you and it seems to functioning properly. However, the Jenkins installation allows for limitless executors which, I believe, is causing commands to time out. For example[0], a simple adb shell whoami command, which should complete in well under a second, instead took 38 seconds. The emulator alone will eat up one core. Is it possible to run these tests on a dedicated Jenkins installation or drop the number of executors down to one?
[0] https://integration.wikimedia.org/ci/job/test-T62720-android-emulator/40/console

I think the tests aren't running because ADB is timing out. I don't know what to do about this short of limiting the executors or moving the installation to a dedicated machine.

EDIT: Just to be clear, the app doesn't even get built into the emulator is confirmed to have booted successfully. AFAIK, this has nothing to do with slow boot. It's slow response time in general.

I ran the job, and marked the node offline to prevent any more jobs to be assigned to the node. The build result is https://integration.wikimedia.org/ci/job/test-T62720-android-emulator/42/console

While running top, the emulator eats 50-100% of a core and java compilation rest of the CPU.

The command went a bit faster this time:

00:03:52.093 + adb shell whoami
00:03:52.933 /system/bin/sh: whoami: not found
00:03:52.949 + adb shell ls

We can surely set a dedicated instance and/or reduce the number of executors.

@hashar, hm, that does look much faster. As near as I can manage to the server configuration, works fine on my machine. What would be the next steps to get a faster setup?

I would get the job finished first.

What we can do is create a dedicated instance with a single executor. We would want a puppet class that includes the required packages and role::ci::slave::labs::common which adds Java / Jenkins key etc.

From there, the slave can be labelled with something like "AndroidEmulator" and we can get the job tied to that label so it will only run on this slave.

@hashar, I'm a little confused on what to do next. You said we should get the job finished first but you disabled it.

@hashar, I re-ran through the steps on my local machine (Linux Mint). I can't quite reproduce the configuration shown in the console. When I connect locally, I get the following configuration:

$ adb devices
List of devices attached
emulator-7869	device
localhost:7870	offline

The offline status is swapped on the build server. Accounting for the swap locally, the tests run (they do fail some ways in for a different reason) so I think they should run on the server too.

@hashar, I'm a little confused on what to do next. You said we should get the job finished first but you disabled it.

https://integration.wikimedia.org/ci/job/test-T62720-android-emulator/ is still enabled though I noticed the slave it is tied to went offline. I have put it back on.

That unleashed https://integration.wikimedia.org/ci/job/test-T62720-android-emulator/43/

The problem right now the job does not seem to achieve must and/or choke when starting the emulator :-(

I changed the job configuration to run on slaves matching contintLabsSlave && AndroidEmulator since a new one has been added (T107336).

The first run on the new slave is https://integration.wikimedia.org/ci/job/test-T62720-android-emulator/45/console . It is dedicated to jobs having AndroidEmulator label so that is only https://integration.wikimedia.org/ci/job/test-T62720-android-emulator/ for now :-)

Wow, this is really great! How would I ssh into the new machine? I tried the following:

$ ssh sniedzielski@integration-slave-jessie-1003.eqiad.wmflabs
channel 0: open failed: administratively prohibited: open failed
ssh_exchange_identification: Connection closed by remote host

@hashar, friendly bump to review the updated patch[0] when you can.

https://gerrit.wikimedia.org/r/#/c/230260/

hashar added a comment.Sep 3 2015, 1:36 PM

The job has been updated by @Niedzielski \O/

Change 230260 merged by jenkins-bot:
Run tests in apps-android-wikipedia-gradlew

https://gerrit.wikimedia.org/r/230260

@hashar, I think this guy can be closed too. We worked around T110243 for now and I think T107336 is resolved.

hashar closed this task as Resolved.Sep 4 2015, 10:28 AM
hashar added a subscriber: thcipriani.

I removed T110243 as a blocker since there is a workaround.

Congratulations on the Jenkins job / Android Emulator setup. It took time but it is done !

Specific thanks to:
@bearND for the very nice Jenkins/Android emulator setup above
@thcipriani to have setup the slave among others
@Niedzielski To have find the JJB patches and catching up with JJB deployment