Page MenuHomePhabricator

Android app build: Gradle checkstyle + app build
Closed, ResolvedPublic

Description

We would like to have some improvements to our Android app builds.

Currently we have two build steps in the Jenkins build:

  • tox-flake8
  • maven-checkstyle
  1. gradle-checkstyle:

I would like to keep the first step (tox-flake8), but replace the second (maven-checkstyle) with a gradle-checkstyle since we have moved from Maven to Gradle for our build system a while ago, and I would like to drop the pom.xml files. I think this should make the checkstyle build faster, and I guess you should be able to run it on the same box as the old Maven checkstyle step.

The command to run: (all based from the root of our git repo: https://gerrit.wikimedia.org/r/apps/android/wikipedia)

		./gradlew checkstyle

The XML report can be found in ./wikipedia/build/reports/checkstyle/checkstyle.xml

FYI, as you can see, Gradle is invoked using the Gradle wrapper script, ./gradlew, which resides in our repo. So, no need to install Gradle on the build system.

  1. gradle-apk:

I would like to see a real Android app build using Gradle happening.
We've got a lab system for our Alpha builds, which we can log into, so we can keep the Android SDK up-to-date manually. If you could use this system for Jenkins then you don't need to worry about installing or updating the Android SDK.

Every 30 minutes it checks to see if anything got merged to master. If so, it kicks off a new build. You can see the result of our Alpha build on https://android-builds.wmflabs.org/.

The Android SDK is installed in /srv/adk. You probably need to set ANDROID_HOME to it.
Our Alpha builds currently run via a cronjob (/srv/builds/src/build.py). More env variables are in there that you most likely will need. (Yes, even TERM.) BTW, the scripts are also in https://gerrit.wikimedia.org/r/labs/tools/wikipedia-android-builds.
The user name is android-build. Please contact YuviPanda or other wmflabs if there are access issues. Yuvi set up the current Alpha build on this machine.
Please keep the Alpha setup intact for now. If needed, e.g. due to suspected conflicts/duplicated build efforts, you can comment out the crontab. It's probably best to use a new folder for the working directory that Jenkins would use for the git repo clone.

The command to run:

		./gradlew -q clean assembleAlphaDebug

This will produce an apk file that would be great to make available through Jenkins:

		./wikipedia/build/outputs/apk/wikipedia-alpha-debug.apk
  1. To replace the current alpha build on https://android-builds.wmflabs.org/ please also add a step for builds that get merged to master:

a) copy the apk to /srv/builds/public_html/runs/latest
b) and create a /srv/builds/public_html/runs/latest/meta.json file with similar contents as it currently has.

The alpha app checks this site periodically to see if there is a newer build available. If there is a new version available then it displays a notification on the device. The code is in org.wikipedia.alphaupdater.AlphaUpdateChecker. It checks commit_hash of meta.json. https://android-builds.wmflabs.org/ uses completed_on.
(Just to be clear. We don't want to update the Alpha apk and meta.json for patches that have not been +2'd yet. We only want this for patches that got merged to master.)

(In the future, it would be great to have the tests run on an emulator, but I'll save that for another task.)
Let me know if you have any questions. Thank you!

Event Timeline

bearND raised the priority of this task from to Needs Triage.
bearND updated the task description. (Show Details)

Thanks for all the details, much helpful.

I noticed gradle maintains a cache under the $HOME/.gradle/cache which is great, preventing to having to download the whole internet over and over.

gradle-checkstyle

I gave it a try by manually creating a jenkins job hashar-test-gradle (you can edit it using your labs credentials given your labs account is in the 'wmf' LDAP group which it should. The job is tied to integration-slave1007.eqiad.wmflabs.

On the instance (Trusty), I have installed gradle Debian package. But that version 1.5 complains it can not find jcenter() (build #1. So lets use your wrapper for now.

On a build #3 it whines with:

> SDK location not found. Define location with sdk.dir in the local.properties file
  or with an ANDROID_HOME environment variable.

Android SDK

Seems the checkstyle target depends on Android SDK being available :-/ A couple years or so ago, we installed Android SDK on a single machine and had build occurs there. Nowadays we spread the load on a dozen of labs instance and I am not sure how we can ship them Android SDK.

AFAIK packaging the Android SDK for Debian is stalled / has little interest. We could get:

  1. a copy in git for tracking purposes and use puppet git::clone to have it deployed on all slaves
  2. add it to /data/project which would make it available on all instances. Not sure how it will play with both Precise and Trusty but I guess we can tie the job to be run only on Trusty instances. There might be some performances issues since /data/project is over NFS.
  3. a dedicated instance to tie jobs to on which you can update Android SDK as needed.

gradle-apk

Once we have a job working, it can be made to invoke checkstyle build step or other on every patchset received.

When a patch is merged or a tag is pushed, Gerrit send an event to Zuul which can triggers a different job which would build the clean assembleAlphaDebug target, archive the resulting .apk and publish it somewhere. In Zuul they are the postmerge and publish pipelines). A stripped example for mediawiki/core:

project:
 - name: mediawiki/core
   test:  # on new patchset
    - mediawiki-phpunit-hhvm
   gate-and-submit:  # on Code-Review +2
    - mediawiki-phpunit-hhvm
   postmerge: # after a patch is merged
    # Publish documentation at docs.wikimedia.org
    - mediawiki-core-doxygen-publish
   postmerge: # on tag
    # Publish documentation at docs.wikimedia.org
    - mediawiki-core-doxygen-publish
    # Craft a MediaWiki tarball
    - mediawiki-core-release

So yeah totally doable with our current stack.


In short: the main blocker is figuring out how to get Android SDK on the Jenkins slaves. Copy pasting from above:

  1. a copy in git for tracking purposes and use puppet git::clone to have it deployed on all slaves
  2. add it to /data/project which would make it available on all instances. Not sure how it will play with both Precise and Trusty but I guess we can tie the job to be run only on Trusty instances. There might be some performances issues since /data/project is over NFS.
  3. a dedicated instance to tie jobs to on which you can update Android SDK as needed.

I am tempted to use /data/project :]

Thanks for looking into this, @hashar!

Yes, let's use the Gradle wrapper. I think not just for now but always. It helps us to keep the Gradle version in sync through our app Git repo. And it's so convenient not needing to install a build system.

I agree, the big thing right now is to get the Android SDK onto the Jenkins slave. You're correct. There are no Debian packages for the Android SDK maintained. You can download a tar or zip file from the Android site.

We have a script that does that. You can find it in https://git.wikimedia.org/tree/labs%2Ftools%2Fwikipedia-android-builds/b58a6f53684503fbe80374fdd342b2307683138e/bin. (The bin folder of the Git repo found in https://gerrit.wikimedia.org/r/#/admin/projects/labs/tools/wikipedia-android-builds). It downloads the SDK, which is basically just the SDK Manager. You probably want to customize the folders the download script puts things in.

The second script is the second step: update the SDK. This will actually download the individual SDK packages needed for our app using the SDK Manager (you can invoke it with "android &" from command line). That part will do more downloading. With the command line options you see in the script, we're able to run both scripts headlessly on the wmflabs machine. So, I think those scripts should be a good starting point to get the Android SDK installed on a Jenkins slave. We as Android devs occasionally want to change the list or version of packages to be installed, e.g. the build-tools version or a newer platform than android-21.

Bonjour @hashar. Is there anything I can help with to remove any blockers for this? Would it help to have some Docker images for Android development? There are plenty of Android Docker images available on https://registry.hub.docker.com/search?q=android&searchfield=.
I would probably prefer something that has been updated lately, like https://registry.hub.docker.com/u/jacekmarchwicki/android/. It has build-tools 21.1.2, even an Nexus 5 emulator image.

bearND set Security to None.

I think the recording of Christopher Orr's Android meetup presentation that was held on March 31 at the WMF office is very helpful and inspiring: https://freeflowapp.com/v/q7ql9a. I think the slides he presented were the same as https://chris.orr.me.uk/files/posts/Dutch_AUG_2013-Building-Testing-Deploying-Android-Apps-With-Jenkins.pdf.

The Gradle build is already using the mentioned com.jakewharton.sdkmanager:gradle-plugin, which takes care of installing and keeping the Android SDK components up-to-date to what the build.gradle files need.

You'll maybe still want to use the Gradle Jenkins plugin, but keep in mind that we have our gradlew wrapper script at the root of our repo, see task description above.
Christopher was also using the Android Emulator Jenkins plugin.
This should make setting up a Jenkins build job easy.

UPDATE: Another presentation is https://www.youtube.com/watch?v=xa2Lx0mcdvI. He starts creating the job at the 14 minute mark. I'm sure there are many others. If you get stuck let me know, and I'm happy to help.

I just ran the Android Gradle build on a plain Ubuntu 14 system. The only thing I needed to install was:

sudo apt-get update 
sudo apt-get install gcc-multilib lib32z1 lib32stdc++6
sudo apt-get install openjdk-7-jdk

Then I was able to successfully run:

./gradlew checkstyle
./gradlew -q clean assembleAlphaDebug

I did not have to set any environment variable, nor did I install Gradle or the Android SDK.
The Gradle command line is already included in our Git repo. The Android SDK is automatically set up through the awesome SDK Manager Gradle plugin. So, there's really not much to do except to clone and fetch the latest from our Git repo and run the above commands.

Change 210177 had a related patch set uploaded (by Hashar):
contint: packages for Android SDK

https://gerrit.wikimedia.org/r/210177

Change 210177 had a related patch set uploaded (by Hashar):
contint: packages for Android SDK

https://gerrit.wikimedia.org/r/210177

I have deployed it on the integration puppetmaster. The change will get merged by ops whenever we poke them about it :)

Change 210197 had a related patch set uploaded (by Hashar):
Build Android mobile app with gradlew

https://gerrit.wikimedia.org/r/210197

In short @bearND made it absolutely trivial to build the Android app. The Jenkins job proposed at https://gerrit.wikimedia.org/r/210197 is only a few lines.

Change 210197 merged by jenkins-bot:
Switch Android mobile app from maven to gradlew

https://gerrit.wikimedia.org/r/210197

Following up some discussions with @bearND, there is no two jobs running different goals:

apps-android-wikipedia-gradlew

./gradlew -q clean checkstyle assembleAlphaDebug

Wich craft a checkstyle report.

apps-android-wikipedia-gradlew-lint

./gradlew -q clean lintAlphaDebug

That fails for now and is made non voting. Once it is passing we can move the lint goal to the other job. The lint job captures artifacts under /build/outputs/ , there is apparently a Jenkins plugin to interpret them ( https://wiki.jenkins-ci.org/display/JENKINS/Android+Lint+Plugin ) but the site is down for now and there is no JJB support for it. So that would be for later.

I have send a dummy change https://gerrit.wikimedia.org/r/210289 which yields:

jshint         SUCCESS in 1s
tox-flake8     SUCCESS in 6s
apps-android-wikipedia-gradlew       SUCCESS in 2m 40s
apps-android-wikipedia-gradlew-lint  FAILURE in 3m 03s (non-voting)

@hashar Thank you very much for getting this done.

I think we keep the lint task separate since it takes so long. I don't want the apk build to be delayed so long. Until we've fixed the lint issues I wouldn't mind reducing the amount of time the lint tasks runs (e.g only after +2 or once a day). That's up to you.

wiki.jenkins-ci.org is up again. https://wiki.jenkins-ci.org/display/JENKINS/Android+Lint+Plugin sounds promising.

I think the "long" build time is due to gradle downloading materials. Once cached it is fast and lint is faster than the assemble task.

Here are the build time for assemble and lint:


The spikes are due to the job running on a slave that has no gradle cache yet. When they run on a slave that ran gradle already, it is much faster.

Looking at the build trend for each job, the lint job is slightly faster (40 secs vs 1 min):

https://integration.wikimedia.org/ci/job/apps-android-wikipedia-gradlew/buildTimeTrend
https://integration.wikimedia.org/ci/job/apps-android-wikipedia-gradlew-lint/buildTimeTrend

So I guess we can keep triggering the lint job on every patchset. It is not being triggered on +2 since it is non voting and is never going to prevent a patch to be merged.

wiki.jenkins-ci.org is up again. https://wiki.jenkins-ci.org/display/JENKINS/Android+Lint+Plugin sounds promising.

Indeed! Can you please fill us a task for it? We will need to install it on Jenkins and add support for that plugin to Jenkins Job Builder. We can even hack it together during the hackathon if you like python :]

Change 210177 merged by Dzahn:
contint: packages for Android SDK

https://gerrit.wikimedia.org/r/210177