Change 338487 had a related patch set uploaded (by Mholloway):
Add current license hashes to repo to ensure auto-downloading in CI

https://gerrit.wikimedia.org/r/338487

Change 338487 merged by Mholloway:
[apps/android/wikipedia] Add current license hashes to repo to ensure auto-downloading in CI

https://gerrit.wikimedia.org/r/338487

Change 341346 had a related patch set uploaded (by mholloway-shell):
[apps/android/wikipedia] [Dev] Upgrade Gradle Plugin to 2.3.0

https://gerrit.wikimedia.org/r/341346

Change 341583 had a related patch set uploaded (by mholloway-shell):
[operations/puppet] [Android] Create symlink to repo licenses dir in the SDK on CI

https://gerrit.wikimedia.org/r/341583

I'm not sure these have been updating correctly in CI even with the Jake Wharton SDK Manager plugin. The build tools seem to be updating as expected, but the SDK tools and platform-tools are very much out of date.

Change 341830 had a related patch set uploaded (by Mholloway):
[apps/android/wikipedia] Hygiene: Remove SDK Manager plugin

https://gerrit.wikimedia.org/r/341830

It appears this plugin is still doing more work than we thought. I'd assumed that the Android Gradle plugin upgrade in https://gerrit.wikimedia.org/r/341346 (which required removing the SDK manager) was causing the lint failure, but it turns out that removing the SDK manager plugin causes the failure even on master.

I guess this means either that the SDK manager has a different copy of the SDK stashed away in some directory that I don't have the right to read on the CI machines, or it's downloading it on demand each time a test is run.

Also interesting that this only causes a failure in the lint job. I've noticed that the Jenkins plugin install-android-sdk step isn't performed for the lint job -- maybe the Jenkins plugin is working better than I think it is, and adding the install-android-sdk step to the lint job (which is appropriate in any case) is a good next step before trying again to remove the SDK manager plugin.

Change 341843 had a related patch set uploaded (by Mholloway):
[integration/config] Add install-android-sdk step to apps-android-wikipedia-lint job

https://gerrit.wikimedia.org/r/341843

Change 341843 merged by jenkins-bot:
[integration/config] Add install-android-sdk step to apps-android-wikipedia-lint job

https://gerrit.wikimedia.org/r/341843

We have something of a conundrum here. On one hand, the Jenkins Android Emulator plugin wiki still recommends using the SDK Manager plugin to manage dependencies[1], and removing it breaks the apps-android-wikipedia-lint job.[2] On the other, the SDK Manager plugin is incompatible with the current version of the Android Gradle plugin,[3] meaning we can't upgrade until we drop it.

[1]

For the Gradle build system, I would recommend including the Android SDK Manager Gradle Plugin in your project. You may have to use JitPack to get the latest version.

--https://wiki.jenkins-ci.org/display/JENKINS/Android+Emulator+Plugin#AndroidEmulatorPlugin-Buildexecution

[2]

FAILURE: Build failed with an exception.

* What went wrong:
A problem occurred configuring project ':app'.
> Failed to notify project evaluation listener.
> SDK location not found. Define location with sdk.dir in the local.properties file or with an ANDROID_HOME environment variable.

Interestingly, the apps-android-wikipedia-test and apps-android-wikipedia-publish workspaces on the CI machines both have a local.properties file containing the path to the SDK, but the apps-android-wikipedia-lint workspace doesn't. It may have been provided in the other workspaces by the SDK Manager plugin, which claims to create it if it's not found, although it's odd that didn't happen for apps-android-wikipedia-lint as well since the job is relying on the SDK Manager plugin to pass.

[3]

FAILURE: Build failed with an exception.

* What went wrong:
A problem occurred configuring project ':app'.
> The "android" command is no longer included in the SDK. Any references to it (e.g. by third-party plugins) should be removed.

I'm sorry if you've already worked through all this but I'm afraid I'm getting these related tasks conflated as an observer and need to start from square one. I took the simple lint case specifically and it seems to work:

git clone https://gerrit.wikimedia.org/r/p/apps/android/wikipedia
cd wikipedia
JAVA_HOME=/usr/lib/jvm/java-8-openjdk-amd64 ANDROID_HOME=$PWD/.android ./gradlew lintAlphaDebug

(Well, it found lints actually.) The changes I made were:

1. Remove apply plugin: 'android-sdk-manager' from app/build.gradle
2. Remove maven { url 'https://jitpack.io' } and classpath 'com.github.JakeWharton:sdk-manager-plugin:1d07987' from build.gradle
3. Upgrade to the Android Gradle Plugin to v2.3.0 in build.gradle: classpath 'com.android.tools.build:gradle:2.3.0'
4. Move the licenses from licenses/ to a new folder in the repo, .android/licenses/

The folder is in /home/sniedzielski/wikipedia of integration-slave-jessie-1002 which I think you can see as super user if you like.

^^^ So we know the builtin Android Gradle Plugin auto-download configuration works fine but we'll need that ANDROID_HOME change you mentioned in one of the patchsets for sure. Can you double check you're in agreement with this and update your outstanding Gerrit patches with any changes needed? We can allow the periodic job to break in CI for a bit as long as the gating jobs keep running. Once all that stuff is merged, let's do a simple command line test like above but this time on the periodic machine, integration-slave-jessie-android.eqiad.wmflabs. If that works, the problem is likely confined to Jenkins and our Jenkins Job Builder configuration and we can fiddle around in a test job for a while then propagate any changes to the JJB config.

Thanks for looking into this, @Niedzielski!

In T147099#3086193, @Niedzielski wrote:

4. Move the licenses from licenses/ to a new folder in the repo, .android/licenses/

The folder is in /home/sniedzielski/wikipedia of integration-slave-jessie-1002 which I think you can see as super user if you like.

I can't see your home folder since I don't have sudo on the CI machines (it would have made working these things out a lot easier, for sure... :[ )

Ideally, setting up the licenses link and setting $ANDROID_HOME would be puppetized. That's easy enough for the periodic job since it gets its own special puppet role (role::ci::slave::android), and I have a patch up to add the symlinking to that role here,[1] but integration-slave-jessie-100[1,2] just get generic CI roles and get needed Android stuff installed via their Jenkins job definitions. Maybe the Android-specific CI role could be added to integration-slave-jessie-100[1,2], as well?

I'll look over the outstanding patches tomorrow and update as needed.

[1] https://gerrit.wikimedia.org/r/#/c/341583/

So sudo whoami fails? :/

I'm not sure about the Puppet stuff offhandedly but that sounds fine. The + of keeping everything in the app folder is everything is self contained, no Puppet config with symlinks is needed, wiping out the workspace takes the tools with it, and it's a lot easier and safer to test. The downsides are if you scrap the workspace every build, you'll have to redownload the tools every time along with the code, and if you have multiple workspaces, you'll have multiple copies of the tools. I don't believe we rm -rf the workspace each time so I'm actually in favor of storing things in app_repo_dir/.android instead of ~/.android or somewhere else centralized.

In T147099#3086676, @Niedzielski wrote:

So sudo whoami fails? :/

Yep, Sorry, user mholloway-shell is not allowed to execute '/usr/bin/whoami' as root on integration-slave-jessie-1002.integration.eqiad.wmflabs. I think I can just bug Reedy for it if I really need it, though. I've been trying to play things on the conservative side.

I'm not sure about the Puppet stuff offhandedly but that sounds fine. The + of keeping everything in the app folder is everything is self contained, no Puppet config with symlinks is needed, wiping out the workspace takes the tools with it, and it's a lot easier and safer to test. The downsides are if you scrap the workspace every build, you'll have to redownload the tools every time along with the code, and if you have multiple workspaces, you'll have multiple copies of the tools. I don't believe we rm -rf the workspace each time so I'm actually in favor of storing things in app_repo_dir/.android instead of ~/.android or somewhere else centralized.

That makes sense. Yeah, thankfully, the workspace is retained rather than scrapped with each build. Let's plan on keeping everything in the app repo, then. That should be much easier to maintain. I'll update my patches.

OTOH, the downside is that we'd then have three copies of the SDK on
integration-slave-jessie-100[1,2], one for each testing workspace (apps-android-wikipedia-test, apps-android-wikipedia-lint, apps-android-wikipedia-publish)...

@Mholloway, in my opinion, even if this cost us an extra several gigabytes on each machine, if there weren't any other downsides, the benefits would outweigh the costs

\o/ It works! https://gerrit.wikimedia.org/r/#/c/341830/

Each downloaded SDK is currently only 641M. That seems reasonable.

mholloway-shell@integration-slave-jessie-1001:/srv/jenkins-workspace/workspace/apps-android-wikipedia-test$ du -sh .sdk/
641M	.sdk/

Eventually there might come a point where we'll want to go in and do some manual pruning. AFAIK the plugin doesn't clean up older stuff after itself.

Change 341583 abandoned by Mholloway:
[Android] Create symlink to repo licenses dir in the SDK on CI

Reason:
Going in a different direction with this. (T147099)

https://gerrit.wikimedia.org/r/341583

Change 341830 merged by jenkins-bot:
[apps/android/wikipedia] Hygiene: Remove SDK Manager plugin

https://gerrit.wikimedia.org/r/341830

Change 341346 merged by jenkins-bot:
[apps/android/wikipedia] [Dev] Upgrade Gradle Plugin to 2.3.0

https://gerrit.wikimedia.org/r/341346