Page MenuHomePhabricator

search platform maven projects failing post merge build
Closed, ResolvedPublic

Description

Some of the maven project from the search platform team started failing their post-merge build with:

10:55:49 [ERROR] Failed to execute goal org.apache.maven.plugins:maven-assembly-plugin:3.1.0:single (default) on project experimental-highlighter-elasticsearch-plugin: Failed to create assembly: Error adding file 'org.wikimedia.search.highlighter:experimental-highlighter-core:jar:6.4.2-SNAPSHOT' to archive: /src/experimental-highlighter-core/target/classes isn't a file. -> [Help 1]

This was observed after upgrading to discovery-parent-pom:1.20, but rolling back to 1.18 does not fix the issue, so probably unrelated.

The post merge build runs mvn clean install site site:stage.

The site and default lifecycles are not really meant to be run in the same execution. And running them separately seems to fix the issue, at least locally. So a better expression of that post-merge build should be: mvn clean install && mvn site site:stage.

Not entirely related, but during this investigation, I was wondering if the same version of maven was used on CI and locally. It seems to be the case, but only accidentally. Since all the search platform projects use maven-wrapper, which freezes the Maven version, it would be even better to not rely on the Maven version deployed on CI, but instead to run ./mvnw clean install && ./mvnw site site:stage.

Event Timeline

Gehel triaged this task as Medium priority.Nov 1 2018, 1:24 PM
Gehel added a subscriber: hashar.

That change is probably reasonably easy to implement in the JJB templates, but I'm lost there. I hope @hashar can provide some support.

TJones subscribed.

Thanks for hunting this down, @Gehel!

A bit more context about the Docker container docker-registry.wikimedia.org/releng/java8.

Maven 3.5.2 is fetched from apache.org

The container sets:

ENV MAVEN_BIN=/opt/apache-maven-3.5.2/bin/mvn

It has a wrapper around mvn specially to enforce --batch-mode when needed and to rewrite some file path and pass it as an argument to --file. It runs $MAVEN_BIN.

I have a couple hacks in mind:

  • a quick one is to have Zuul to set a parameter: MAVEN_BIN=./mvnw
  • a probably better one is to have the mvn wrapper in the container to always use mvnw when it exists.

The job runs a single container which is being passed the goals: clean install site site:stage. We should be able to split the goals and reuse the states between two runs, though I would prefer to avoid that.

Why not drop the wrapper completely and just allow running an arbitrary command in the project directory? That way, we'll be able to configure ./mvnw clean install && ./mvnw site site:stage (or whatever we need). And not care about transient state needed to be propagated between containers. I'm not sure what value is that wrapper bringing, so I might be missing something.

Change 472452 had a related patch set uploaded (by Hashar; owner: Hashar):
[integration/config@master] Decouple maven install and site stages

https://gerrit.wikimedia.org/r/472452

Mentioned in SAL (#wikimedia-releng) [2018-11-08T14:30:49Z] <hashar> updated search-highlighter-maven-java8-docker-site-publish to decouple install and site goals | T208496

We have updated the publish job for search/highlighter, I have manually rebuild the last build (which was for https://gerrit.wikimedia.org/r/#/c/470610/ ) and the outcome is a success:

https://integration.wikimedia.org/ci/job/search-highlighter-maven-java8-docker-site-publish/16/console
https://doc.wikimedia.org/search-highlighter/

If that works properly, I will refresh the other jobs, check them and then we can merge the ci config change :)

The build succeeded. The new docs are published at https://doc.wikimedia.org/search-highlighter/experimental/, the publication date has been updated, and the site looks fine (minus the usual broken links).

Looks good now :)

Change 472452 merged by jenkins-bot:
[integration/config@master] Decouple maven install and site stages

https://gerrit.wikimedia.org/r/472452