@APizzata-WMF: if you have the cycles, we need to investigate this before it becomes a blocker.

The configuration change I mentioned in the other task's comment was triggered by a jjb change to handle the recent removal of mirrors.wm.org.

It doesn't actually seem to me related to the issue after all. What's more, the job hadn't run since March 11th before the first failed build on May 6th; almost two months. So most likely the job actually broke in between and the config change is a red herring.

@xcollazo I was thinking about the gitlab access token you pointed me to earlier. If that is indeed the one your failing job needs then I've been looking at the job config and I think it may not be using that token at all.

From what I can grok, the config for that token gets added here. I believe in your case the token would actually be taken from archiva-credentials.xml since it's a maven run ( I can see archiva-credentials.xml has a gitlab token configured). But if I'm getting right how jjb injects config, it seems the job needs to follow the pattern {name}-maven-release for it to get these credentials added; since the name of your job is analytics-refinery-maven-java8-site-publish it doesn't get the creds config. I just checked the config of one of your other jobs that does follow the pattern, analytics-refinery-maven-release, and I can see that one did get the creds.

Just tried running analytics-refinery-maven-release, it failed with:

14:12:36 [ERROR] Failed to execute goal org.apache.maven.plugins:maven-release-plugin:3.0.1:prepare (default-cli) on project refinery: Unable to commit files
14:12:36 [ERROR] Provider message:
14:12:36 [ERROR] The git-push command failed.
14:12:36 [ERROR] Command output:
14:12:36 [ERROR] To https://gerrit.wikimedia.org/r/analytics/refinery/source
14:12:36 [ERROR]  ! [rejected]        master -> master (fetch first)
14:12:36 [ERROR] error: failed to push some refs to 'https://gerrit.wikimedia.org/r/analytics/refinery/source'
14:12:36 [ERROR] hint: Updates were rejected because the remote contains work that you do
14:12:36 [ERROR] hint: not have locally. This is usually caused by another repository pushing
14:12:36 [ERROR] hint: to the same ref. You may want to first integrate the remote changes
14:12:36 [ERROR] hint: (e.g., 'git pull ...') before pushing again.
14:12:36 [ERROR] hint: See the 'Note about fast-forwards' in 'git push --help' for details.
14:12:36 [ERROR] -> [Help 1]

In T426369#11924739, @jnuche wrote:

...
@xcollazo I was thinking about the gitlab access token you pointed me to earlier. If that is indeed the one your failing job needs then I've been looking at the job config and I think it may not be using that token at all.

From what I can grok, the config for that token gets added here. I believe in your case the token would actually be taken from archiva-credentials.xml since it's a maven run ( I can see archiva-credentials.xml has a gitlab token configured). But if I'm getting right how jjb injects config, it seems the job needs to follow the pattern {name}-maven-release for it to get these credentials added; since the name of your job is analytics-refinery-maven-java8-site-publish it doesn't get the creds config. I just checked the config of one of your other jobs that does follow the pattern, analytics-refinery-maven-release, and I can see that one did get the creds.

Thing is that this repo was working fine and the link with Gitlab for artifact publishing was done a long while ago?

Last successful post-merge build was Mar 11:
patch: 1250525: Update changelog.md for v0.3.13 | https://gerrit.wikimedia.org/r/c/analytics/refinery/source/+/1250525
post-merge build: https://integration.wikimedia.org/ci/job/analytics-refinery-maven-java8-site-publish/130/console

Then on May 6 I updated the .gitignore and it started failing:
patch: 1283113: Add .DS_Store, logs/, and *.bak to .gitignore | https://gerrit.wikimedia.org/r/c/analytics/refinery/source/+/1283113
post-merge build: https://integration.wikimedia.org/ci/job/analytics-refinery-maven-java8-site-publish/131/console

In T426369#11927028, @xcollazo wrote:

In T426369#11924739, @jnuche wrote:

...
@xcollazo I was thinking about the gitlab access token you pointed me to earlier. If that is indeed the one your failing job needs then I've been looking at the job config and I think it may not be using that token at all.

From what I can grok, the config for that token gets added here. I believe in your case the token would actually be taken from archiva-credentials.xml since it's a maven run ( I can see archiva-credentials.xml has a gitlab token configured). But if I'm getting right how jjb injects config, it seems the job needs to follow the pattern {name}-maven-release for it to get these credentials added; since the name of your job is analytics-refinery-maven-java8-site-publish it doesn't get the creds config. I just checked the config of one of your other jobs that does follow the pattern, analytics-refinery-maven-release, and I can see that one did get the creds.

Thing is that this repo was working fine and the link with Gitlab for artifact publishing was done a long while ago?

Yeah, that's true. My best guess right now is that at some point between March 11th and May 6th some configuration somewhere changed and broke the job. Maybe in early March a token wasn't necessary to access to the GitLab repo? Or the wmf-jvm-parent-pom was located somewhere else? or not used at all by the job?

Something telling is that in the failed jobs you can see Downloading from gitlab-maven: https://gitlab.wikimedia.org/api/v4/groups/186/-/packages/maven/org/wikimedia/wmf-jvm-parent-pom/1.96/wmf-jvm-parent-pom-1.96-site_en.xml, whereas in the last successful run wmf-jvm-parent-pom doesn't show up at all. In fact, even though the job was marked as SUCCESS, its console output doesn't look very healthy.

What I can see is that the only credential analytics-refinery-maven-java8-site-publish has currently configured is a SONAR_API_KEY, so it's not surprising that GitLab is responding with a 401. The other job I mentioned, analytics-refinery-maven-release, does have what seems to be the required creds configured (even though it seems it's failing due to an unrelated issue with gerrit?)

I think it could be worth looking into adding the creds to analytics-refinery-maven-java8-site-publish and see if that fixes the issue. Otherwise you're gonna need to find someone who actually knows how these jobs are supposed to be wired.

Just my 2 cents, but I hope it helps.

Just re-ran analytics-refinery-maven-release and it succeeded, I guess my specific problem was transitory

This continues to happen on every merge. It is not a blocker, but it is annoying.

CC @APizzata-WMF, @JAllemandou for awareness.