We're now reaching crunch time for removing Jessie instances, so in a week's time I'm going to disable this job from jenkins (and remove the executors it runs on).

As far as I can tell this needs to be actioned by the WMF search / query service team.

Change 581055 had a related patch set uploaded (by Jforrester; owner: Jforrester):
[integration/config@master] jjb: Disable 'wikidata-query-rd-release-silent' job until it can be migrated

https://gerrit.wikimedia.org/r/581055

The jessie boxes this tries to run on have now been disabled. You will need to port this job for it to work again, sorry.

A bit of history, the wikidata-query-rdf-release has been build upon the one for refinery. It really "just" do maven release:prepare release:perform but there is a lot more complexity around it to provide an environment to run in (Docker with some java and whatever dependencies) as well as credentials.

Notably the release plugin requires ssh write access to push commits/tag. That was previously done using a ssh-agent and is now down over https.

A question @Mstyles was regarding signing the release with gpg. I don't think the refinery build is signed nor were the wikidata/query/rdf ones. The container does not have gpg and there is apparently no gpg key / credentials in Jenkins. So I guess signing is only enabled for a manual build?

I haven't looked at the difference between the two jobs. But the wikidata/query/rdf one should be able to reuse the same template I have used for Refinery.

I will be happy to drive you through the integration/config and how the Jenkins jobs are generated (or @Jdforrester-WMF might as well).

Change 591522 had a related patch set uploaded (by Mstyles; owner: Mstyles):
[integration/config@master] jjb: migrate wdqs job to Docker

https://gerrit.wikimedia.org/r/591522

Change 591522 merged by jenkins-bot:
[integration/config@master] jjb: migrate wdqs job to Docker

https://gerrit.wikimedia.org/r/591522

Mentioned in SAL (#wikimedia-releng) [2020-04-27T23:33:24Z] <James_F> Zuul: Migrate WDQS jobs to Docker T247123

@hashar the jenkins job failed due to no git auth to push. you can see it here: https://integration.wikimedia.org/ci/job/wikidata-query-rdf-maven-release-docker-wdqs/3/

Change 593298 had a related patch set uploaded (by Hashar; owner: Hashar):
[wikidata/query/rdf@master] Allow developerConnection override

https://gerrit.wikimedia.org/r/593298

The job fails with:

[ERROR] Failed to execute goal org.apache.maven.plugins:maven-release-plugin:3.0.0-M1:prepare (default-cli) on project query-service-parent: Unable to commit files
[ERROR] Provider message:
[ERROR] The git-push command failed.
[ERROR] Command output:
[ERROR] Could not create directory '/nonexistent/.ssh'.
[ERROR] Permission denied (publickey).
[ERROR] fatal: Could not read from remote repository.

The Maven release plugin attempts to push over ssh but there is no such credential. It is not trivial to expose the credentials using ssh-agent. For the analytics/refinery/source repository I instead went to push over https with credentials exposed in a .netrc file (which git can read to get the username and password).

The URL to use is defined in the wikidata/query/rdf pom.xml: project.scm.developerConnection = scm:git:ssh://gerrit.wikimedia.org:29418/wikidata/query/rdf.

We need to be able to override it when invoking Maven which can be done by using a property.

The proposed patch above ( https://gerrit.wikimedia.org/r/#/c/wikidata/query/rdf/+/593298 ) lets us override the url to push to.

The CI job will have to be adjusted to pass -DdeveloperConnection="scm:git:$ZUUL_URL/$ZUUL_PROJECT"

Next we will want to verify that the maven-release-user user in Gerrit has push access.

In T247123#6095073, @hashar wrote:`

Next we will want to verify that the maven-release-user user in Gerrit has push access.

Verified, it has access.

Change 593300 had a related patch set uploaded (by Hashar; owner: Hashar):
[integration/config@master] Override scm url for wikidata/query/rdf

https://gerrit.wikimedia.org/r/593300

Change 593302 had a related patch set uploaded (by Hashar; owner: Hashar):
[integration/config@master] Remove wikidata/query/rdf release job from postmerge

https://gerrit.wikimedia.org/r/593302

Change 593303 had a related patch set uploaded (by Hashar; owner: Hashar):
[integration/config@master] Restore wikidata/query/rdf jobs

https://gerrit.wikimedia.org/r/593303

Change 593302 merged by jenkins-bot:
[integration/config@master] Remove wikidata/query/rdf release job from postmerge

https://gerrit.wikimedia.org/r/593302

Change 593303 merged by jenkins-bot:
[integration/config@master] Restore wikidata/query/rdf jobs

https://gerrit.wikimedia.org/r/593303

Change 593298 abandoned by Hashar:
Allow developerConnection override

Reason:
Let us the maven override directly in the CI job. Thank you Guillaume.

Updated by: https://gerrit.wikimedia.org/r/#/c/integration/config/ /593300/2..3/jjb/wikidata.yaml

https://gerrit.wikimedia.org/r/593298

Change 593298 restored by Hashar:
Allow developerConnection override

Reason:
Turns out we can not outsmart maven and need this property :)

https://gerrit.wikimedia.org/r/593298

Change 593298 merged by jenkins-bot:
[wikidata/query/rdf@master] Allow developerConnection override

https://gerrit.wikimedia.org/r/593298

Change 593300 merged by jenkins-bot:
[integration/config@master] Override scm url for wikidata/query/rdf

https://gerrit.wikimedia.org/r/593300

I know what that is, I forgot to add the archiva deployment profile. I'll put a patch out

Change 594228 had a related patch set uploaded (by Mstyles; owner: Mstyles):
[integration/config@master] jjb: add archiva profile to wdqs deploy job

https://gerrit.wikimedia.org/r/594228

Change 594228 merged by jenkins-bot:
[integration/config@master] jjb: add archiva profile to wdqs deploy job

https://gerrit.wikimedia.org/r/594228

Moved to next error:

[INFO] [ERROR] Failed to execute goal org.apache.maven.plugins:maven-deploy-plugin:3.0.0-M1:deploy (default-deploy) on project query-service-parent: ArtifactDeployerException: Failed to deploy artifacts: Could not transfer artifact org.wikidata.query.rdf:query-service-parent:pom:0.3.25 from/to wikimedia.releases (https://archiva.wikimedia.org/repository/releases/): Failed to transfer file https://archiva.wikimedia.org/repository/releases/org/wikidata/query/rdf/query-service-parent/0.3.25/query-service-parent-0.3.25.pom with status code 401 -> [Help 1]

401 Unauthorized, so there is some issue with the credentials when attempting to upload to archiva. We reuse the same credentials used for Refinery.

Talked to @Gehel and the issue is probably that the archiva credentials that come from analytics have different server ID's than the ones we use. https://github.com/wikimedia/wikimedia-discovery-discovery-parent-pom/blob/master/pom.xml#L917 vs https://github.com/wikimedia/analytics-refinery-source/blob/master/pom.xml#L96. We can either change our pom to match analytics or add a separate credential in jenkins. @Gehel prefers changing our pom to match analytics for uniformity

Change 594280 had a related patch set uploaded (by Mstyles; owner: Mstyles):
[wikimedia/discovery/discovery-parent-pom@master] Rename deploy archiva profile repository Id

https://gerrit.wikimedia.org/r/594280

Change 594280 merged by Gehel:
[wikimedia/discovery/discovery-parent-pom@master] Rename deploy archiva profile repository Id

https://gerrit.wikimedia.org/r/594280

In T247123#6106324, @Mstyles wrote:

Talked to @Gehel and the issue is probably that the archiva credentials that come from analytics have different server ID's than the ones we use. https://github.com/wikimedia/wikimedia-discovery-discovery-parent-pom/blob/master/pom.xml#L917 vs https://github.com/wikimedia/analytics-refinery-source/blob/master/pom.xml#L96. We can either change our pom to match analytics or add a separate credential in jenkins. @Gehel prefers changing our pom to match analytics for uniformity

The credentials are stored inside Jenkins. Then the Maven configuration file is generated on the fly by a Jenkins plugin. Its configuration is restricted but it is available to release engineering folks at https://integration.wikimedia.org/ci/configfiles/editConfig?id=org.jenkinsci.plugins.configfiles.maven.MavenSettingsConfig.ArchivaCredentialsSettings

It lists two maven credentials:

The template for the setting file is:

<?xml version="1.0" encoding="UTF-8"?>
<settings xmlns="http://maven.apache.org/SETTINGS/1.0.0" 
          xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" 
          xsi:schemaLocation="http://maven.apache.org/SETTINGS/1.0.0 http://maven.apache.org/xsd/settings-1.0.0.xsd">
  <servers>
  	<!-- Should be automatically injected from credentials -->          
  </servers>
</settings>

Change 594304 had a related patch set uploaded (by Hashar; owner: Mstyles):
[wikidata/query/rdf@master] upgrading to latest parent pom (1.39)

https://gerrit.wikimedia.org/r/594304

Change 594304 merged by jenkins-bot:
[wikidata/query/rdf@master] upgrading to latest parent pom (1.39)

https://gerrit.wikimedia.org/r/594304

Eventually @Mstyles and I managed to get a build passing! https://integration.wikimedia.org/ci/job/wikidata-query-rdf-maven-release-docker-wdqs/8/console

Gotta verify whether that resulted in a proper release and maybe update whatever releasing doc you might have and point to the new job.

Docs are updated -> https://wikitech.wikimedia.org/wiki/Wikidata_query_service#Production_Deployment and I verified that the zip file from the latest version can be downloaded from Archiva

Change 581055 abandoned by Jforrester:
[integration/config@master] jjb: Disable 'wikidata-query-rd-release-silent' job until it can be migrated

Reason:
This was migrated, happily.

https://gerrit.wikimedia.org/r/581055