|Resolved||MoritzMuehlenhoff||T241719 Migrate remaining self-hosted puppet masters to Puppet 5 / facter 3|
|Resolved||Jdforrester-WMF||T236576 Move all Wikimedia CI (WMCS integration project) instances from jessie to stretch|
|Resolved||Jdforrester-WMF||T225031 On CI Jenkins, audit worker labels and remove unused ones|
|Resolved||Jdforrester-WMF||T239981 Delete Jenkins label DebianJessie|
|Resolved||Mstyles||T247123 Migrate wikidata-query-rdf-release-silent release job to Docker|
A bit of history, the wikidata-query-rdf-release has been build upon the one for refinery. It really "just" do maven release:prepare release:perform but there is a lot more complexity around it to provide an environment to run in (Docker with some java and whatever dependencies) as well as credentials.
Notably the release plugin requires ssh write access to push commits/tag. That was previously done using a ssh-agent and is now down over https.
A question @Mstyles was regarding signing the release with gpg. I don't think the refinery build is signed nor were the wikidata/query/rdf ones. The container does not have gpg and there is apparently no gpg key / credentials in Jenkins. So I guess signing is only enabled for a manual build?
I haven't looked at the difference between the two jobs. But the wikidata/query/rdf one should be able to reuse the same template I have used for Refinery.
I will be happy to drive you through the integration/config and how the Jenkins jobs are generated (or @Jdforrester-WMF might as well).
The job fails with:
[ERROR] Failed to execute goal org.apache.maven.plugins:maven-release-plugin:3.0.0-M1:prepare (default-cli) on project query-service-parent: Unable to commit files [ERROR] Provider message: [ERROR] The git-push command failed. [ERROR] Command output: [ERROR] Could not create directory '/nonexistent/.ssh'. [ERROR] Permission denied (publickey). [ERROR] fatal: Could not read from remote repository.
The Maven release plugin attempts to push over ssh but there is no such credential. It is not trivial to expose the credentials using ssh-agent. For the analytics/refinery/source repository I instead went to push over https with credentials exposed in a .netrc file (which git can read to get the username and password).
The URL to use is defined in the wikidata/query/rdf pom.xml: project.scm.developerConnection = scm:git:ssh://gerrit.wikimedia.org:29418/wikidata/query/rdf.
We need to be able to override it when invoking Maven which can be done by using a property.
The proposed patch above ( https://gerrit.wikimedia.org/r/#/c/wikidata/query/rdf/+/593298 ) lets us override the url to push to.
The CI job will have to be adjusted to pass -DdeveloperConnection="scm:git:$ZUUL_URL/$ZUUL_PROJECT"
Next we will want to verify that the maven-release-user user in Gerrit has push access.
Change 593298 abandoned by Hashar:
Allow developerConnection override
Let us the maven override directly in the CI job. Thank you Guillaume.
Updated by: https://gerrit.wikimedia.org/r/#/c/integration/config/ /593300/2..3/jjb/wikidata.yaml
I ran the job again https://integration.wikimedia.org/ci/job/wikidata-query-rdf-maven-release-docker-wdqs/6/console and it now fails with an entirely new error:
Failed to execute goal org.apache.maven.plugins:maven-deploy-plugin:3.0.0-M1:deploy (default-deploy) on project query-service-parent: ArtifactDeployerException: Failed to deploy artifacts: Could not transfer artifact org.wikidata.query.rdf:query-service-parent:pom:0.3.24 from/to ossrh (https://oss.sonatype.org/service/local/staging/deploy/maven2/): Failed to transfer file https://oss.sonatype.org/service/local/staging/deploy/maven2/org/wikidata/query/rdf/query-service-parent/0.3.24/query-service-parent-0.3.24.pom with status code 502 -> [Help 1]
Moved to next error:
[INFO] [ERROR] Failed to execute goal org.apache.maven.plugins:maven-deploy-plugin:3.0.0-M1:deploy (default-deploy) on project query-service-parent: ArtifactDeployerException: Failed to deploy artifacts: Could not transfer artifact org.wikidata.query.rdf:query-service-parent:pom:0.3.25 from/to wikimedia.releases (https://archiva.wikimedia.org/repository/releases/): Failed to transfer file https://archiva.wikimedia.org/repository/releases/org/wikidata/query/rdf/query-service-parent/0.3.25/query-service-parent-0.3.25.pom with status code 401 -> [Help 1]
401 Unauthorized, so there is some issue with the credentials when attempting to upload to archiva. We reuse the same credentials used for Refinery.
Talked to @Gehel and the issue is probably that the archiva credentials that come from analytics have different server ID's than the ones we use. https://github.com/wikimedia/wikimedia-discovery-discovery-parent-pom/blob/master/pom.xml#L917 vs https://github.com/wikimedia/analytics-refinery-source/blob/master/pom.xml#L96. We can either change our pom to match analytics or add a separate credential in jenkins. @Gehel prefers changing our pom to match analytics for uniformity
The credentials are stored inside Jenkins. Then the Maven configuration file is generated on the fly by a Jenkins plugin. Its configuration is restricted but it is available to release engineering folks at https://integration.wikimedia.org/ci/configfiles/editConfig?id=org.jenkinsci.plugins.configfiles.maven.MavenSettingsConfig.ArchivaCredentialsSettings
It lists two maven credentials:
The template for the setting file is:
<?xml version="1.0" encoding="UTF-8"?> <settings xmlns="http://maven.apache.org/SETTINGS/1.0.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:schemaLocation="http://maven.apache.org/SETTINGS/1.0.0 http://maven.apache.org/xsd/settings-1.0.0.xsd"> <servers> <!-- Should be automatically injected from credentials --> </servers> </settings>
Eventually @Mstyles and I managed to get a build passing! https://integration.wikimedia.org/ci/job/wikidata-query-rdf-maven-release-docker-wdqs/8/console
Gotta verify whether that resulted in a proper release and maybe update whatever releasing doc you might have and point to the new job.
Docs are updated -> https://wikitech.wikimedia.org/wiki/Wikidata_query_service#Production_Deployment and I verified that the zip file from the latest version can be downloaded from Archiva