@hashar Hi! I could use some help on this task - I've been trying to use the Maven release Jenkins plugin - https://wiki.jenkins-ci.org/display/JENKINS/M2+Release+Plugin to see if I can automate releases. This plugin is enabled in our setup for maven projects, so I tried following the README(which is pitiful) and set up the options. I set this job up - all on the UI - https://integration.wikimedia.org/ci/job/analytics-release-test, and tried to build it giving it required zuul parameters. It finishes but marks Failure for Build Refinery Jobs - I'm not sure why - and doesn't try to launch a release. The plugin has a env variable IS_M2RELEASEBUILD to denote if this build should trigger a release, and I tried using a pre build option to export is as true.

It would be great if you could take a look. This is step 1 - if it attempts to release - it will probably fail needing Archiva credentials - need to figure that out, and then figure out if the release plugin stuff can only be configured using the Jenkins UI or there is a way to translate it to JJB config.

Tagging releng team here. I'm happy to do the work for this task - and it would be great if someone from releng could help - since this hasn't been done before, and also the release plugin is not currently supported by JJB as far as I know. Thanks!

From a quick conversation with Nuria / Madhumitha , I have poked the Release Engineering internal mailing list to raise attention to this.

From a quick look, looks like @madhuvishy aced the Jenkins configuration (kudos really). There is some random failure to determine which might be either in the code or due to some CI infrastructure oddity :(

I made some progress yesterday. Trying to recollect the bumps I hit, and how they were solved here:

1. The build job was failing - was fixed by specifying maven goals (clean package)

2. It was not attempting to release - Had to use the Perform Maven Release option on the UI to trigger a release - It then started trying to do release:perform

3. Wasn't picking up the release branch, was failing with

Failed to execute goal org.apache.maven.plugins:maven-release-plugin:2.5.1:prepare (default-cli) on project refinery: An error is occurred in the checkin process: Exception while executing SCM command. Detecting the current branch failed: fatal: ref HEAD is not a symbolic ref -> [Help 1]

Fixed by - Unchecking shallow clone (not sure if this helped, need to test), Changed Branch Specifier in SCM section to $ZUUL_BRANCH from $ZUUL_COMMIT, and added advanced option Checkout/merge to local branch - $ZUUL_BRANCH (The last two were required to get past this)

4. Unable to push to gerrit - and failing with

[INFO] Executing: /bin/sh -c cd /mnt/jenkins-workspace/workspace/analytics-release-test && git push ssh://jenkins-deploy@gerrit.wikimedia.org:29418/analytics/refinery/source refs/heads/release:refs/heads/release
...
[ERROR] Failed to execute goal org.apache.maven.plugins:maven-release-plugin:2.5.1:prepare (default-cli) on project refinery: Unable to commit files
[ERROR] Provider message:
[ERROR] The git-push command failed.
[ERROR] Command output:
[ERROR] Host key verification failed.
[ERROR] fatal: Could not read from remote repository.
[ERROR]
[ERROR] Please make sure you have the correct access rights
[ERROR] and the repository exists.
[ERROR] -> [Help 1]

Reason is that jenkins-deploy user cannot push to gerrit - Made a test user (ldaptestaccount123, shell username: testaccount123), and added this user via SSH-Agent config on gerrit by supplying the private key. Also added -Duser.name options to maven release commands to pass the right user.

5. The right push url started being picked up but continued to fail with the repository access error - Reason: the jenkins slaves don't trust the gerrit host (https://phabricator.wikimedia.org/P2857). Working on a puppet patch to fix that - https://gerrit.wikimedia.org/r/#/c/281706/

Continuing to push through this (with lots of help from @hashar and @dduvall) - will keep task updated with progress.

More progress!

6. Ran into some issues while adding the gerrit ssh key to /etc/ssh/ssh_known_hosts on the Jenkins slaves - they are described on the patch here - https://gerrit.wikimedia.org/r/#/c/281706/ - solved with some ops helped. @dduvall helped cherry pick and test patch on the integration puppetmaster, and once it worked, got it merged - cherry pick has been removed and changes rebased in the puppetmaster at this point. Jenkins slaves now trust gerrit!

7. Gerrit rejected the commit push because the commit author and the push user were not the same. (testaccount123 belongs to chase)

23:35:30 [ERROR] remote: ERROR: In commit 465fbace14e1d82da0909160dd9e662f5c30f005
23:35:30 [ERROR] remote: ERROR: committer email address Jenkins (Wikimedia)
23:35:30 [ERROR] remote: ERROR: does not match your user account.
23:35:30 [ERROR] remote: ERROR:
23:35:30 [ERROR] remote: ERROR: The following addresses are currently registered:
23:35:30 [ERROR] remote: ERROR: chase.mp@gmail.com
23:35:30 [ERROR] remote: ERROR:
23:35:30 [ERROR] remote: ERROR: To register an email address, please visit:
23:35:30 [ERROR] remote: ERROR: https://gerrit.wikimedia.org/r/#/settings/contact
23:35:30 [ERROR] remote:

Fixed by specifying user.name and user.email from the Jenkins job config under Additional SCM options -> Specify user name and email.

8. Maven release:prepare now pushes the preparing for release commit! - https://github.com/wikimedia/analytics-refinery-source/commit/5f53b9f806a39dfbc2d603cde7e0323bff924e75

9. It now fails saying it cannot push a tag - which baffles me - it can push a commit but not a tag? what sorcery

02:24:46 [INFO] Working directory: /mnt/jenkins-workspace/workspace/analytics-release-test
02:24:46 [INFO] Executing: /bin/sh -c cd /mnt/jenkins-workspace/workspace/analytics-release-test && git commit --verbose -F /tmp/maven-scm-1951409183.commit pom.xml refinery-core/pom.xml refinery-tools/pom.xml refinery-hive/pom.xml refinery-camus/pom.xml refinery-job/pom.xml refinery-cassandra/pom.xml
02:24:46 [INFO] Working directory: /mnt/jenkins-workspace/workspace/analytics-release-test
02:24:46 [INFO] Executing: /bin/sh -c cd /mnt/jenkins-workspace/workspace/analytics-release-test && git symbolic-ref HEAD
02:24:46 [INFO] Working directory: /mnt/jenkins-workspace/workspace/analytics-release-test
02:24:46 [INFO] Executing: /bin/sh -c cd /mnt/jenkins-workspace/workspace/analytics-release-test && git push ssh://testaccount123@gerrit.wikimedia.org:29418/analytics/refinery/source refs/heads/release:refs/heads/release
02:24:46 [INFO] Working directory: /mnt/jenkins-workspace/workspace/analytics-release-test
02:24:47 [INFO] Tagging release with the label v0.0.29...
02:24:47 [INFO] Executing: /bin/sh -c cd /mnt/jenkins-workspace/workspace/analytics-release-test && git tag -F /tmp/maven-scm-1052121632.commit v0.0.29
02:24:47 [INFO] Working directory: /mnt/jenkins-workspace/workspace/analytics-release-test
02:24:47 [INFO] Executing: /bin/sh -c cd /mnt/jenkins-workspace/workspace/analytics-release-test && git push ssh://testaccount123@gerrit.wikimedia.org:29418/analytics/refinery/source refs/tags/v0.0.29
02:24:47 [INFO] Working directory: /mnt/jenkins-workspace/workspace/analytics-release-test
02:24:47 [INFO] ------------------------------------------------------------------------
02:24:47 [INFO] Reactor Summary:
...
02:24:47 [INFO] ------------------------------------------------------------------------
02:24:47 [INFO] BUILD FAILURE
02:24:47 [INFO] ------------------------------------------------------------------------
02:24:47 [INFO] Total time: 2:15.758s
02:24:47 [INFO] Finished at: Wed Apr 06 02:24:47 UTC 2016
02:24:48 [INFO] Final Memory: 14M/221M
02:24:48 [INFO] ------------------------------------------------------------------------
02:24:48 Waiting for Jenkins to finish collecting data
02:24:48 [ERROR] Failed to execute goal org.apache.maven.plugins:maven-release-plugin:2.5.1:prepare (default-cli) on project refinery: Unable to tag SCM
02:24:48 [ERROR] Provider message:
02:24:48 [ERROR] The git-push command failed.
02:24:48 [ERROR] Command output:
02:24:48 [ERROR] remote:
02:24:48 [ERROR] remote: Processing changes: refs: 1
02:24:48 [ERROR] remote: Processing changes: refs: 1, done
02:24:48 [ERROR] To ssh://testaccount123@gerrit.wikimedia.org:29418/analytics/refinery/source
02:24:48 ha:AAAAYB+LCAAAAAAAAP9b85aBtbiIQSmjNKU4P0+vJLE4u1gvPjexLDVPzxdEuhYV5Rf55ZekOlc7RKnPKH7IxMBQUcQgBdWQnJ9XnJ+TqucMoUEKGSCAEaSwAACsNFCqYAAAAA==[ERROR] ! [remote rejected] v0.0.29 -> v0.0.29 (prohibited by Gerrit)
02:24:48 [ERROR] error: failed to push some refs to 'ssh://testaccount123@gerrit.wikimedia.org:29418/analytics/refinery/source'
02:24:48 ha:AAAAYB+LCAAAAAAAAP9b85aBtbiIQSmjNKU4P0+vJLE4u1gvPjexLDVPzxdEuhYV5Rf55ZekOlc7RKnPKH7IxMBQUcQgBdWQnJ9XnJ+TqucMoUEKGSCAEaSwAACsNFCqYAAAAA==[ERROR] -> [Help 1]

I have removed the Ldaptestaccount123 user from the Gerrit Analytics-devs group since the password has been made public here. https://gerrit.wikimedia.org/r/#/admin/groups/833,members

Pushing tag is a different permission in Gerrit , should be tweak able via https://gerrit.wikimedia.org/r/#/admin/projects/analytics/refinery/source,access . The Ldaptestaccount123 would need permission to push tags.

The account is listed with chase email:

That was a workaround for:

Gerrit rejected the commit push because the commit author and the push user were not the same. (testaccount123 belongs to chase)
23:35:30 [ERROR] remote: ERROR: In commit 465fbace14e1d82da0909160dd9e662f5c30f005
23:35:30 [ERROR] remote: ERROR: committer email address Jenkins (Wikimedia)
23:35:30 [ERROR] remote: ERROR: does not match your user account.
23:35:30 [ERROR] remote: ERROR:
23:35:30 [ERROR] remote: ERROR: The following addresses are currently registered:
23:35:30 [ERROR] remote: ERROR: chase.mp@gmail.com
23:35:30 [ERROR] remote: ERROR:
23:35:30 [ERROR] remote: ERROR: To register an email address, please visit:
23:35:30 [ERROR] remote: ERROR: https://gerrit.wikimedia.org/r/#/settings/contact
23:35:30 [ERROR] remote:
Fixed by specifying user.name and user.email from the Jenkins job config under Additional SCM options -> Specify user name and email.

The reason is by default Gerrit prevent you from pushing patchsets for which you are not the author. The user doing the push would need in Gerrit to be granted the permissions:

• Forge Author Identity
• Forge Committer Identity

To have it a bit more secure we would need to have the credentials in the Jenkins credential store ( https://integration.wikimedia.org/ci/credential-store/ ) and then bind the credentials to the job which would get them injected as environment variables solely to that job.

For pushing one can either:

• push over https, would need to generate a token in Gerrit settings of the user
• push over ssh: generate a SSH key pair, add the pub on in Gerrit settings and the private one in Jenkins credential store.

We did something similar to hold credentials for the browser tests which have to login various wikis.

Spoke about this on irc already but leaving it here - Only the username of the test user was public, not the password. The commit author and push user confict arose because it was using the jenkins-deploy user to make the commits (which was default git user.name and user.email) and finally trying to push as the testaccount123 user - it was resolved by setting the user.name and user.email from the additional SCM options. The private key of test user was loaded via SSH Agent to make pushing commits possible - these are already available only via the credential store.

Need to figure out how to add permissions to the user for pushing tags. The credential store will come in handy for the next step - where we have to configure archiva username and password.

More things!

11. It failed saying it couldn't push a tag because the test user needed Push Annotated Tag permissions in Gerrit. Added this and it was able to push tags. It then failed with

16:42:55 [INFO] [ERROR] Failed to execute goal org.apache.maven.plugins:maven-deploy-plugin:2.7:deploy (default-deploy) on project refinery: Failed to deploy artifacts: Could not transfer artifact org.wikimedia.analytics.refinery:refinery:pom:0.0.29 from/to archiva.releases (https://archiva.wikimedia.org/repository/releases/): Failed to transfer file:
https://archiva.wikimedia.org/repository/releases/org/wikimedia/analytics/refinery/refinery/0.0.29/refinery-0.0.29.pom. Return code is: 401, ReasonPhrase:Unauthorized. -> [Help 1]

which makes sense - it doesn't have credentials!

12. Needed to figure out a way to supply archiva credentials to the job - for releases from our local machines these are provided through ~/.m2/settings.xml. Installed the Config File provider plugin that lets you do this - See: T131958: Jenkins: Install Config File Provider Plugin. Followed the instructions to provide a maven settings file and added the credentials for archiva deploy using the Jenkins credential store. Configured the analytics-release-test job to use this settings.xml file.

Voila: https://integration.wikimedia.org/ci/job/analytics-release-test/38/. An actual release!!! :D

Going to call this spike done, and make new tasks to translate all of this to yaml in integration/config, and other things like making a release user.

well done, @madhuvishy! (and @dduvall!)