| hashar | |
| Nov 23 2018, 1:14 PM |
| F31700254: mvnrelease.png | |
| Mar 24 2020, 4:01 PM |
For analytics/refinery/source, we have a couple job to automatize releases to Archiva. They should be migrated to Docker containers.
Potentially, we could use the same mechanism for other maven based repositories whic @Gehel suggested a while back.
| Status | Subtype | Assigned | Task | ||
|---|---|---|---|---|---|
| Resolved | MoritzMuehlenhoff | T241719 Migrate remaining self-hosted puppet masters to Puppet 5 / facter 3 | |||
| Resolved | Jdforrester-WMF | T236576 Move all Wikimedia CI (WMCS integration project) instances from jessie to stretch | |||
| Resolved | Jdforrester-WMF | T225031 On CI Jenkins, audit worker labels and remove unused ones | |||
| Resolved | Jdforrester-WMF | T239981 Delete Jenkins label DebianJessie | |||
| Resolved | hashar | T210271 Migrate analytics/refinery/source release jobs to Docker |
The job has to be refactored and generalized so we can use it on other maven repositories. mwdumper is one such use case ( T213874#4883586 ).
I haven't started looking at it yet, I am dealing with some other tasks on Continuous-Integration-Infrastructure (Slipway) .
For clarity, this needs the owner of the job (not sure which bit of Analytics?) to commit to re-writing it into stretch- or buster-compatible code running inside a Docker container on our CI infrastructure, or being OK with us deleting it and them running it somewhere else.
Ah I don't think we were aware there was action on our part, sorry! Moving this out of radar for re-grooming.
@JAllemandou I guess we should pair on it? :) I am not familiar with those jobs though but it should not be too complicated. A potential interesting outcome is the work can potentially be reused for other maven based repositories.
The jessie boxes this tries to run on have now been disabled. You will need to port this job for it to work again, sorry.
Thanks @Jdforrester-WMF for the warning.
@hashar I'm sorry for missing your ping :) I'm interested in pairing on that indeed! I also think the job shouldn't be too complicated.
Let's see how we can organize on this next week.
@Jdforrester-WMF ok...seems a bit harsh! This job is essential for weekly deployments of Hadoop jobs. I know this task has been around for a while, but AFAIK Analytics Eng was given no timeline or urgency about it. @Nuria @JAllemandou we need do this ASAP.
+1 :)
I guess my main trouble is that I have not much idea as to what this job is achieving or what we could do to fit it in a Docker container. But a few pairing should be able to solve that.
The jessie boxes this tries to run on have now been disabled. You will need to port this job for it to work again, sorry.
Sorry but we need to have those jobs enabled again. We got no warning this work was needed and without them, to be clear, we cannot deploy any software to hadoop, They are pretty critical. Let's please restore those jobs and be all aware that we need much better communication about changes such us these, the best time for us to pick up a project we were not counting on is not when a significant part of the team has a reduced schedule.
cc @Jdforrester-WMF cc @greg
That's not true, sorry.
https://www.mediawiki.org/wiki/Scrum_of_scrums/2020-02-05#Analytics
https://www.mediawiki.org/wiki/Scrum_of_scrums/2020-02-12#Analytics
https://www.mediawiki.org/wiki/Scrum_of_scrums/2020-02-19#Analytics
Also mentioned in IRC repeatedly by me (and mentioned at All Hands).
and without them, to be clear, we cannot deploy any software to hadoop, They are pretty critical. Let's please restore those jobs and be all aware that we need much better communication about changes such us these, the best time for us to pick up a project we were not counting on is not when a significant part of the team has a reduced schedule.
I can re-enable to boxes for now, but it'll be brief; they were meant to be deleted at the end of December and we negotiated an extension. The deadline is set by WMCS, not us.
Sounds like a communication breakdown! Do any Analytics Engineers go to SoS? Not sure, I don't think so. None of those links mention a deadline. We were aware of this task however, it would have been nice to have a deadline mentioned and negotiated here.
Sorry. Negotiated in TechMgmt and reflected in grandparent task, T236576. Fair point that Analytics have other things to focus on than the intracacies of SRE/WMCS decommissioning work. :-)
Hey ya'll,
tl;dr: Sorry about the miscommunication. The deprecated instance will be restored but we need to move quickly to migrate it.
Next things for us to prevent this happening again: how can we make sure that communication voids don't happen in the future? We normally use Scrum of Scrums to let other teams (eg SRE, CPT, Product teams) know that we're waiting on them (or when they're waiting on us). Looks like in this case potentially just pinging on the task itself would have been preferred.
Let's work on a time to pair on this (see @hashar's comments above) in the near future.
The deprecated instance will be restored but we need to move quickly to migrate it.
Thanks you. What is the timeframe?
Next things for us to prevent this happening again: how can we make sure that communication voids don't happen in the future?
Since there is a task we will have expected a deadline to be communicated on the task rather than elsewhere, let's proceed going forward assuming that communication about issues happens in phab.
From the call with @JAllemandou : we have investigated into what the job is doing and made good progress.
Maven does most of the magic via release:perform and release:prepare.
The job requires a ssh credential in order to push commits to Gerrit. In Jenkins its stored in the credential store and exposed to the build using the ssh-agent plugin (which spawns a ssh-agent for us). We can keep the agent on the host and expose the agent socket into the container by using a volume mount:
docker run --rm -it \ -v $(SSH_AUTH_SOCK):/ssh-agent \ --env SSH_AUTH_SOCK=/ssh-agent \ docker-registry.wikimedia.org/releng/java8:latest release:perform
For the Archiva credentials, they are stored in the Jenkins configfiles plugin which fetch credentials from the credentials plugin and forge a maven settings.xml file holding those credentials. Again, we can expose that file from the host to the container via a volume mount and use mvn --setting to have maven to read it. If we configure the Jenkins plugin to write the file to archiva-credentials/settings.xml we can thus do:
docker run \ -v "$WORKSPACE/archiva-credentials":/archiva-credentials \ docker-registry.wikimedia.org/releng/java8:latest release:perform -s /archiva-credentials/settings.xml
Additional notes:
Change 583092 had a related patch set uploaded (by Hashar; owner: Hashar):
[integration/config@master] jjb: migrate refinery release job to Docker
The whole process is extensively described at https://wikitech.wikimedia.org/wiki/Analytics/Systems/Cluster/Deploy/Refinery-source
The way the release is done is via the Jenkins Maven Plugin which ask for a release and development version. They have to be manually filed https://integration.wikimedia.org/ci/job/analytics-refinery-release/m2release/ which then trigger the build with three extra parameters. For the last build:
| MVN_RELEASE_VERSION | 0.0.119 |
| MVN_DEV_VERSION | 0.0.120-SNAPSHOT |
| MVN_ISDRYRUN | (checkbox) |
From the grandparent task:
"All instances in the integration project need to upgrade as soon as possible. Instances not upgraded by 2019-12-31 may be subject to deletion unless prior arrangements for an extended deadline has been approved by the Cloud VPS administration team."
That ^ was from WMCS. But we negotiated an extension until the end of q3. If we ask nicely again we *might* be able to get another extension, but truly, this must be done ASAP.
One nit: On the screen above, we tick the Specify custom SCM tag box and modify the default value by hand
I have noticed that in the refinery documentation:
Change refinery-x.y.z to vx.y.z in the "SCM tag" input text-box and update the number. Example: refinery-0.0.40 is bad, v0.0.40 is good
Looks like we should be able to save you that manual step. The plugin (since version 2.2.0, refinery uses 2.5.1) has the option tagNameFormat to change the tag format!
https://maven.apache.org/maven-release/maven-release-plugin/prepare-mojo.html
tagNameFormat:
Format to use when generating the tag name if none is specified. Property interpolation is performed on the tag, but in order to ensure that the interpolation occurs during release, you must use @{...} to reference the properties rather than ${...}. The following properties are available:
groupId or project.groupId - The groupId of the root project.
artifactId or project.artifactId - The artifactId of the root project.
version or project.version - The release version of the root project.Type: java.lang.String
Since: 2.2.0
Required: No
User Property: tagNameFormat
Default: @{project.artifactId}-@{project.version}
So I guess we can change it:
- @{project.artifactId}-@{project.version} + v@{project.version}
Bah forget me, the refinery pom.xml already has the proper configuration. I guess the Jenkins Maven Release plugin does not recognize that option and always offer the default of @{project.artifactId}-@{project.version}.
I guess we can add build parameters MVN_RELEASE_VERSION and MVN_DEV_VERSION to allow one to override the version if need be, but otherwise let them unset and let maven figure out the next version.
Change 583356 had a related patch set uploaded (by Hashar; owner: Hashar):
[integration/config@master] docker: bump java and add ssh to java8
Change 583356 merged by jenkins-bot:
[integration/config@master] docker: bump java and add ssh to java8
Change 583374 had a related patch set uploaded (by Hashar; owner: Hashar):
[integration/config@master] dockerfiles: java8: add ssh host key for Gerrit
Change 583374 merged by jenkins-bot:
[integration/config@master] dockerfiles: java8: add ssh host key for Gerrit
I have tried the job, and exposing the SSH agent socket inside the container does not work. The agent runs on the host as the jenkins-slave user and is only readable by that user. When Docker volume mount the file inside the container, it caries the UID and file permissions. The container has the process running as nobody and it can not access the socket ...
Change 583392 had a related patch set uploaded (by Hashar; owner: Hashar):
[operations/puppet@production] contint: add acl package for file permissions tweak
So I have played a bit more with the job. Sharing the ssh agent from the host into the container does not work: the agent checks whether the client connecting has the same uid and refuses the connection otherwise:
Start an agent in foreground with debug mode:
jenkins$ ssh-agent -a agent.sock -d
Load a key and grant read/write access to the nobody user:
jenkins$ SSH_AUTH_SOCK=agent.sock ssh-add id_rsa jenkins$ setfacl -m user:65534:rw agent.sock
As the nobody user, the identity can not be fetched:
nobody$ SSH_AUTH_SOCK=agent.sock ssh-add -l error fetching identities for protocol 2: communication with agent failed The agent has no identities.
The debug log in the ssh-agent shows:
uid mismatch: peer euid 65534 != uid 2947
:-(
Change 583392 abandoned by Hashar:
contint: add acl package for file permissions tweak
Change 583392 restored by Hashar:
contint: add acl package for file permissions tweak
Reason:
Turns out I actually need setfacl :)
Instead of using ssh-agent and doing Gerrit access over ssh, I have:
Jenkins then creates a temporary file that has that content and exposes the filename in the NETRC_FILE environment variable. Then it is all about using:
docker run -v "$NETRC_FILE":/nonexistent/.netrc
The credentials for archiva is in the config file provider and is owned by the Jenkins user with only user and group access. To grant access to the nobody user I went with applying a custom file ACL to grant read access to the nobody user, then bind mount it:
setfacl -m user:65534:r archiva-credentials.xml docker run -v archiva-credentials.xml:/archiva-credentials.xml
And surprisingly, it works! The first build managing to run maven is https://integration.wikimedia.org/ci/job/analytics-refinery-maven-release-docker/31/console though it fails for some reason.
Change 583683 had a related patch set uploaded (by Hashar; owner: Hashar):
[integration/config@master] docker: java8 should checkout in a branch
Change 583683 merged by jenkins-bot:
[integration/config@master] docker: java8 should checkout in a branch
I have instructed Maven release plugin to use https with:
mvn -Dproject.scm.developerConnection=scm:git:https://maven-release-user@gerrit.wikimedia.org/r/analytics/refinery/source
The build itself seems to work until release:prepare attempts to push. The console log shows:
[INFO] Checking in modified POMs... [INFO] Executing: /bin/sh -c cd /src && git add -- pom.xml refinery-core/pom.xml refinery-spark/pom.xml refinery-tools/pom.xml refinery-hive/pom.xml refinery-job/pom.xml refinery-camus/pom.xml refinery-cassandra/pom.xml [INFO] Working directory: /src [INFO] Executing: /bin/sh -c cd /src && git rev-parse --show-toplevel [INFO] Working directory: /src [INFO] Executing: /bin/sh -c cd /src && git status --porcelain . [INFO] Working directory: /src [INFO] Executing: /bin/sh -c cd /src && git commit --verbose -F /tmp/maven-scm-140771911.commit pom.xml refinery-core/pom.xml refinery-spark/pom.xml refinery-tools/pom.xml refinery-hive/pom.xml refinery-job/pom.xml refinery-camus/pom.xml refinery-cassandra/pom.xml [INFO] Working directory: /src [INFO] Executing: /bin/sh -c cd /src && git symbolic-ref HEAD [INFO] Working directory: /src [INFO] Executing: /bin/sh -c cd /src && git push ssh://maven-release-user@gerrit.wikimedia.org:29418/analytics/refinery/source refs/heads/master:refs/heads/master [INFO] Working directory: /src [INFO] ------------------------------------------------------------------------
I went with a terrible attempt at being able to override the property https://gerrit.wikimedia.org/r/#/c/analytics/refinery/source/+/583714/
Finally a successful build! https://integration.wikimedia.org/ci/job/analytics-refinery-maven-release-docker/38/console
I have built it using the unmerged change https://gerrit.wikimedia.org/r/#/c/analytics/refinery/source/+/583714/ and maven crafted its commits on top of it, made a v0.0.120 tag and pushed. That has caused Gerrit to consider my pending change to be merged and to close the change.
But at least the job seems to be working now!
Change 584923 had a related patch set uploaded (by Hashar; owner: Hashar):
[analytics/refinery/source@master] document -DdeveloperConnection
The job seems to be working fine. @JAllemandou is going to use the job this week and confirms everything works as expected.
The doc at https://wikitech.wikimedia.org/wiki/Analytics/Systems/Cluster/Deploy/Refinery-source will need some minor updates. Notably:
Change 583392 merged by Dzahn:
[operations/puppet@production] contint: add acl package for file permissions tweak
Change 584984 had a related patch set uploaded (by Hashar; owner: Hashar):
[analytics/refinery@master] Skip fetching commit hook when it is already present
Change 584998 had a related patch set uploaded (by Hashar; owner: Hashar):
[integration/config@master] docker: container for refinery jar updater
A couple of things I 'd like to add:
I would urge to avoid the above pattern and instead push the result of the build to a different repo, that is not under CI to mitigate point 1 above. I would also recommend to populate a file and environmental variables with the credentials instead of the ssh-agent trickery to mitigate points 2,3,4,5 above.
Change 584923 merged by jenkins-bot:
[analytics/refinery/source@master] document -DdeveloperConnection
The previous CI job did use a ssh-agent spawned by the Jenkins agent and loaded with credentials. Until I found out ssh-agent checks the uid of the client that is what the commit message reflects in https://gerrit.wikimedia.org/r/#/c/operations/puppet/+/585038/ I then went with a netrc file provided by Jenkins and uses setfacl to make it readable by the container and push over ssh.
That being said, all your points equally apply to that netrc file. It is readable by the executed process.
The release itself is being done by the Maven release plugin via the release:prepare and release:perform goals. They craft a first commit to set the version in pom.xml and create a git tag (5fe8cb57c90ed28b4179d7caf561cc5143fec69d) and then a second commit for the next version ( 98fddef2b57a21c03d6d1a0dd7038ea9cb99afcb ).
I guess we could look at passing parameter to have it not do any git push operation then use another container or job that would have the writable credentials. And thus promote from there. It is still not ideal though, another build might still have access to the credentials :-/
I am having trouble figuring out from the commit message of that change how this is related (it talks about CSP headers and video). Am I missing something?
Also, should I take from what you say above that ssh-agent sharing is no longer in use? That is, the comment of https://gerrit.wikimedia.org/r/#/c/583392/ is no longer applicable? (despite the content of the change still applying).
I then went with a netrc file provided by Jenkins and uses setfacl to make it readable by the container and push over ssh.
/me curious. Which programs use setfacl specifically? Got a link to a change?
That being said, all your points equally apply to that netrc file. It is readable by the executed process.
Not really, or at least not all of my points. A .netrc file is not a gateway that allows communication between the container and the host in the same way that a bind mounted socket is (not to mention that it can be mounted readonly as well). In fact, if the ssh-agent is no longer used, half of my points are no longer applicable.
The release itself is being done by the Maven release plugin via the release:prepare and release:perform goals. They craft a first commit to set the version in pom.xml and create a git tag (5fe8cb57c90ed28b4179d7caf561cc5143fec69d) and then a second commit for the next version ( 98fddef2b57a21c03d6d1a0dd7038ea9cb99afcb ).
I guess we could look at passing parameter to have it not do any git push operation then use another container or job that would have the writable credentials. And thus promote from there. It is still not ideal though, another build might still have access to the credentials :-/
My issue isn't with the credentials leaking so much but rather with the fact those credentials can send commits to the git repo itself (at least that's my reading of the commits you posted above, let me know if I am wrong). That approach creates a feedback loop between CI and the git repo and as such allows attacks with the potential to pollute the repo and execute arbitrary code in multiple places, including the CI infrastructure. A simple workaround is to have it create changes that are reviewed by a human and merged with their approval. That workaround would also mitigate the impact in case those credentials leaked (as a human would have to approve repo modifying actions).
Mentioned in SAL (#wikimedia-releng) [2020-04-02T13:49:41Z] <hashar> Add Jenkins bot wmf-insecte to #wikimedia-analytics # T210271
Change 585497 had a related patch set uploaded (by Hashar; owner: Hashar):
[integration/config@master] jjb: irc notification for analytics jobs
Change 584984 merged by Joal:
[analytics/refinery@master] Fetch commit hook over https and skip if already present
refinery-source-release job has been successful today. Our deployment doc is updated :)
Still refinery-update-jars to go and we're done!
Change 583092 merged by jenkins-bot:
[integration/config@master] jjb: migrate refinery release job to Docker
Change 585497 merged by jenkins-bot:
[integration/config@master] jjb: irc notification for analytics jobs
A couple patches in the refinery repo got merged earlier today.
I still have to test the container ( https://gerrit.wikimedia.org/r/#/c/integration/config/+/584998/ ) and craft the associated JJB job. I have been side tracked with other duties today though.
Change 589589 had a related patch set uploaded (by Hashar; owner: Hashar):
[integration/config@master] Port analytics-update-jars to Docker
Change 584998 merged by jenkins-bot:
[integration/config@master] docker: container for refinery jar updater
https://gerrit.wikimedia.org/r/#/c/integration/config/+/589589/ is the port of update-jars script to Docker.
Change 589589 merged by jenkins-bot:
[integration/config@master] Port analytics-update-jars to Docker
Paired with Joseph and the last bit has been completed a minute or so ago (switch the update-jars job).
We can probably reuse a lot of that logic for other maven based repositories.