Page MenuHomePhabricator
Create Task
Maniphest T301886

Upload required datahub dependencies to Archiva
Closed, ResolvedPublic
Actions

Description

Based on discussion with @elukey and feedback on this commit, we think it would be a good idea to use Archiva to host the Java dependencies for datahub.

Referring to the instructions here: https://wikitech.wikimedia.org/wiki/Archiva#Uploading_dependency_artifacts

We should make available the following:

  • jetty-runner-9.4.20.v20190813.jar
  • jetty-jmx-9.4.20.v20190813.jar
  • jetty-util-9.4.20.v20190813.jar
  • jmx_prometheus_javaagent-0.16.1.jar

Other (e.g. more recent) versions might also work, but these are the versions of the files currently specified in the official datahub Dockerfiles.

Related Objects
Search...

Event Timeline

BTullis created this task.Feb 16 2022, 4:13 PM
Restricted Application removed a project: Patch-For-Review. · View Herald TranscriptFeb 16 2022, 4:13 PM
BTullis triaged this task as High priority.Feb 16 2022, 4:14 PM
BTullis moved this task from Incoming (new tickets) to Security & Governance on the Data-Engineering board.
BTullis moved this task from Backlog to In Progress on the Data-Catalog board.
BTullis moved this task from Next Up to In Progress on the Data-Engineering-Kanban board.
BTullis added a comment.Feb 16 2022, 5:01 PM

I have downloaded all of the jetty artifacts from Maven central

btullis@marlin-wsl:~/tmp$ wget -q https://repo1.maven.org/maven2/org/eclipse/jetty/jetty-runner/9.4.20.v20190813/jetty-runner-9.4.20.v20190813.pom
btullis@marlin-wsl:~/tmp$ wget -q https://repo1.maven.org/maven2/org/eclipse/jetty/jetty-runner/9.4.20.v20190813/jetty-runner-9.4.20.v20190813.jar
btullis@marlin-wsl:~/tmp$ wget -q https://repo1.maven.org/maven2/org/eclipse/jetty/jetty-util/9.4.20.v20190813/jetty-util-9.4.20.v20190813.pom
btullis@marlin-wsl:~/tmp$ wget -q https://repo1.maven.org/maven2/org/eclipse/jetty/jetty-util/9.4.20.v20190813/jetty-util-9.4.20.v20190813.jar
btullis@marlin-wsl:~/tmp$ wget -q https://repo1.maven.org/maven2/org/eclipse/jetty/jetty-jmx/9.4.20.v20190813/jetty-jmx-9.4.20.v20190813.jar
btullis@marlin-wsl:~/tmp$ wget -q https://repo1.maven.org/maven2/org/eclipse/jetty/jetty-jmx/9.4.20.v20190813/jetty-jmx-9.4.20.v20190813.pom

I then downloaded all of the checksums

btullis@marlin-wsl:~/tmp$ wget -q https://repo1.maven.org/maven2/org/eclipse/jetty/jetty-runner/9.4.20.v20190813/jetty-runner-9.4.20.v20190813.pom.sha1
btullis@marlin-wsl:~/tmp$ wget -q https://repo1.maven.org/maven2/org/eclipse/jetty/jetty-runner/9.4.20.v20190813/jetty-runner-9.4.20.v20190813.jar.sha1
btullis@marlin-wsl:~/tmp$ wget -q https://repo1.maven.org/maven2/org/eclipse/jetty/jetty-util/9.4.20.v20190813/jetty-util-9.4.20.v20190813.jar.sha1
btullis@marlin-wsl:~/tmp$ wget -q https://repo1.maven.org/maven2/org/eclipse/jetty/jetty-util/9.4.20.v20190813/jetty-util-9.4.20.v20190813.pom.sha1
btullis@marlin-wsl:~/tmp$ wget -q https://repo1.maven.org/maven2/org/eclipse/jetty/jetty-jmx/9.4.20.v20190813/jetty-jmx-9.4.20.v20190813.pom.sha1
btullis@marlin-wsl:~/tmp$ wget -q https://repo1.maven.org/maven2/org/eclipse/jetty/jetty-jmx/9.4.20.v20190813/jetty-jmx-9.4.20.v20190813.jar.sha1
`

I then checked all of these checksums with the command:

btullis@marlin-wsl:~/tmp$ for i in runner jmx util; do for j in jar pom; do diff -Z <(shasum jetty-$i-9.4.20.v20190813.$j|awk '{print $1}') <(cat jetty-$i-9.4.20.v20190813.$j.sha1); done; done

All good, so I will upload these.

BTullis added a comment.Feb 16 2022, 5:11 PM

Ah, it looks like we already have these jetty artifacts in archiva:
https://archiva.wikimedia.org/#artifact/org.eclipse.jetty/jetty-runner
https://archiva.wikimedia.org/#artifact/org.eclipse.jetty/jetty-jmx
https://archiva.wikimedia.org/#artifact/org.eclipse.jetty/jetty-util

Not the same version as we are currently using, but the same major version. I think I can use 9.4.32.v20200930

BTullis added a comment.Feb 16 2022, 5:36 PM

I have uploaded the v20190813 artifacts anyway

The strange thing is that the download link is broken on Archiva. It puts an extra version string in the URL.

image.png (1×3 px, 426 KB)

I've updated the setup_gms.sh file to pull these artifacts.

I will now work on the next files, the prometheus_jmx_agent and the opentelemetry agent.

BTullis added a comment.EditedFeb 17 2022, 10:00 AM

Downloaded the jmx_prometheus_agent files.

btullis@marlin-wsl-wsl:~/tmp$ wget -q  https://repo1.maven.org/maven2/io/prometheus/jmx/jmx_prometheus_javaagent/0.16.1/jmx_prometheus_javaagent-0.16.1.jar
btullis@marlin-wsl-wsl:~/tmp$ wget -q https://repo1.maven.org/maven2/io/prometheus/jmx/jmx_prometheus_javaagent/0.16.1/jmx_prometheus_javaagent-0.16.1.jar.sha1
btullis@marlin-wsl-wsl:~/tmp$ wget -q https://repo1.maven.org/maven2/io/prometheus/jmx/jmx_prometheus_javaagent/0.16.1/jmx_prometheus_javaagent-0.16.1.pom
btullis@marlin-wsl-wsl:~/tmp$ wget -q https://repo1.maven.org/maven2/io/prometheus/jmx/jmx_prometheus_javaagent/0.16.1/jmx_prometheus_javaagent-0.16.1.pom.sha1

Performed a manual validation using shasum.

btullis@marlin-wsl-wsl:~/tmp$ for j in jar pom; do diff -Z <(shasum jmx_prometheus_javaagent-0.16.1.$j|awk '{print $1}')
 <(cat jmx_prometheus_javaagent-0.16.1.$j.sha1); done

Uploaded to archiva.

image.png (703×1 px, 103 KB)

BTullis added a comment.Feb 17 2022, 10:14 AM

Downloaded the opentelemetry-javaagent files.

btullis@marlin-wsl-wsl:~/tmp$ wget -q https://repo1.maven.org/maven2/io/opentelemetry/javaagent/instrumentation/opentelemetry-javaagent/0.14.0/opentelemetry-javaagent-0.14.0.jar
btullis@marlin-wsl-wsl:~/tmp$ wget -q https://repo1.maven.org/maven2/io/opentelemetry/javaagent/instrumentation/opentelemetry-javaagent/0.14.0/opentelemetry-javaagent-0.14.0.jar.sha1
btullis@marlin-wsl-wsl:~/tmp$ wget -q https://repo1.maven.org/maven2/io/opentelemetry/javaagent/instrumentation/opentelemetry-javaagent/0.14.0/opentelemetry-javaagent-0.14.0.pom
btullis@marlin-wsl-wsl:~/tmp$ wget -q https://repo1.maven.org/maven2/io/opentelemetry/javaagent/instrumentation/opentelemetry-javaagent/0.14.0/opentelemetry-javaagent-0.14.0.pom.sha1

Performed a manual validation using shasum

btullis@marlin-wsl-wsl:~/tmp$ for j in jar pom; do diff -Z <(shasum opentelemetry-javaagent-0.14.0.$j|awk '{print $1}') <(cat opentelemetry-javaagent-0.14.0.$j.sha1); done

Uploaded to archiva.

image.png (714×1 px, 104 KB)

https://archiva.wikimedia.org/#artifact/io.opentelemetry.javaagent.instrumentation/opentelemetry-javaagent/0.14.0

BTullis added a comment.Feb 17 2022, 10:23 AM

I'm not going to store the dockerize componenet in archiva, but I'll validate that against its sha1 hash instead.

BTullis closed this task as Resolved.Feb 17 2022, 10:23 AM
BTullis moved this task from In Progress to Done on the Data-Catalog board.
BTullis moved this task from In Progress to Done on the Data-Engineering-Kanban board.
Content licensed under Creative Commons Attribution-ShareAlike (CC BY-SA) 4.0 unless otherwise noted; code licensed under GNU General Public License (GPL) 2.0 or later and other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL