[Epic] Replace Archiva with Gitlab artifact repositories
Open, HighPublic13 Estimated Story Points
Actions

Assigned To

None

Authored By

	Gehel
	Jun 12 2024, 2:25 PM

Description

As discussed in T358612, we want to migrate from Archiva (which is unsupported) to Gitlab. See https://docs.google.com/document/d/1RmNnWduezwQC6246YvfBJMnG74kUCncNzwsY1Z5E4ZU/edit for more context.

Create a global package registry in Gitlab - T367322 -> T379187 -> repos/wmf-packages
Setup a test project to validate upload to the package registry, including changes to our parent pom needed to upload to Gitlab by default - T367391
Move Maven support projects to the Maven Gitlab group - T369901
Validate CI integration so that Ci can release artifacts on user's demand - T367403
Update parent pom to disable fetching dependencies from Archiva and use Gitlab instead - T367404
Migrate existing Java packages to deploying to Gitlab, including new version of parent pom, validation that all dependencies are available, and validation that deployment to production still works - T367405
Migration Python projects that depend on Archiva for deployment - T367406
Migrate [[datahub-cli|https://github.com/wikimedia/analytics-refinery/tree/master/packaged-environments/datahub-cli]] - Conda environment hosted by Archiva
Retire Archiva, including keeping a read only copy of all previously published artifacts - T367407

Details

Related Changes in Gerrit:

	Subject	Repo	Branch	Lines +/-
	Remove rsync from archiva	operations/puppet	production	+1 -11
	Remove firewall rule for rsync on archiva	operations/puppet	production	+0 -10

Customize query in gerrit

Related Objects
Search...

Status	Assigned	Task
Open	None	T368927 [Epic] Migrate Data Platform Engineering maintained git repos to GitLab
Open	None	T367315 [Epic] Replace Archiva with Gitlab artifact repositories
Resolved	Gehel	T358612 Investigate replacing Archiva with Gitlab repositories
		Restricted Task
Resolved	Gehel	T367322 Create a Maven package registry in Gitlab
Resolved	EBernhardson	T379203 Add CI_RELEASE_TOKEN secret for {name}-maven-release jobs in jenkins
Resolved	Gehel	T367391 Setup a test project to validate upload to the Gitlab package registry
Resolved	amastilovic	T367403 Validate CI integration so that Ci can release Maven artifacts on user's demand
Resolved	Gehel	T367404 Update parent pom to disable fetching dependencies from Archiva and use Gitlab instead
Open	pfischer	T367405 Migrate existing Java packages to deploying to Gitlab, including new version of parent pom, validation that all dependencies are available, and validation that deployment to production still works
Duplicate	None	T370400 Java projects hosted on Gerrit should publish artifacts to Gitlab
Resolved	EBernhardson	T377153 Migrate Glent to Gitlab for publication of artifacts
Duplicate	amastilovic	T389248 Create a template for building Maven projects
Resolved	pfischer	T386406 Create Gitlab CI templates for JVM packages
Open	None	T386056 Resolve conflict between GitLab CI automated package deployment token variable names
Open	None	T367406 Migrate Python projects that depend on Archiva for deployment
Open	None	T373450 List python projects that are published to Archiva and how they are deployed
Resolved	brennen	T379187 Create new GitLab project: repos/wmf-packages
Open	None	T367407 Retire Archiva, including keeping a read only copy of all previously published artifacts
Resolved	Gehel	T369901 Migrate wmf-jvm-parent-pom and supporting components to the Maven group on Gitlab

Event Timeline

There are a very large number of changes, so older changes are hidden. Show Older Changes

Restricted Application added a subscriber: Aklapper. · View Herald TranscriptJun 12 2024, 2:25 PM

Gehel added a subtask: T358612: Investigate replacing Archiva with Gitlab repositories.Jun 12 2024, 2:25 PM

Gehel closed subtask T358612: Investigate replacing Archiva with Gitlab repositories as Resolved.

Gehel mentioned this in T358612: Investigate replacing Archiva with Gitlab repositories.

MoritzMuehlenhoff subscribed.Jun 12 2024, 2:29 PM

Gehel mentioned this in T367322: Create a Maven package registry in Gitlab.Jun 12 2024, 2:39 PM

Ottomata subscribed.Jun 12 2024, 2:41 PM

Gehel updated the task description. (Show Details)Jun 13 2024, 10:11 AM

Gehel added a project: Java-Scala-Standardization.Jun 13 2024, 10:13 AM

Gehel updated the task description. (Show Details)Jun 13 2024, 12:36 PM

Gehel updated the task description. (Show Details)Jun 13 2024, 12:39 PM

Gehel triaged this task as High priority.Jun 13 2024, 1:16 PM

Gehel updated the task description. (Show Details)Jun 13 2024, 3:38 PM

BTullis updated the task description. (Show Details)Jun 13 2024, 3:41 PM

LSobanski subscribed.Jun 17 2024, 3:16 PM

Gehel moved this task from Backlog to In Progress on the Java-Scala-Standardization board.Jun 18 2024, 6:36 PM

Ottomata mentioned this in T368927: [Epic] Migrate Data Platform Engineering maintained git repos to GitLab.Jul 1 2024, 3:46 PM

Gehel moved this task from needs triage to [epic] on the Discovery-Search board.Jul 4 2024, 8:16 AM

Gehel moved this task from Incoming to Epics on the Data-Platform-SRE board.Jul 9 2024, 10:59 AM

Gehel updated the task description. (Show Details)Jul 12 2024, 12:58 PM

hashar mentioned this in T373352: Archiva release repository yields 404 for release repository breaking builds.Aug 26 2024, 3:25 PM

Ahoelzl moved this task from Incoming (new tickets) to Q1 2024 July 1st - September 30th on the Data-Engineering board.Aug 26 2024, 5:17 PM

Ahoelzl edited projects, added Data-Engineering (Q1 2024 July 1st - September 30th); removed Data-Engineering.

I had a link to the documentation at https://wikitech.wikimedia.org/wiki/Data_Platform/Systems/Archiva which surely should be marked as historical (with https://wikitech.wikimedia.org/wiki/Template:Archive )

Gehel closed subtask T367391: Setup a test project to validate upload to the Gitlab package registry as Resolved.Aug 27 2024, 2:47 PM

Gehel changed the status of subtask T367403: Validate CI integration so that Ci can release Maven artifacts on user's demand from Open to Stalled.Aug 27 2024, 2:50 PM

Ahoelzl set the point value for this task to 13.Sep 6 2024, 8:53 PM

hashar unsubscribed.Sep 9 2024, 8:07 AM

Gehel closed subtask T369901: Migrate wmf-jvm-parent-pom and supporting components to the Maven group on Gitlab as Resolved.Sep 9 2024, 1:47 PM

Gehel updated the task description. (Show Details)Sep 9 2024, 1:49 PM

Gehel closed subtask T367404: Update parent pom to disable fetching dependencies from Archiva and use Gitlab instead as Resolved.

Gehel updated the task description. (Show Details)

Gehel closed subtask T367322: Create a Maven package registry in Gitlab as Resolved.Sep 25 2024, 7:59 AM

Gehel removed a subtask: T370400: Java projects hosted on Gerrit should publish artifacts to Gitlab.Sep 25 2024, 8:01 AM

dr0ptp4kt changed the status of subtask T367403: Validate CI integration so that Ci can release Maven artifacts on user's demand from Stalled to Open.Sep 30 2024, 3:09 PM

Ahoelzl edited projects, added Data-Engineering (Q2 2024 October 1st - December 31th); removed Data-Engineering (Q1 2024 July 1st - September 30th).Oct 9 2024, 1:50 PM

Ahoelzl closed subtask T367403: Validate CI integration so that Ci can release Maven artifacts on user's demand as Resolved.Oct 9 2024, 1:58 PM

Ottomata updated the task description. (Show Details)Nov 6 2024, 7:31 PM

Ottomata updated the task description. (Show Details)

Ahoelzl moved this task from Next Up to If we have time on the Data-Engineering (Q2 2024 October 1st - December 31th) board.Dec 18 2024, 1:58 PM

We still have rsync offered globally on that host: While we have deployed the latest rsync security fixes (and the most critical one didn't affect Bullseye, what Archiva is running on), we should remove this if no longer needed. Is still needed for Wikimedia use cases or can we remove it?

Is still needed for Wikimedia use cases or can we remove it?

Hm, I think it might not be!

rsync was used here for git-fat support. I believe the main use case I know of (analytics/refinery jar deployment via scap) now uses git-lfs, which I don't think uses rsync. Someone else should confirm though!

In T367315#10462560, @Ottomata wrote:

Is still needed for Wikimedia use cases or can we remove it?

Hm, I think it might not be!

rsync was used here for git-fat support. I believe the main use case I know of (analytics/refinery jar deployment via scap) now uses git-lfs, which I don't think uses rsync. Someone else should confirm though!

This is also my understanding. I think we should be able to disable the public-facing rsync service on this host, without causing any inconvenience.

Change #1112226 had a related patch set uploaded (by Muehlenhoff; author: Muehlenhoff):

[operations/puppet@production] Remove firewall rule for rsync on archiva

https://gerrit.wikimedia.org/r/1112226

gerritbot added a project: Patch-For-Review.Jan 17 2025, 2:44 PM

Change #1112228 had a related patch set uploaded (by Muehlenhoff; author: Muehlenhoff):

[operations/puppet@production] Remove rsync from archiva

https://gerrit.wikimedia.org/r/1112228

Ahoelzl edited projects, added Data-Engineering; removed Data-Engineering (Q2 2024 October 1st - December 31th).Jan 18 2025, 12:27 AM

Ahoelzl moved this task from Q2 2024 October 1st - December 31th to Next Up on the Data-Engineering board.

Change #1112226 merged by Muehlenhoff:

[operations/puppet@production] Remove firewall rule for rsync on archiva