Page MenuHomePhabricator
Create Task
Maniphest T367405

Migrate existing Java packages to deploying to Gitlab, including new version of parent pom, validation that all dependencies are available, and validation that deployment to production still works
Open, MediumPublic8 Estimated Story Points
Actions

Description

Existing active projects need to be migrated to fetch dependencies from Gitlab and publish artifacts to Gitlab.

Parent pom might need some further adaptation as we discover the specificity of existing projects.

Some external dependencies might not be available from maven central, those need to be tracked and added at the project level.

For each project, we want to validate:

  • releases are created by CI and published to Gitlab
  • production deployment are working
Projects to migrate
Data Engineering Maintained Repos

(See also T368927: [Epic] Migrate Data Platform Engineering maintained git repos to GitLab)

Details

Related Changes in Gerrit:
Show related patches Customize query in gerrit
Related Changes in GitLab:
TitleReferenceAuthorSource BranchDest Branch
Update gitlab-ci to use wmcs for more resourcerepos/data-engineering/patches/wmf-sparksqlclidriver!5joalupdate_cimain
Update gitlab-ci to use WMCS for more resourcerepos/data-engineering/hdfs-tools!3joalupdate_cimain
Add .gitlab-ci.yml and maven-wrapper propertiesrepos/data-engineering/hdfs-tools!2joaladd_cimain
Add .gitlab-ci.yml filerepos/data-engineering/patches/wmf-sparksqlclidriver!4joaladd_gitlab-cimain
Increase compatibility with projects hosted outside GitLabrepos/maven/wmf-jvm-parent-pom!24pfischerwmf-packagesmain
Use Maven CI/CD components.repos/data-engineering/gobblin-wmf!3pfischermaven-cimain
Migrate from https://gitlab.wikimedia.org/repos/maven/gerrit-artefactsrepos/wmf-packages!1pfischerpackage-importermain
Forge mvn.ymlrepos/maven/wmf-maven-ci!1pfischertinkermain
Add gitlab-ci script to populate project coordinatesrepos/maven/wmf-jvm-parent-pom!23pfischergitlab-ci-profilemain
Draft: CI: build java via child pipelinerepos/data-engineering/metrics-platform!87pfischermaven-cimain
CI: build java via child pipelinerepos/data-engineering/metrics-platform!86pfischermaven-cimain
Add release/deployment hint for Gerrit-hosted projectsrepos/maven/wmf-jvm-parent-pom!22pfischergerrit-artifactsmain
Add maven release capability to CI pipelinerepos/maven/wmf-jvm-parent-pom!21pfischergitlab-maven-releasemain
Show related patches Customize query in GitLab

Related Objects
Search...

Event Timeline

There are a very large number of changes, so older changes are hidden. Show Older Changes
gerritbot added a comment.May 13 2025, 12:11 PM

Change #1145174 had a related patch set uploaded (by Peter Fischer; author: Peter Fischer):

[integration/config@master] assign maven-release job to search/wmf-jvm-utils

https://gerrit.wikimedia.org/r/1145174

CodeReviewBot added a comment.May 14 2025, 8:06 PM

pfischer opened https://gitlab.wikimedia.org/repos/maven/wmf-jvm-parent-pom/-/merge_requests/24

Updated README

CodeReviewBot added a comment.May 15 2025, 11:20 AM

pfischer merged https://gitlab.wikimedia.org/repos/maven/wmf-jvm-parent-pom/-/merge_requests/24

Increase compatibility with projects hosted outside GitLab

gerritbot added a comment.May 16 2025, 7:11 AM

Change #1145174 merged by jenkins-bot:

[integration/config@master] jjb: add maven-release job to search/wmf-jvm-utils

https://gerrit.wikimedia.org/r/1145174

gerritbot added a comment.May 16 2025, 12:38 PM

Change #1146975 had a related patch set uploaded (by Peter Fischer; author: Peter Fischer):

[wmf-jvm-utils@master] Prepare POM for deployment to wmf-packages

https://gerrit.wikimedia.org/r/1146975

gerritbot added a comment.May 16 2025, 2:15 PM

Change #1147003 had a related patch set uploaded (by Peter Fischer; author: Peter Fischer):

[integration/config@master] jjb: use local settings for all maven goals

https://gerrit.wikimedia.org/r/1147003

gerritbot added a comment.May 16 2025, 2:20 PM

Change #1147003 merged by jenkins-bot:

[integration/config@master] jjb: use local settings for all maven goals

https://gerrit.wikimedia.org/r/1147003

gerritbot added a comment.May 16 2025, 3:17 PM

Change #1147017 had a related patch set uploaded (by Hashar; author: Hashar):

[integration/config@master] Revert "jjb: use local settings for all maven goals"

https://gerrit.wikimedia.org/r/1147017

gerritbot added a comment.May 16 2025, 3:24 PM

Change #1147017 merged by jenkins-bot:

[integration/config@master] Revert "jjb: use local settings for all maven goals"

https://gerrit.wikimedia.org/r/1147017

gerritbot added a comment.May 19 2025, 10:18 AM

Change #1146975 merged by jenkins-bot:

[wmf-jvm-utils@master] Prepare POM for deployment to wmf-packages

https://gerrit.wikimedia.org/r/1146975

jenkins-bot mentioned this in rWJVMf1af523f0f61: Prepare POM for deployment to wmf-packages.May 19 2025, 10:20 AM
gerritbot added a comment.May 19 2025, 1:55 PM

Change #1147771 had a related patch set uploaded (by Peter Fischer; author: Peter Fischer):

[analytics/refinery/source@master] Update parent POM

https://gerrit.wikimedia.org/r/1147771

gerritbot added a comment.May 19 2025, 2:09 PM

Change #1147774 had a related patch set uploaded (by Peter Fischer; author: Peter Fischer):

[wikimedia-event-utilities@master] Update parent POM

https://gerrit.wikimedia.org/r/1147774

gerritbot added a comment.May 20 2025, 9:03 AM

Change #1148277 had a related patch set uploaded (by Peter Fischer; author: Peter Fischer):

[wikidata/query/rdf@master] Update parent POM

https://gerrit.wikimedia.org/r/1148277

gerritbot added a comment.May 20 2025, 11:07 AM

Change #1148304 had a related patch set uploaded (by Peter Fischer; author: Peter Fischer):

[wikidata/query/deploy@master] Use GitLab package registry to fetch dist archive

https://gerrit.wikimedia.org/r/1148304

amastilovic added a comment.May 20 2025, 6:50 PM

Hey @pfischer , with regards to Gobblin-WMF we've migrated the project to GitLab and enabled the CI/CD release of package to the GitLab's Maven repository: https://gitlab.wikimedia.org/repos/data-engineering/gobblin-wmf

It'd be greatly appreciated if you'd take a look at the blubber.yaml and pom.xml files and ensure we're conforming to the new parent POM/release standards.

gerritbot added a comment.May 21 2025, 7:53 AM

Change #1148793 had a related patch set uploaded (by Peter Fischer; author: Peter Fischer):

[search/extra@master] Deploy to central via Sonatype's Central Portal

https://gerrit.wikimedia.org/r/1148793

gerritbot added a comment.May 21 2025, 7:56 AM

Change #1148794 had a related patch set uploaded (by Peter Fischer; author: Peter Fischer):

[search/extra-analysis@master] Deploy to central via Sonatype's Central Portal

https://gerrit.wikimedia.org/r/1148794

gerritbot added a comment.May 21 2025, 9:55 AM

Change #1148817 had a related patch set uploaded (by Peter Fischer; author: Peter Fischer):

[search/glent@master] Update parent POM

https://gerrit.wikimedia.org/r/1148817

gerritbot added a comment.May 21 2025, 10:05 AM

Change #1148821 had a related patch set uploaded (by Peter Fischer; author: Peter Fischer):

[search/highlighter@master] Deploy to central via Sonatype's Central Portal

https://gerrit.wikimedia.org/r/1148821

pfischer added a comment.EditedMay 21 2025, 11:40 AM
Projects to migrate
Data Engineering Maintained Repos

(See also T368927: [Epic] Migrate Data Platform Engineering maintained git repos to GitLab)

CodeReviewBot added a comment.May 21 2025, 4:33 PM

amastilovic merged https://gitlab.wikimedia.org/repos/data-engineering/gobblin-wmf/-/merge_requests/3

Use Maven CI/CD components.

amastilovic updated the task description. (Show Details)May 21 2025, 4:33 PM
gerritbot added a comment.May 22 2025, 9:13 AM

Change #1148277 merged by jenkins-bot:

[wikidata/query/rdf@master] Update parent POM

https://gerrit.wikimedia.org/r/1148277

gerritbot added a comment.May 22 2025, 2:00 PM

Change #1148821 merged by jenkins-bot:

[search/highlighter@master] Deploy to central via Sonatype's Central Portal

https://gerrit.wikimedia.org/r/1148821

gerritbot added a comment.May 22 2025, 2:03 PM

Change #1148793 merged by jenkins-bot:

[search/extra@master] Deploy to central via Sonatype's Central Portal

https://gerrit.wikimedia.org/r/1148793

gerritbot added a comment.May 22 2025, 2:07 PM

Change #1148794 merged by jenkins-bot:

[search/extra-analysis@master] Deploy to central via Sonatype's Central Portal

https://gerrit.wikimedia.org/r/1148794

gerritbot added a comment.May 22 2025, 2:09 PM

Change #1148817 merged by jenkins-bot:

[search/glent@master] Update parent POM

https://gerrit.wikimedia.org/r/1148817

Gehel edited projects, added: Discovery-Search (2025.05.24 - 2025.06.13); removed: Discovery-Search (2025.05.02 - 2025.05.23).May 23 2025, 1:01 PM
Gehel moved this task from Incoming to In Progress on the Discovery-Search (2025.05.24 - 2025.06.13) board.
pfischer moved this task from In Progress to Needs Review on the Discovery-Search (2025.05.24 - 2025.06.13) board.May 26 2025, 2:59 PM
gerritbot added a comment.May 27 2025, 3:06 PM

Change #1147774 merged by jenkins-bot:

[wikimedia-event-utilities@master] Update parent POM

https://gerrit.wikimedia.org/r/1147774

pfischer updated the task description. (Show Details)May 27 2025, 3:26 PM
gerritbot added a comment.May 27 2025, 4:25 PM

Change #1148304 merged by DCausse:

[wikidata/query/deploy@master] Use GitLab package registry to fetch dist archive

https://gerrit.wikimedia.org/r/1148304

Gehel edited projects, added: Discovery-Search (2025.06.13 - 2025.07.04); removed: Discovery-Search (2025.05.24 - 2025.06.13).Jun 16 2025, 1:35 PM
Gehel moved this task from Incoming to Needs Review on the Discovery-Search (2025.06.13 - 2025.07.04) board.
pfischer edited projects, added: Discovery-Search (2025.07.04 - 2025.07.25); removed: Discovery-Search (2025.06.13 - 2025.07.04).Jul 17 2025, 2:19 PM
pfischer moved this task from Incoming to Needs Review on the Discovery-Search (2025.07.04 - 2025.07.25) board.
pfischer edited projects, added: Discovery-Search (2025.07.25 - 2025.08.15); removed: Discovery-Search (2025.07.04 - 2025.07.25).Aug 6 2025, 7:29 AM
pfischer moved this task from Incoming to Needs Review on the Discovery-Search (2025.07.25 - 2025.08.15) board.
JAllemandou subscribed.Aug 14 2025, 11:02 AM

In order to migrate the refinery-source project, we need to update this script: https://github.com/wikimedia/analytics-refinery/blob/master/bin/update-refinery-source-jars#L101
I'll prioritize this so that this patch can be merged and applied: https://gerrit.wikimedia.org/r/c/analytics/refinery/source/+/1147771

gerritbot added a comment.Aug 14 2025, 3:33 PM

Change #1178895 had a related patch set uploaded (by Joal; author: Joal):

[analytics/refinery@master] Update artifacts download script to use gitlab

https://gerrit.wikimedia.org/r/1178895

pfischer edited projects, added: Discovery-Search (2025.08.15 - 2025.09.05); removed: Discovery-Search (2025.07.25 - 2025.08.15).Aug 20 2025, 2:38 PM
pfischer moved this task from Incoming to Needs Review on the Discovery-Search (2025.08.15 - 2025.09.05) board.
gerritbot added a comment.Aug 21 2025, 12:06 PM

Change #1147771 merged by jenkins-bot:

[analytics/refinery/source@master] Update parent POM

https://gerrit.wikimedia.org/r/1147771

gerritbot added a comment.Aug 21 2025, 12:14 PM

Change #1178895 merged by Joal:

[analytics/refinery@master] Update artifacts download script to use gitlab

https://gerrit.wikimedia.org/r/1178895

gerritbot added a comment.Aug 21 2025, 1:35 PM

Change #1180854 had a related patch set uploaded (by Joal; author: Joal):

[analytics/refinery/source@master] Update project to release v0.3.0

https://gerrit.wikimedia.org/r/1180854

gerritbot added a comment.Aug 21 2025, 1:47 PM

Change #1180854 merged by jenkins-bot:

[analytics/refinery/source@master] Update project to release v0.3.0

https://gerrit.wikimedia.org/r/1180854

pfischer updated the task description. (Show Details)Aug 22 2025, 2:39 PM
Gehel edited projects, added: Discovery-Search (2025.09.05 - 2025.09.26); removed: Discovery-Search (2025.08.15 - 2025.09.05).Sep 5 2025, 8:23 AM
Gehel moved this task from Incoming to Needs Review on the Discovery-Search (2025.09.05 - 2025.09.26) board.
Gehel added a project: Essential-Work.Sep 11 2025, 12:19 PM
pfischer edited projects, added: Discovery-Search (2025.09.26 - 2025.10.17); removed: Discovery-Search (2025.09.05 - 2025.09.26).Oct 2 2025, 9:53 AM
pfischer moved this task from Incoming to Needs Review on the Discovery-Search (2025.09.26 - 2025.10.17) board.
Gehel closed subtask T386406: Create Gitlab CI templates for JVM packages as Resolved.Oct 3 2025, 8:11 AM
pfischer edited projects, added: Discovery-Search (2025.10.20 - 2025.12.31); removed: Discovery-Search (2025.09.26 - 2025.10.17).Oct 27 2025, 1:00 PM
pfischer moved this task from Incoming to Needs Review on the Discovery-Search (2025.10.20 - 2025.12.31) board.
pfischer added a comment.Dec 18 2025, 5:00 PM

As discussed, to finalise the migration of HDFS Tools, scap deploy must be adapted, @JAllemandou.

gerritbot added a comment.Dec 19 2025, 8:16 AM

Change #1219678 had a related patch set uploaded (by Joal; author: Joal):

[analytics/hdfs-tools/deploy@master] Update README.md for hdfs-tools gitlab migration

https://gerrit.wikimedia.org/r/1219678

JAllemandou renamed this task from Migrate existing Java packages to deploying to Gitlab, including new version of parent pom, validation that all dependencies are available, and validation that deployment to production still works to Migrate existing Java packages to deploying to Gitlab, including new version of parent pom, validation that all dependencies are available, and validation that deployment to production still works.Dec 19 2025, 8:17 AM
JAllemandou updated the task description. (Show Details)
JAllemandou updated the task description. (Show Details)
pfischer edited projects, added: Discovery-Search (2026.01.05 - 2026.01.30); removed: Discovery-Search (2025.10.20 - 2025.12.31).Jan 5 2026, 12:12 PM
pfischer moved this task from Incoming to Needs Review on the Discovery-Search (2026.01.05 - 2026.01.30) board.
gerritbot added a comment.Jan 6 2026, 7:36 AM

Change #1219678 merged by Joal:

[analytics/hdfs-tools/deploy@master] Update README.md for hdfs-tools gitlab migration

https://gerrit.wikimedia.org/r/1219678

JAllemandou updated the task description. (Show Details)Jan 6 2026, 7:36 AM
pfischer edited projects, added: Discovery-Search (2026.02.02 - 2026.02.27); removed: Discovery-Search (2026.01.05 - 2026.01.30).Feb 9 2026, 12:46 PM
pfischer moved this task from Incoming to Needs Review on the Discovery-Search (2026.02.02 - 2026.02.27) board.
pfischer updated the task description. (Show Details)Feb 9 2026, 1:30 PM
pfischer edited projects, added: Discovery-Search (2026.03.03 - 2026.04.03); removed: Discovery-Search (2026.02.02 - 2026.02.27).Mar 9 2026, 11:14 AM
pfischer moved this task from Incoming to Needs Review on the Discovery-Search (2026.03.03 - 2026.04.03) board.
dr0ptp4kt unsubscribed.Mar 15 2026, 2:00 PM
pfischer edited projects, added: Discovery-Search (2026.04.06 - 2026.05.01); removed: Discovery-Search (2026.03.03 - 2026.04.03).Apr 13 2026, 1:56 PM
pfischer moved this task from Incoming to Needs Review on the Discovery-Search (2026.04.06 - 2026.05.01) board.
pfischer edited projects, added: Discovery-Search (2026.05.04 - 2026.05.29); removed: Discovery-Search (2026.04.06 - 2026.05.01).May 18 2026, 1:40 PM
pfischer moved this task from Incoming to Needs Review on the Discovery-Search (2026.05.04 - 2026.05.29) board.
pfischer edited projects, added: Discovery-Search (2026.06.01 - 2026.07.03); removed: Discovery-Search (2026.05.04 - 2026.05.29).Jun 8 2026, 2:34 PM
pfischer moved this task from Incoming to Needs Review on the Discovery-Search (2026.06.01 - 2026.07.03) board.
EBernhardson removed a project: Discovery-Search (2026.06.01 - 2026.07.03).Jun 29 2026, 3:09 PM
EBernhardson updated the task description. (Show Details)
EBernhardson subscribed.

We believe all search projects are complete.

CodeReviewBot added a comment.Jun 30 2026, 6:51 PM

cjming merged https://gitlab.wikimedia.org/repos/data-engineering/metrics-platform/-/merge_requests/86

CI: build java via child pipeline

cjming subscribed.EditedJun 30 2026, 6:53 PM

as confirmed in Slack, MP java lib isn't being used in production and we haven't cut a release in a long time - i went ahead and merged the MR anyway so this ticket can be closed out -- we'll probably remove the Java lib and related infrastructure as part of some future cleanup
cc @KReid-WMF

cjming added a subscriber: KReid-WMF.Jun 30 2026, 6:55 PM
cjming updated the task description. (Show Details)Jun 30 2026, 7:05 PM
JAllemandou updated the task description. (Show Details)Jul 1 2026, 1:45 PM
amastilovic added a comment.Tue, Jul 21, 10:14 PM

@Gehel I see that Peter just merged https://gitlab.wikimedia.org/repos/data-engineering/patches/wmf-sparksqlclidriver/-/merge_requests/3 so I think the last checkbox in the data engineering list can be ticked.

amastilovic updated the task description. (Show Details)Tue, Jul 21, 10:14 PM
CodeReviewBot added a comment.Fri, Jul 31, 12:11 PM

joal opened https://gitlab.wikimedia.org/repos/data-engineering/patches/wmf-sparksqlclidriver/-/merge_requests/4

Add .gitlab-ci.yml file

CodeReviewBot added a comment.Fri, Jul 31, 12:13 PM

joal closed https://gitlab.wikimedia.org/repos/data-engineering/patches/wmf-sparksqlclidriver/-/merge_requests/4

Add .gitlab-ci.yml file

CodeReviewBot added a comment.Mon, Aug 3, 11:24 AM

joal opened https://gitlab.wikimedia.org/repos/data-engineering/hdfs-tools/-/merge_requests/2

Add .gitlab-ci.yml

CodeReviewBot added a comment.Mon, Aug 3, 12:25 PM

joal merged https://gitlab.wikimedia.org/repos/data-engineering/hdfs-tools/-/merge_requests/2

Add .gitlab-ci.yml and maven-wrapper properties

CodeReviewBot added a comment.Mon, Aug 3, 3:19 PM

joal opened https://gitlab.wikimedia.org/repos/data-engineering/hdfs-tools/-/merge_requests/3

Update gitlab-ci to use WMCS for more resource

CodeReviewBot added a comment.Mon, Aug 3, 3:24 PM

joal merged https://gitlab.wikimedia.org/repos/data-engineering/hdfs-tools/-/merge_requests/3

Update gitlab-ci to use WMCS for more resource

CodeReviewBot added a comment.Tue, Aug 4, 8:08 AM

joal opened https://gitlab.wikimedia.org/repos/data-engineering/patches/wmf-sparksqlclidriver/-/merge_requests/5

Update gitlab-ci to use wmcs for more resource

CodeReviewBot added a comment.Tue, Aug 4, 8:20 AM

joal merged https://gitlab.wikimedia.org/repos/data-engineering/patches/wmf-sparksqlclidriver/-/merge_requests/5

Update gitlab-ci to use wmcs for more resource

amastilovic created subtask T434160: Decouple airflow-dags repository from Archiva.Thu, Aug 6, 12:22 AM
Content licensed under Creative Commons Attribution-ShareAlike (CC BY-SA) 4.0 unless otherwise noted; code licensed under GNU General Public License (GPL) 2.0 or later and other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL · Credits