Page MenuHomePhabricator
Create Task
Maniphest T368927

[Epic] Migrate Data Platform Engineering maintained git repos to GitLab
Open, Needs TriagePublic
Actions

Description

This is an epic parent task. Actual work will be done in subtasks.

Java repos

Deployment of these repo's jar artifacts to GitLab is tracked in T367405: Migrate existing Java packages to deploying to Gitlab, including new version of parent pom, validation that all dependencies are available, and validation that deployment to production still works. Ideally we could migrate these to GitLab before work is done in that ticket to make Jenkins deploy to GitLab. That way the work to deploy via CI only has to be done once in Gitlab CI.

Other repos

Note: scap & 'deploy' repos are listed here, but we may not need to migrate them if we change our deployment process. See T322690: Add support for repository artifacts in Airflow

Other work

Related Objects
Search...

StatusSubtypeAssignedTask
 OpenNoneT368927 [Epic] Migrate Data Platform Engineering maintained git repos to GitLab
 OpenNoneT366614 [Epic] Migrate Data Engineering maintained NodeJS repositories to GitLab
 ResolvedSnwachukwuT344730 Migrate Data Engineering Pipelinelib repos to GitLab
 ResolvedSnwachukwuT366611 Migrate Data Engineering NodeJS library repos to GitLab
 ResolvedtchinT366537 Create gitlab ci npm publish pipeline and job in workflow_utils gitlab_ci_templates
 ResolvedtchinT366612 Publish Data Engineering maintained NodeJS packages to GitLab and use them in depender code
 ResolvedOttomataT383814 Upgrade eventgate-wikimedia to node20
 ResolvedSnwachukwuT366836 Migrate Event Platform Schema Respositories to Gitlab
 OpenpfischerT367405 Migrate existing Java packages to deploying to Gitlab, including new version of parent pom, validation that all dependencies are available, and validation that deployment to production still works
 DuplicateNoneT370400 Java projects hosted on Gerrit should publish artifacts to Gitlab
 ResolvedEBernhardsonT377153 Migrate Glent to Gitlab for publication of artifacts
 DuplicateamastilovicT389248 Create a template for building Maven projects
 ResolvedpfischerT386406 Create Gitlab CI templates for JVM packages
 OpenNoneT386056 Resolve conflict between GitLab CI automated package deployment token variable names
 OpenamastilovicT362832 Migrate refinery HQL files to CI/CD supported GitLab repository
 Resolvednshahquinn-wmfT304544 Move Wmfdata-Python from Github to Gitlab

Event Timeline

Ottomata created this task.Jul 1 2024, 3:25 PM
Restricted Application added a subscriber: Aklapper. · View Herald TranscriptJul 1 2024, 3:25 PM
Ottomata added a subtask: T366614: [Epic] Migrate Data Engineering maintained NodeJS repositories to GitLab.Jul 1 2024, 3:25 PM
Ottomata added a subtask: T367405: Migrate existing Java packages to deploying to Gitlab, including new version of parent pom, validation that all dependencies are available, and validation that deployment to production still works.
Ottomata added a subtask: T362832: Migrate refinery HQL files to CI/CD supported GitLab repository.Jul 1 2024, 3:27 PM
Ottomata renamed this task from [Epic] Migrate Data Engineering maintained git repos to GitLab to [Epic] Migrate Data Platform Engineering maintained git repos to GitLab.Jul 1 2024, 3:46 PM
Ottomata updated the task description. (Show Details)
Ottomata updated the task description. (Show Details)
Ottomata updated the task description. (Show Details)Jul 1 2024, 3:49 PM
Ottomata added a project: Data-Platform-SRE.
Ottomata added subscribers: Gehel, BTullis, brouberol.

@Ahoelzl here's my first attempt at pulling this together.

Note that most of the work is around migrating Java lib repos to GitLab. These are tracked in T367315: [Epic] Replace Archiva with Gitlab artifact repositories and are dependent on Data-Platform-SRE work being done for T367315: [Epic] Replace Archiva with Gitlab artifact repositories.

We should coordinate with DPE-SRE before embarking and/or committing to this work.

Ottomata updated the task description. (Show Details)Jul 1 2024, 6:08 PM
Ottomata mentioned this in T367405: Migrate existing Java packages to deploying to Gitlab, including new version of parent pom, validation that all dependencies are available, and validation that deployment to production still works.Jul 1 2024, 6:10 PM
Gehel added a comment.Jul 1 2024, 6:34 PM
In T368927#9940491, @Ottomata wrote:

Note that most of the work is around migrating Java lib repos to GitLab. These are tracked in T367315: [Epic] Replace Archiva with Gitlab artifact repositories and are dependent on Data-Platform-SRE work being done for T367315: [Epic] Replace Archiva with Gitlab artifact repositories.

We should coordinate with DPE-SRE before embarking and/or committing to this work.

I think that those 2 tickets are mostly orthogonal. T367315 is about the publication of artifacts not about where the code is hosted.

Ottomata added a comment.Jul 1 2024, 8:00 PM

Yes sorry, phab ticket # copy/paste fail. I meant to link T367405: Migrate existing Java packages to deploying to Gitlab, including new version of parent pom, validation that all dependencies are available, and validation that deployment to production still works.

Just checking, did I interpret T367405 wrong then? Is that only about the deployment stuff, and not about migration of repos and CI to gitlab? If so then I'll update and make tickets for that work.

Gehel added a comment.Jul 9 2024, 8:06 AM
In T368927#9942044, @Ottomata wrote:

Just checking, did I interpret T367405 wrong then? Is that only about the deployment stuff, and not about migration of repos and CI to gitlab? If so then I'll update and make tickets for that work.

The Archiva to Gitlab migration is purely about deployment, not about code hosting. We should still be able to have code hosted on Gerrit, built in Jenkins and artifacts published to Gitlab. Obviously, if everything is hosted on Gitlab, it makes things slightly easier, but not all that much (I think it would actually be easier if everything was hosted on Gerrit :) )

Gehel removed a project: Data-Platform-SRE.Jul 9 2024, 8:09 AM

Removing Data Platform SRE, code management isn't our area of concern. Ping us if you need help, but you should be able to move forward without us.

Ottomata added a comment.Jul 9 2024, 3:41 PM

Got it, thanks @Gehel. I'll update this task to incorporate GitLab code hosting of Java projects directly.

Ottomata updated the task description. (Show Details)Jul 19 2024, 2:04 PM
Restricted Application added a project: Epic. · View Herald TranscriptJul 19 2024, 2:04 PM
Ottomata mentioned this in T377492: Update hdfs-tools to new parent pom.Oct 17 2024, 5:39 PM
Ottomata mentioned this in T360219: Migrate WMF Maven projects to new parent pom.
Ahoelzl moved this task from Incoming (new tickets) to Backlog on the Data-Engineering board.Oct 17 2024, 5:53 PM
nshahquinn-wmf added a subtask: T304544: Move Wmfdata-Python from Github to Gitlab.Oct 25 2024, 7:10 PM
nshahquinn-wmf mentioned this in T304544: Move Wmfdata-Python from Github to Gitlab.Oct 25 2024, 7:48 PM
Ottomata updated the task description. (Show Details)Oct 26 2024, 5:14 PM
Ottomata updated the task description. (Show Details)Jan 10 2025, 2:36 PM
BTullis closed subtask T383417: Create a container image for analytics/refinery to be used with Airflow tasks as Resolved.Jan 10 2025, 4:58 PM
Gehel removed a subtask: T383417: Create a container image for analytics/refinery to be used with Airflow tasks.Jan 17 2025, 10:06 AM
Ottomata mentioned this in T386406: Create Gitlab CI templates for JVM packages.Feb 13 2025, 8:38 PM
nshahquinn-wmf closed subtask T304544: Move Wmfdata-Python from Github to Gitlab as Resolved.Jun 26 2025, 5:42 PM
Content licensed under Creative Commons Attribution-ShareAlike (CC BY-SA) 4.0 unless otherwise noted; code licensed under GNU General Public License (GPL) 2.0 or later and other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL · Credits