Page MenuHomePhabricator

Migrate PipelineLib repos to GitLab
Closed, ResolvedPublic

Description

Between implementing kokkuri and scaling our CI runners, we're ready to migrate any projects using our deployment pipeline from Gerrit to GitLab.

🤔 Why am I/my team on this ticket?

If you or your team were added to this ticket it's because I found a .pipeline/config.yaml somewhere in Gerrit with your name on it.

Please update the task description to edit the table below

(or leave a comment telling @thcipriani since he created this awful table [sorry]):


📢 Help edit these tables

  1. Team confirmedYES if you own all the projects/NO if I've got the wrong owner—leave a comment to explain (or move projects to the Unknown table)
  2. Needs to move—column put Move if you need to move to GitLab or Archive if the repo doesn't need to make the jump to GitLab
  3. Blockers—Can't migrate right now? Put a link to blocker task or say YES—leave a comment on this task to say more

Teams

ProjectTeam(s)/PersonTeam confirmedNeeds to move? (vs. Archive)Blockers?
wikimedia/toolhub (Last update: 2023-03-23T13:21:12+01:00)
wikimedia/developer-portal (Last update: 2023-03-23T13:23:48+01:00)
labs/striker (Last update: 2023-03-10T13:47:09-07:00)
Technical Engagement@bd808Move (T343261)T334419: Set up translatewiki.net exports to push (and merge) to Wikimedia GitLab
Toolforge repos, tracked in T334399: Move Toolforge PipelineLib repositories to GitLab instead
labs/tools/registry-admission-webhook (Last update: 2023-03-16T19:22:06+01:00)
cloud/toolforge/delete-crashing-pods (Last update: 2021-12-14T12:00:07+02:00)
cloud/toolforge/kube-container-updater (Last update: 2022-11-09T10:26:17+02:00)
cloud/toolforge/volume-admission-controller (Last update: 2023-02-20T13:10:53+02:00)
cloud/toolforge/ingress-admission-controller (Last update: 2023-02-16T15:44:54+02:00)
Cloud-ServicesYES @dcaro MoveT334399: Move Toolforge PipelineLib repositories to GitLabT341084: [toolforge] Move all the components to the gitlab ci/cd flow
wikidata/query/flink-rdf-streaming-updater (Last update: 2023-06-01T11:55:03+01:00)
migrated to /repos/search-platform/flink-rdf-streaming-updater, the gerrit repo can be archived
Discovery-SearchYES (thanks @dcausse) Moved NO
analytics/datahub This repository can now be archived
Data-EngineeringYes @BTullis Moved T341194T303381: Review and improve the build process for DataHub containers
eventgate (GitHub 🐱)
eventgate-wikimedia (Last update: 2023-02-14T11:21:04-05:00)
node-rdkafka-factory (Last update: 2022-05-05T15:14:55-07:00)
mediawiki/services/eventstreams (Last update: 2022-05-04T11:51:51-07:00)
Event-Platform
Data-Engineering
@Ottomata Move
machinelearning/liftwing/inference-services (Last update: 2023-03-23T16:03:46+00:00)
Machine-Learning-TeamYES (@elukey Moveno
mediawiki/libs/metrics-platform (Last update: 2023-03-22T16:15:29+00:00)
Metrics Platform BacklogYES ( (/^ヮ^)/*:・゚✧ @cjming )MoveNo
mediawiki/services/kartotherian (Last update: 2023-03-21T09:57:04+01:00)
mediawiki/services/geoshapes (Last update: 2021-03-03T20:59:34+01:00)
mediawiki/services/push-notifications (Last update: 2021-11-17T15:59:51-03:00)
mediawiki/services/chromium-render (Last update: 2023-03-14T17:25:56+00:00)
Content-Transform-TeamYes (@MSantos)Move
mediawiki/services/citoid (Last update: 2023-02-23T14:40:41+00:00)
mediawiki/services/zotero (Last update: 2023-02-02T16:27:02+00:00)
Editing-team
mediawiki/services/machinetranslation (Last update: 2023-03-16T09:11:57+00:00)
mediawiki/services/cxserver (Last update: 2023-03-20T14:42:43+05:30)
mediawiki/services/apertium (Last update: 2022-01-04T03:53:57+00:00)
Language-Team
mediawiki/services/mobileapps (Last update: 2023-03-23T13:19:02+01:00)
mediawiki/services/wikifeeds (Last update: 2023-03-23T12:47:03-04:00)
mediawiki/services/recommendation-api (Last update: 2022-05-05T14:26:51-07:00)
Content-Transform-Team Wikipedia-Android-App-Backlog Wikipedia-iOS-App-Backlog Yes (@MSantos) Move
mediawiki/services/servicelib-node (Last update: 2022-05-01T00:34:47+00:00)
mediawiki/services/kask (Last update: 2023-02-14T09:56:02-06:00)
mediawiki/services/restbase (Last update: 2022-10-18T16:06:07+02:00)
mediawiki/services/example-node-api (Last update: 2021-12-13T14:00:36-05:00)
generated-data-platform/aqs/device-analytics (Last update: 2023-03-06T15:41:02-06:00)
generated-data-platform/datasets/image-suggestions (Last update: 2023-03-15T15:23:58-05:00)
operations/software/thumbor-plugins
mediawiki/services/change-propagation (Last update: 2023-03-22T17:14:53+00:00)
Platform Engineering Yes (@hnowlan)MoveRESTbase may be a special case - sync goes from Github->Gerrit currently. Would love to break that relationship though.
mediawiki/services/image-suggestion-api (Last update: 2023-01-30T15:00:51-08:00)
API Platform
mediawiki/services/function-orchestrator (Last update: 2023-03-21T12:41:24+00:00)
mediawiki/services/function-evaluator (Last update: 2023-03-21T12:41:18+00:00)
Abstract Wikipedia teamYes MovedNo
mediawiki/services/wikispeech/mary-tts (Last update: 2021-05-20T14:14:43+02:00)
mediawiki/services/wikispeech/ahotts (Last update: 2021-07-06T11:21:26+02:00)
mediawiki/services/wikispeech/wikispeech-server (Last update: 2022-07-03T23:51:43+02:00)
mediawiki/services/wikispeech/mishkal (Last update: 2022-04-04T10:27:33+02:00)
mediawiki/services/wikispeech/pronlex (Last update: 2021-04-23T15:33:20+02:00)
mediawiki/services/wikispeech/symbolset (Last update: 2021-04-20T16:00:09+02:00)
Wikispeech
mediawiki/services/similar-users (Last update: 2022-08-17T10:38:29+00:00)
Anti-Harassment YesMoveNo
wikibase/termbox (Last update: 2023-03-13T14:40:52+01:00)
Wikibase Product Platform Team WPP yesmoveanswer pending
operations/software/thumbor-plugins (Last update: 2023-03-21T11:25:54+00:00)
operations/container/miscweb (Last update: 2022-05-18T17:56:55+00:00)
SRE
operations/alerts (Last update: 2023-03-23T16:53:59+00:00)
SRE Observability yesok to movewill require puppet patch to update repo URL, coordinate with @fgiunchedi
research/landing-page (Last update: 2023-03-14T11:01:09-04:00)
research/mwaddlink (Last update: 2023-02-21T12:37:30+01:00)
Research
Machine-Learning-Team
Landing page -> @fkaelin
mwaddlink -> Yes (@calbon)
Landing page -> Move
mwaddlink -> Move (T334605#8788130)
wikimedia/services/ipoid (Last update: 2023-01-05T13:06:20+00:00)
Security-Team, Anti-HarassmentYesMoveYes, scheduling the work

Individuals

ProjectTeam(s)/PersonTeam confirmedNeeds to move? (vs. Archive)Blockers?
wikimedia/portals (Last update: 2023-01-23T12:42:41-05:00)
@Jdrewniak (maybe web team as a whole?)
wikimedia/irc/ircservserv-config (Last update: 2023-02-24T16:18:42+00:00)
@Legoktm / @taavi
mediawiki/libs/Shellbox (Last update: 2023-03-12T16:12:47+00:00)
@tstarling / @LegoktmYesMove
wikidata/propertysuggester/RecommenderServer (Last update: 2022-03-02T20:04:53+01:00)
@MichaelcochezYes (also @Martaannaj ) Move
mediawiki/services/mathoid (Last update: 2023-02-21T16:47:10+00:00)
@Physikerwelt YESMoveT344747 (no real blockers, just outstanding deployment)
labs/libraryupgrader (Last update: 2023-01-29T19:40:42+00:00)
@Legoktm
labs/codesearch (Last update: 2023-03-14T20:38:29+00:00)
@Legoktm / @Ladsgroup / @Krinkle

Unknowns

ProjectTeam(s)/PersonTeam confirmedNeeds to move? (vs. Archive)Blockers?

Repos not needing migration

  1. integration/pipelinelib (Last update: 2023-03-14T16:23:44+00:00)
  2. wikimedia/production (Last update: 2023-03-23T20:55:18+00:00) (releng experiment from long ago)
  3. node-rdkafka-statsd (Last update: 2022-05-05T15:10:05-07:00) (we don't use statsd anymore, will need to remove support from code)

Repos needing to be archived

  1. mediawiki/services/apple-search (Last update: 2023-03-12T16:14:29+00:00) -> T333409: Archive the mediawiki/services/apple-search repository

What happens next?

If you're ready to move, here's the roles we'll each play

  • Release-Engineering-Team will
    • Archive your gerrit repo
    • Make you a GitLab repo
    • Add needed folks to the repo
    • Translate your .pipeline/config.yaml to work with GitLab (docs)
  • You will
    • Use GitLab and give us feedback (and we'll update our GitLab documentation/tooling as needed)
    • Be available to bug with questions if your tests break or repo ownership is confusing for RelEngers

Related Objects

Event Timeline

There are a very large number of changes, so older changes are hidden. Show Older Changes

Hello, regarding the wikibase/termbox service -- we'd be fine with a move to gitlab but have a question for ourselves to find answer for and a (minor) change that would need to happen
question: there's some testing/CI set up in the wikibase/termbox repository which currently uses WMF's Jenkins to run tests etc. We'd like to have it replicated on Gitlab before we do the switch to be certain we do not put up broken code etc. I do not know as of now what is possible CI-wise on Gitlab but should find out.
additionally: wikibase/termbox code is included in Wikibase git repository on gerrit as a git submodule. When moved to github the submodule location reference should be changed. This does not seem like a big deal but I shall make a blocker task for it nonetheless, I figure?

cheers

Change 910439 had a related patch set uploaded (by Ilias Sarantopoulos; author: Ilias Sarantopoulos):

[mediawiki/extensions/ORES@master] feat: use Lift Wing instead of ORES

https://gerrit.wikimedia.org/r/910439

The old image suggestions api (mediawiki/services/image-suggestion-api) is unlikely to receive further development. The Jan 2023 update listed in the table above was a deployment-related housekeeping update by Ahmon that, while appreciated, does not indicate code maintenance or new development. It hasn't otherwise been touched in over a year, and probably won't be.

The only reason I'm aware of to even keep this service around is that it is used in Growth's default developer setup. Per Slack conversation with @Tgr , this could be switched to the anticipated new image suggestions api contemplated under T306349: Public-facing API for querying image suggestion recommendations and submitting user feedback, once such a thing is actually created.

Given all that, is it necessary to move this one to GitLab, or can it just hang out in Gerrit while it waits to die?

Change 912386 had a related patch set uploaded (by Ahmon Dancy; author: Ahmon Dancy):

[mediawiki/services/kask@master] Add .gitlab-ci.yml

https://gerrit.wikimedia.org/r/912386

Change 912386 merged by jenkins-bot:

[mediawiki/services/kask@master] Add .gitlab-ci.yml

https://gerrit.wikimedia.org/r/912386

Will the GitHub mirrors be switched over to replicate from GitLab? This is necessary for libraries like Shellbox that use a GitHub webhook to update Packagist.

Will the GitHub mirrors be switched over to replicate from GitLab? This is necessary for libraries like Shellbox that use a GitHub webhook to update Packagist.

Mirroring is built-in to GitLab but at the repo level (go to https://gitlab.wikimedia.org/repos/<repo>/-/settings/repository to set up / manage). Automating the set-up of this is I think T321597.

However, for packagist auto-publishing I'd rather that instead of relying on this hack we migrate use to the in-repo controllable GitLab service integration (see https://gitlab.wikimedia.org/repos/<repo>/-/settings/integrations/packagist/edit).

Change 908563 abandoned by Ilias Sarantopoulos:

[mediawiki/extensions/ORES@master] feat: use Lift Wing instead of ORES for fetching scores

Reason:

has too much duplicate code - this work is now covered in https://gerrit.wikimedia.org/r/c/mediawiki/extensions/ORES/+/910439/

https://gerrit.wikimedia.org/r/908563

We do our development on github. Does it make more sense to restart with a new repository on gitlab to mirror that, or better to migrate?

We do our development on github. Does it make more sense to restart with a new repository on gitlab to mirror that, or better to migrate?

You must do a mirroring, else you would loose the development history from the original repository on GitHub. Something like:

git clone --mirror https://github.com/wikimedia/<your_repo>
cd <your_repo>
git add gitlab https://gitlab.wikimedia.org/repos/<path_to_repo.git>
git remote set-url --push gitlab git@gitlab.wikimedia.org:repos/<path_to_repo.git>
git push --mirror gitlab

Note: that creates a bare repository, there is no workspace files.

With that Gitlab fork you can create a new branch and work on the Pipeline integration there. Once ready for the switch, you would:

  • resync the repositories (`git fetch --mirror origin; git push --mirror gitlab)
  • in a local check out merge the branch having the Gitlab adjustements and merge that in the HEAD branch to catch up with changes made to the "upstream" GitHub repository.

Change 915541 had a related patch set uploaded (by Ilias Sarantopoulos; author: Ilias Sarantopoulos):

[mediawiki/extensions/ORES@master] feat: hardcode threshold calls to switch to Lift Wing

https://gerrit.wikimedia.org/r/915541

I've confirmed information related to analytics/datahub above.
Whilst we could move it as-is, the repository is currently very large as it is a fork of an upstream project.

There are several problems with the way that we currently maintain our build of datahub, so I have put in a blocker which is T303381: Review and improve the build process for DataHub containers.
Ideally, I'd like to be able to address some of these problems and reduce the size of the repository at the same time, rather than bring over the whole repository as it is.

Wow, sorry for the noise by referencing wrong ticket :)

In a project's .gitlab-ci.yml, it is now possible to publish documentation and test coverage results to doc.wikimedia.org using docpub.

You can find usage examples in the README and the scap repo.

If you're interested in using docpub in your new GitLab projects and would like a hand with it, please feel free to give me a shout!

When you're ready to publish your docs/coverage with docpub, your project members will need to be added to the docpub project in order for you to be able to trigger the downstream pipeline there.

As part of using docpub in your project, please reach out to me so I can add you. Other RelEng members can also help with this in case I'm not available.

Gehel subscribed.

Removing Search Platform, our work here is done.

In Gerrit / PipelineLib workflow, the PipelineBot makes a comment in Gerrit with the newly published image tag names, example. Are there plans to implement similar functionality for GitLab merge requests?

dcaro updated the task description. (Show Details)
dcaro subscribed.
thcipriani claimed this task.

This task got too big to be useful.

I've broken down each individual row in the table from this task into into its own task and tagged teams/individuals where appropriate:

These tasks are managed on the GitLab (Pipeline Services Migration🐤) workboard.

I'm going to call this task complete since it's served it's initial purpose. I'll try to reparent subtasks on this tasks (if needed). If you see a task or repo that I missed, I'd appreciate help (or a comment).

Thanks and sorry!