Page MenuHomePhabricator
Create Task
Maniphest T341504

Migrate SRE repositories to GitLab - operations/software
Open, MediumPublic
Actions

Description

This is a task for activities related to migrating operations/software to GitLab. See T341468: Migrate SRE repositories to GitLab for global migration prerequisites and T343707: Migrate SRE repositories to GitLab - Archiving unused Gerrit repositories for the status of archival requests.

  • operations/software/atskafka
  • operations/software/benchmw
  • operations/software/bernard
  • operations/software/bird-lg
  • operations/software/bitu
  • operations/software/bitu-ldap
  • operations/software/cas-overlay-template
  • operations/software/censorship-monitoring
  • operations/software/cfssl-issuer
  • operations/software/charon
  • operations/software/coal
  • operations/software/conftool
  • operations/software/cumin
  • operations/software/debmonitor (has sub-repositories)
  • operations/software/dropwizard-metrics
  • operations/software/druid_exporter
  • operations/software/durum
  • operations/software/elasticsearch
  • operations/software/elasticsearch-tool
  • operations/software/envoyproxy
  • operations/software/etcd-mirror
  • operations/software/external-monitoring
  • operations/software/helm-state-metrics
  • operations/software/hhvm-dev (has sub-repositories)
  • operations/software/homer
  • operations/software/httpbb
  • operations/software/ircyall
  • operations/software/keyholder
  • operations/software/klaxon
  • operations/software/labsdb-auditor
  • operations/software/locust
  • operations/software/logstash-logback-encoder
  • operations/software/mailman-templates
  • operations/software/matterircd
  • operations/software/mattermost
  • operations/software/mwprof
  • operations/software/ncmonitor
  • operations/software/netbox-deploy
  • operations/software/netbox-extras
  • operations/software/netbox
  • operations/software/netbox-reports
  • operations/software/nginx
  • operations/software/pipermail-redirector
  • operations/software/prometheus-rdkafka-exporter
  • operations/software/puppet-compiler
  • operations/software/puppetdb-microproxy
  • operations/software/python-poolcounter
  • operations/software/pywmflib
  • operations/software/redactatron
  • operations/software/rescue-pxe
  • operations/software/sentry
  • operations/software/servermon
  • operations/software/service-checker
  • operations/software/spicerack
  • operations/software/statograph
  • operations/software/tegola
  • operations/software/thumbor-plugins : T344739: Migrate former Platform team service to GitLab
  • operations/software/tools-checker
  • operations/software/tools-manifest
  • operations/software/xhgui
  • operations/software/xhprof
Collaboration Services
Active
  • operations/software/gerrit (has sub-repositories) - won't be migrated but needs to remain active as long as Gerrit is used
  • operations/software/otrs
Deprecated
Data Persistence
Active
  • operations/software/mediabackups
  • operations/software/pampinus
  • operations/software/transferpy
  • operations/software/wmfbackups
  • operations/software/wmfdb
  • operations/software/wmfmariadbpy
  • dbtools directory (not a separate repository)
Deprecated
  • operations/software/cassandra-metrics-collector
  • operations/software/cassandra-table-properties
  • operations/software/dbtree
  • operations/software/swift-ring
  • operations/software/swift-utils
  • operations/software/tendril
  • operations/software/cassandra-twcs - Archival requested
  • operations/software/schema-changes - Archival requested
DC Ops
  • quotereviewer directory (not a separate repository)
Active
Deprecated
Infrastructure Foundations
Active
Deprecated
Observability
Active
  • operations/software/ecs
  • operations/software/librenms
Deprecated
  • operations/software/brrd
  • operations/software/ganglia-logtailer
  • operations/software/ganglios
  • operations/software/gdash
  • operations/software/grafana
  • operations/software/hhvm_exporter
  • operations/software/kibana
  • operations/software/prometheus_jmx_exporter
  • operations/software/shinkengen
  • operations/software/statsdlb
  • operations/software/tessera
Service Ops
Active
Deprecated
  • operations/software/kubernetes
Traffic : T347623: Migrate Traffic repositories from Gerrit to Gitlab
Active
  • operations/software/acme-chief
  • operations/software/fifo-log-demux
  • operations/software/knead-wikidough
  • operations/software/latency-measurement
  • operations/software/liberica
  • operations/software/nss-dnsdc
  • operations/software/purged
  • operations/software/varnish/libvmod-header
  • operations/software/varnish/libvmod-netmapper
  • operations/software/varnish/libvmod-querysort
  • operations/software/varnish/libvmod-re2
  • operations/software/varnish/libvmod-tbf
  • operations/software/varnish/libvmod-uuid
  • operations/software/varnish/libvmod-vslp
  • operations/software/varnish/varnishkafka
  • operations/software/varnish/varnishkafka/testing
  • operations/software/varnish/vhtcpd
Deprecated
  • operations/software/certcentral
  • operations/software/varnish
Unowned
Active
Deprecated
  • operations/software/analytics-eventlogging-maintenance
  • operations/software/blazegraph
  • operations/software/certpy
  • operations/software/tools-webservice

Details

SubjectRepoBranchLines +/-
Archive operations/software/schema-changesintegration/configmaster+4 -0
Customize query in gerrit

Related Objects
Search...

Event Timeline

LSobanski created this task.Jul 10 2023, 5:16 PM
Restricted Application added a subscriber: Aklapper. · View Herald TranscriptJul 10 2023, 5:16 PM
LSobanski added a parent task: T341468: Migrate SRE repositories to GitLab.Jul 10 2023, 5:16 PM
LSobanski mentioned this in T341468: Migrate SRE repositories to GitLab.
Dzahn subscribed.Jul 10 2023, 5:51 PM

cases where puppet code git pulls from operations/software repos:

  • klaxon: modules/klaxon/manifests/init.pp: git::clone { 'operations/software/klaxon':
  • netbox: modules/profile/manifests/netbox.pp: git::clone { 'operations/software/netbox-extras':
  • idp/cas: modules/profile/manifests/idp/build.pp: git::clone { 'operations/software/cas-overlay-template':
  • idm: modules/idm/manifests/deployment.pp: git::clone { 'operations/software/bitu':
  • swiftrepl: modules/swift/manifests/swiftrepl.pp: git::clone { 'operations/software':

So these cases are where actual prod services rely on it. The git::clone does have a parameter to change the source to gitlab.

Aklapper added a project: GitLab (Project Migration).Jul 10 2023, 7:27 PM
LSobanski triaged this task as Medium priority.Jul 12 2023, 9:19 AM
LSobanski moved this task from Incoming to Backlog on the collaboration-services board.
LSobanski updated the task description. (Show Details)Jul 12 2023, 2:22 PM
LSobanski added a subscriber: Eevans.
LSobanski updated the task description. (Show Details)Jul 13 2023, 11:24 AM
LSobanski added a comment.Jul 20 2023, 10:43 AM
In T341504#9002682, @Dzahn wrote:

cases where puppet code git pulls from operations/software repos:

  • klaxon: modules/klaxon/manifests/init.pp: git::clone { 'operations/software/klaxon':
  • netbox: modules/profile/manifests/netbox.pp: git::clone { 'operations/software/netbox-extras':
  • idp/cas: modules/profile/manifests/idp/build.pp: git::clone { 'operations/software/cas-overlay-template':
  • idm: modules/idm/manifests/deployment.pp: git::clone { 'operations/software/bitu':
  • swiftrepl: modules/swift/manifests/swiftrepl.pp: git::clone { 'operations/software':

So these cases are where actual prod services rely on it. The git::clone does have a parameter to change the source to gitlab.

Swiftrepl no longer uses this after the recent replacement in T299125: Swiftrepl doesn't work on bullseye (and swiftrepl.conf is deployed by hand).

MatthewVernon mentioned this in T342334: Remove / tidy up old swiftrepl code.Jul 20 2023, 10:54 AM
MatthewVernon subscribed.Jul 20 2023, 11:52 AM
In T341504#9030751, @LSobanski wrote:
In T341504#9002682, @Dzahn wrote:

cases where puppet code git pulls from operations/software repos:

  • swiftrepl: modules/swift/manifests/swiftrepl.pp: git::clone { 'operations/software':

So these cases are where actual prod services rely on it. The git::clone does have a parameter to change the source to gitlab.

Swiftrepl no longer uses this after the recent replacement in T299125: Swiftrepl doesn't work on bullseye (and swiftrepl.conf is deployed by hand).

...and the associated puppetry is now removed.

fgiunchedi subscribed.Aug 2 2023, 2:19 PM
colewhite subscribed.Aug 2 2023, 2:25 PM
colewhite updated the task description. (Show Details)Aug 2 2023, 11:12 PM
fgiunchedi updated the task description. (Show Details)Aug 3 2023, 7:51 AM
LSobanski updated the task description. (Show Details)Aug 3 2023, 10:24 AM
LSobanski updated the task description. (Show Details)Aug 7 2023, 11:57 AM
LSobanski updated the task description. (Show Details)Aug 7 2023, 12:09 PM
MatthewVernon updated the task description. (Show Details)Aug 7 2023, 1:51 PM
LSobanski updated the task description. (Show Details)Aug 7 2023, 1:52 PM
LSobanski updated the task description. (Show Details)
MatthewVernon updated the task description. (Show Details)Aug 7 2023, 1:54 PM
LSobanski updated the task description. (Show Details)Aug 7 2023, 3:18 PM
MatthewVernon updated the task description. (Show Details)Aug 8 2023, 1:46 PM
dancy subscribed.Aug 8 2023, 10:29 PM

We have a wrinkle:

In Gerrit a repository can have repositories underneath it in the namespace. For example, operations/software is a git repo and operations/software/acme-chief is another git repo. In GitLab, a repository cannot have a repository beneath it. Only groups can have repositories underneath them. It seems logical for repos/sre/software to be a group containing repos/sre/software/acme-chief, etc. That leaves the question of where to put the former operations/software repo's content.

LSobanski updated the task description. (Show Details)Aug 10 2023, 4:38 PM
LSobanski added a comment.Aug 11 2023, 4:29 PM

My preference would be moving the specific directories to separate repos. So far I confirmed this would be the right approach for dbtools and quotereviewer.

LSobanski updated the task description. (Show Details)Aug 11 2023, 4:36 PM
colewhite updated the task description. (Show Details)Aug 16 2023, 11:09 PM
LSobanski updated the task description. (Show Details)Aug 29 2023, 10:31 AM
LSobanski updated the task description. (Show Details)Aug 29 2023, 11:04 AM
LSobanski updated the task description. (Show Details)Aug 31 2023, 2:12 PM
LSobanski updated the task description. (Show Details)Sep 4 2023, 12:08 PM
Aklapper updated the task description. (Show Details)Sep 14 2023, 3:50 PM
Aklapper added a subtask: T346305: Move purged repository from Gerrit to GitLab.
LSobanski updated the task description. (Show Details)Sep 18 2023, 11:41 AM
LSobanski updated the task description. (Show Details)Sep 19 2023, 2:32 PM
hashar subscribed.Sep 20 2023, 12:09 PM

I have noticed repositories get migrated but are not archived in the old system so they end up still being maintained despite the Gerrit repository no more being the canonical place.

To archive a Gerrit repository we went through a process under the Projects-Cleanup project. The sidebar has a Fill an archive request which lists all the necessary steps. The form has several steps which are solely for MediaWiki material or formally removing the code, but the rest would apply to any repository being migrated. Without those steps, the Gerrit repository is still under shared maintenance, might receive patches, is mirrored to GitHub/Phabricator, get CI jobs and images updated etc :)

We can probably create a new form dedicated to migrating repositories which will list the relevant steps from the cleanup form with extra steps for the Gitlab side.

gerritbot added a comment.Sep 20 2023, 12:12 PM

Change 959204 had a related patch set uploaded (by Hashar; author: Hashar):

[integration/config@master] Archive operations/software/schema-changes

https://gerrit.wikimedia.org/r/959204

gerritbot added a project: Patch-For-Review.Sep 20 2023, 12:12 PM

Change 959204 merged by jenkins-bot:

[integration/config@master] Archive operations/software/schema-changes

https://gerrit.wikimedia.org/r/959204

Maintenance_bot removed a project: Patch-For-Review.Sep 20 2023, 12:31 PM
Ladsgroup subscribed.Sep 20 2023, 12:52 PM

My personal preference here is to actually remove the config instead of using archive template. Here is my reasoning: for a project that is finished and archived, using the template makes sense but if we want to add the archived template on every repo we move to gitlab, soon that configuration file will become a giant graveyard of archived projects.

LSobanski updated the task description. (Show Details)Sep 20 2023, 2:06 PM
Aklapper mentioned this in T347623: Migrate Traffic repositories from Gerrit to Gitlab.Sep 28 2023, 7:07 PM
Fabfur closed subtask T346305: Move purged repository from Gerrit to GitLab as Invalid.Nov 2 2023, 3:59 PM
LSobanski updated the task description. (Show Details)Nov 6 2023, 10:55 AM
LSobanski updated the task description. (Show Details)Nov 6 2023, 11:00 AM
LSobanski updated the task description. (Show Details)Nov 6 2023, 11:04 AM
LSobanski updated the task description. (Show Details)Nov 6 2023, 11:15 AM
LSobanski updated the task description. (Show Details)Nov 14 2023, 3:10 PM
LSobanski updated the task description. (Show Details)
LSobanski updated the task description. (Show Details)Nov 15 2023, 8:43 AM
LSobanski updated the task description. (Show Details)Nov 17 2023, 1:36 PM
LSobanski updated the task description. (Show Details)Dec 5 2023, 1:09 PM
LSobanski updated the task description. (Show Details)Dec 8 2023, 11:57 AM
LSobanski updated the task description. (Show Details)Dec 12 2023, 11:20 AM
Content licensed under Creative Commons Attribution-ShareAlike (CC BY-SA) 4.0 unless otherwise noted; code licensed under GNU General Public License (GPL) 2.0 or later and other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL · Credits