Page MenuHomePhabricator

Migrate SRE repositories to GitLab - Archiving unused Gerrit repositories
Open, MediumPublic

Description

This is a task to track archival requests for SRE repositories in Gerrit that are no longer used. See T341468: Migrate SRE repositories to GitLab for overview of the migration process.

How to archive

Modified version of Projects-Cleanup for operations/* repositories:

Wikitech docs

  • Archive documentation on wikitech: replace wikitech page with {{historical|year=2023|reason="thing" has been archived|note=See {{phab|T343707}}}}

Phabricator

  • Mark all Phabricator tasks for the project as Invalid. Add a comment pointing to this task when doing so for reference.
  • Archive Phabricator project <#ProjectX>
  • Edit Phabricator project <#ProjectX> description for the extension/skin with a link to this ticket.

Repo + Configuration (TODO: make this one step :))

  • Remove tests using the archive-repo.py script: tox -e archive-repo -- <repo> T343707.
  • Archive gerrit repository using archive.py script: archive.py --task T343707 <repo> (see README.md for setup details)

Event Timeline

Here's the first batch of repositories that can be archived:

operations/debs/check_ganglia
operations/debs/elastalert
operations/debs/ganglia
operations/debs/jmxtrans
operations/debs/logster
operations/debs/prometheus-storage-migrator
operations/debs/python-diamond
operations/debs/salt
operations/debs/txstatsd
operations/software/cassandra-metrics-collector
operations/software/cassandra-table-properties
operations/software/ganglia-logtailer
operations/software/ganglios
operations/software/gdash
operations/software/grafana
operations/software/kibana
operations/software/shinkengen
operations/software/statsdlb
operations/software/tessera

LSobanski triaged this task as Medium priority.Aug 7 2023, 1:54 PM
LSobanski moved this task from Incoming to Work in Progress on the collaboration-services board.

operations/debs/cqlsh4
operations/software/dbtree
operations/software/swift-ring
operations/software/swift-utils
operations/software/tendril

Change 948188 had a related patch set uploaded (by Thcipriani; author: Thcipriani):

[integration/config@master] Zuul: Archive unused gerrit repositories

https://gerrit.wikimedia.org/r/948188

Change 948188 merged by jenkins-bot:

[integration/config@master] Zuul: Archive unused gerrit repositories

https://gerrit.wikimedia.org/r/948188

All done with anything before this comment!

Change 956890 had a related patch set uploaded (by Hashar; author: Hashar):

[integration/config@master] jjb: remove cassandra-table-properties-tox-docker

https://gerrit.wikimedia.org/r/956890

Change 956890 merged by jenkins-bot:

[integration/config@master] jjb: remove cassandra-table-properties-tox-docker

https://gerrit.wikimedia.org/r/956890

operations/software/cassandra-twcs

operations/software/schema-changes

This comment was removed by LSobanski.

operations/software/schema-changes

This one seems to have migrated, so treated it a bit different. Archived on Gerrit (with note in the description) and changed diffusion monitoring observing from Gerrit → GitLab.

operations/debs/wikistats

  • deactivated in diffusion (manage -> deactivate)
  • ran ./justarchive.sh operations/debs/wikistats T343707 (after also downloading utils.sh which was needed and answering Y to both questions)

operations/debs/wikistats

  • deactivated in diffusion (manage -> deactivate)
  • ran ./justarchive.sh operations/debs/wikistats T343707 (after also downloading utils.sh which was needed and answering Y to both questions)

I didn't see that one as archived. Probably something wrong with my script that it doesn't give you error notifications, I'll do some refactoring. In the interim, I re-ran and it's archived now.


Noticed some other repos under operations/* that maybe ripe for archiving (let me know if that's true):

As far as I can tell, none of the above have tests, most have not been updated since before 2018.

I confirmed all from the above list can be archived:

  • operations/docker-images/debian
  • operations/dumps/html
  • operations/software/bird-lg
  • operations/software/certpy
  • operations/software/hhvm_exporter
  • operations/software/hhvm-dev
  • operations/software/hhvm-dev/third-party
  • operations/software/hhvm-dev/folly
  • operations/software/mattermost
  • operations/software/matterircd

I collided creating T351600: Archive Gerrit repositories "operations/software/hhvm-dev*" (20141017) as I hadn't inspected this task beforehand - sorry!

Can operations/dumps/html/deploy be added to the list above? Or maybe that's implicit by operations/dumps/html already being listed?

I migrated:

  • operations/software/mediabackups

Question: what is the right default for the protected/main branch? "Allow maintainers to push and merge"? I assume we want to encourage everybody to use the MR workflow, but for SREs repos I see a point of emergency force push ? It would be nice to document a suggestion, even if only within SRE.

I collided creating T351600: Archive Gerrit repositories "operations/software/hhvm-dev*" (20141017) as I hadn't inspected this task beforehand - sorry!

Can operations/dumps/html/deploy be added to the list above? Or maybe that's implicit by operations/dumps/html already being listed?

It's not implicit but should be archived as well, thanks for pointing it out.

I migrated:

  • operations/software/mediabackups

Question: what is the right default for the protected/main branch? "Allow maintainers to push and merge"? I assume we want to encourage everybody to use the MR workflow, but for SREs repos I see a point of emergency force push ? It would be nice to document a suggestion, even if only within SRE.

cc @thcipriani for awareness.

The default setting is to allow maintainers to push and merge and set force push to off. All SREs are owners of /repos/sre so in an emergency situation would be able to enable force push for the main branch of a repository if needed - would that be sufficient from your point of view (with additional documentation on how to do it)?

The default setting is to allow maintainers to push and merge and set force push to off. All SREs are owners of /repos/sre so in an emergency situation would be able to enable force push for the main branch of a repository if needed - would that be sufficient from your point of view (with additional documentation on how to do it)?

Yes, the migration process was well documented except that option, so if it can be added to the mediawiki doc (if ok in general) or linked to SRE docs (if only ok for SRE), that would fix all my doubts. Thank you!

operations/software/wmfmariadbpy

I did operations/software/pampinus

Please note both wmfbackups and mediabackups mentioned above are still pointing to an outdated revision on diffusion.

The default setting is to allow maintainers to push and merge and set force push to off. All SREs are owners of /repos/sre so in an emergency situation would be able to enable force push for the main branch of a repository if needed - would that be sufficient from your point of view (with additional documentation on how to do it)?

Yes, the migration process was well documented except that option, so if it can be added to the mediawiki doc (if ok in general) or linked to SRE docs (if only ok for SRE), that would fix all my doubts. Thank you!

This will be addressed as part of the ongoing documentation update: T342317: Deployment pipeline (GitLab/Kokkuri/Blubber) documentation improvements

I confirmed all from the above list can be archived:

  • operations/docker-images/debian

This one is done, I'll try to grab the rest next week.

Change 985058 had a related patch set uploaded (by Thcipriani; author: Thcipriani):

[integration/config@master] Archive: operations/software/hhvm_exporter

https://gerrit.wikimedia.org/r/985058

Change 985058 merged by jenkins-bot:

[integration/config@master] Archive: operations/software/hhvm_exporter

https://gerrit.wikimedia.org/r/985058

I confirmed all from the above list can be archived:

  • operations/docker-images/debian
  • operations/dumps/html
  • operations/software/bird-lg
  • operations/software/certpy
  • operations/software/hhvm_exporter
  • operations/software/hhvm-dev
  • operations/software/hhvm-dev/third-party
  • operations/software/hhvm-dev/folly
  • operations/software/mattermost
  • operations/software/matterircd

Done!

operations/software/wmfbackups

operations/software/wmfdb

operations/software/wmfmariadbpy

Can you confirm these for me? These all seem to have fairly recent commits (for a repos-on-gerrit-tuned value of "recent")

operations/software/wmfbackups

operations/software/wmfdb

operations/software/wmfmariadbpy

Can you confirm these for me? These all seem to have fairly recent commits (for a repos-on-gerrit-tuned value of "recent")

The recent commits are related to migration so these three are OK to archive.

The recent commits are related to migration so these three are OK to archive.

oh! These are migrations. Got it.

operations/software/wmfbackups

operations/software/wmfdb

operations/software/wmfmariadbpy

Can you confirm these for me? These all seem to have fairly recent commits (for a repos-on-gerrit-tuned value of "recent")

The recent commits are related to migration so these three are OK to archive.

These are mostly done. I need to update GitHub repos still. Waiting to do that until Phabricator has fetched successfully from the new "Observe" uri. I'll monitor.

Change 988109 had a related patch set uploaded (by Thcipriani; author: Thcipriani):

[integration/config@master] GitLab Migration: wmfmariadbpy wmfdb wmfbackups

https://gerrit.wikimedia.org/r/988109

Change 988109 merged by jenkins-bot:

[integration/config@master] GitLab Migration: wmfmariadbpy wmfdb wmfbackups

https://gerrit.wikimedia.org/r/988109

The recent commits are related to migration so these three are OK to archive.

oh! These are migrations. Got it.

operations/software/wmfbackups

operations/software/wmfdb

operations/software/wmfmariadbpy

Can you confirm these for me? These all seem to have fairly recent commits (for a repos-on-gerrit-tuned value of "recent")

The recent commits are related to migration so these three are OK to archive.

These are mostly done. I need to update GitHub repos still. Waiting to do that until Phabricator has fetched successfully from the new "Observe" uri. I'll monitor.

Phabricator is fetching via the new observe URI:

I added a Mirror URI to these to github, so the main branch should show up there next. Once that happens, I'll switch the main branch on GitHub and these'll be done!

Here's an unusual case: operations/debs/prometheus has been migrated but still has active sub-repositories on Gerrit.