Page MenuHomePhabricator
Create Task
Maniphest T343707

Migrate SRE repositories to GitLab - Archiving unused Gerrit repositories
Open, MediumPublic
Actions

Description

This is a task to track archival requests for SRE repositories in Gerrit that are no longer used. See T341468: Migrate SRE repositories to GitLab for overview of the migration process.

How to archive

Modified version of Projects-Cleanup for operations/* repositories:

Wikitech docs

  • Archive documentation on wikitech: replace wikitech page with {{historical|year=2023|reason="thing" has been archived|note=See {{phab|T343707}}}}

Phabricator

  • Mark all Phabricator tasks for the project as Invalid. Add a comment pointing to this task when doing so for reference.
  • Archive Phabricator project <#ProjectX>
  • Edit Phabricator project <#ProjectX> description for the extension/skin with a link to this ticket.

Repo + Configuration (TODO: make this one step :))

  • Remove tests using the archive-repo.py script: tox -e archive-repo -- <repo> T343707.
  • Archive gerrit repository using archive.py script: archive.py --task T343707 <repo> (see README.md for setup details)

Details

SubjectRepoBranchLines +/-
GitLab Migration: wmfmariadbpy wmfdb wmfbackupsintegration/configmaster+12 -12
Archive: operations/software/hhvm_exporterintegration/configmaster+4 -4
jjb: remove cassandra-table-properties-tox-dockerintegration/configmaster+0 -7
Zuul: Archive unused gerrit repositoriesintegration/configmaster+29 -16
Customize query in gerrit

Related Objects
Search...

Event Timeline

LSobanski created this task.Aug 7 2023, 11:51 AM
Restricted Application added a subscriber: Aklapper. · View Herald TranscriptAug 7 2023, 11:51 AM
LSobanski added a parent task: T341468: Migrate SRE repositories to GitLab.Aug 7 2023, 11:52 AM
LSobanski added a subscriber: thcipriani.Aug 7 2023, 11:56 AM

Here's the first batch of repositories that can be archived:

operations/debs/check_ganglia
operations/debs/elastalert
operations/debs/ganglia
operations/debs/jmxtrans
operations/debs/logster
operations/debs/prometheus-storage-migrator
operations/debs/python-diamond
operations/debs/salt
operations/debs/txstatsd
operations/software/cassandra-metrics-collector
operations/software/cassandra-table-properties
operations/software/ganglia-logtailer
operations/software/ganglios
operations/software/gdash
operations/software/grafana
operations/software/kibana
operations/software/shinkengen
operations/software/statsdlb
operations/software/tessera

LSobanski mentioned this in T341991: Migrate SRE repositories to GitLab - operations/debs.Aug 7 2023, 11:56 AM
LSobanski mentioned this in T341504: Migrate SRE repositories to GitLab - operations/software.
LSobanski triaged this task as Medium priority.Aug 7 2023, 1:54 PM
LSobanski moved this task from Incoming to Work in Progress on the collaboration-services board.
LSobanski added a comment.Aug 11 2023, 4:35 PM

operations/debs/cqlsh4
operations/software/dbtree
operations/software/swift-ring
operations/software/swift-utils
operations/software/tendril

gerritbot added a comment.Aug 11 2023, 9:12 PM

Change 948188 had a related patch set uploaded (by Thcipriani; author: Thcipriani):

[integration/config@master] Zuul: Archive unused gerrit repositories

https://gerrit.wikimedia.org/r/948188

gerritbot added a project: Patch-For-Review.Aug 11 2023, 9:12 PM
gerritbot added a comment.Aug 11 2023, 9:34 PM

Change 948188 merged by jenkins-bot:

[integration/config@master] Zuul: Archive unused gerrit repositories

https://gerrit.wikimedia.org/r/948188

thcipriani added a comment.Aug 11 2023, 9:38 PM

All done with anything before this comment!

Maintenance_bot removed a project: Patch-For-Review.Aug 11 2023, 10:10 PM
thcipriani edited projects, added Release-Engineering-Team (Priority Backlog 📥); removed Release-Engineering-Team.Aug 30 2023, 4:44 PM
gerritbot added a comment.Sep 12 2023, 1:50 PM

Change 956890 had a related patch set uploaded (by Hashar; author: Hashar):

[integration/config@master] jjb: remove cassandra-table-properties-tox-docker

https://gerrit.wikimedia.org/r/956890

gerritbot added a project: Patch-For-Review.Sep 12 2023, 1:50 PM

Change 956890 merged by jenkins-bot:

[integration/config@master] jjb: remove cassandra-table-properties-tox-docker

https://gerrit.wikimedia.org/r/956890

Maintenance_bot removed a project: Patch-For-Review.Sep 12 2023, 2:12 PM
LSobanski added a comment.Sep 19 2023, 2:31 PM

operations/software/cassandra-twcs

Eevans mentioned this in T332961: Archive operations/software/cassandra-metrics-collector.Sep 19 2023, 4:56 PM
LSobanski added a comment.Sep 20 2023, 11:29 AM

operations/software/schema-changes

LSobanski added a comment.Sep 26 2023, 2:36 PM
This comment was removed by LSobanski.
LSobanski mentioned this in T347623: Migrate Traffic repositories from Gerrit to Gitlab.Sep 29 2023, 5:11 PM
thcipriani updated the task description. (Show Details)Sep 29 2023, 6:37 PM
thcipriani added a comment.Sep 29 2023, 9:43 PM
In T343707#9182549, @LSobanski wrote:

operations/software/schema-changes

This one seems to have migrated, so treated it a bit different. Archived on Gerrit (with note in the description) and changed diffusion monitoring observing from Gerrit → GitLab.

Peachey88 updated the task description. (Show Details)Sep 29 2023, 10:19 PM
LSobanski moved this task from Work in Progress to Work in Progress (Tracking tasks) on the collaboration-services board.Oct 6 2023, 5:32 AM
LSobanski added a comment.Nov 3 2023, 5:14 PM

operations/debs/wikistats

Dzahn awarded a token.Nov 3 2023, 5:45 PM
Dzahn subscribed.Nov 3 2023, 6:00 PM
In T343707#9305428, @LSobanski wrote:

operations/debs/wikistats

  • deactivated in diffusion (manage -> deactivate)
  • ran ./justarchive.sh operations/debs/wikistats T343707 (after also downloading utils.sh which was needed and answering Y to both questions)
hashar added a project: Projects-Cleanup.Nov 3 2023, 8:49 PM
thcipriani added a comment.Nov 3 2023, 11:34 PM
In T343707#9305534, @Dzahn wrote:
In T343707#9305428, @LSobanski wrote:

operations/debs/wikistats

  • deactivated in diffusion (manage -> deactivate)
  • ran ./justarchive.sh operations/debs/wikistats T343707 (after also downloading utils.sh which was needed and answering Y to both questions)

I didn't see that one as archived. Probably something wrong with my script that it doesn't give you error notifications, I'll do some refactoring. In the interim, I re-ran and it's archived now.

Noticed some other repos under operations/* that maybe ripe for archiving (let me know if that's true):

As far as I can tell, none of the above have tests, most have not been updated since before 2018.

Peachey88 moved this task from Untriaged to Archive repos (Gerrit) on the Projects-Cleanup board.Nov 4 2023, 4:49 AM
LSobanski added a comment.EditedNov 6 2023, 11:14 AM

I confirmed all from the above list can be archived:

  • operations/docker-images/debian
  • operations/dumps/html
  • operations/software/bird-lg
  • operations/software/certpy
  • operations/software/hhvm_exporter
  • operations/software/hhvm-dev
  • operations/software/hhvm-dev/third-party
  • operations/software/hhvm-dev/folly
  • operations/software/mattermost
  • operations/software/matterircd
LSobanski added a comment.Nov 17 2023, 12:46 PM

operations/software/wmfbackups

Aklapper added a comment.Nov 27 2023, 9:27 PM

I collided creating T351600: Archive Gerrit repositories "operations/software/hhvm-dev*" (20141017) as I hadn't inspected this task beforehand - sorry!

Can operations/dumps/html/deploy be added to the list above? Or maybe that's implicit by operations/dumps/html already being listed?

jcrespo subscribed.Dec 5 2023, 1:12 PM

I migrated:

  • operations/software/mediabackups

Question: what is the right default for the protected/main branch? "Allow maintainers to push and merge"? I assume we want to encourage everybody to use the MR workflow, but for SREs repos I see a point of emergency force push ? It would be nice to document a suggestion, even if only within SRE.

LSobanski added a comment.Dec 5 2023, 1:59 PM
In T343707#9361446, @Aklapper wrote:

I collided creating T351600: Archive Gerrit repositories "operations/software/hhvm-dev*" (20141017) as I hadn't inspected this task beforehand - sorry!

Can operations/dumps/html/deploy be added to the list above? Or maybe that's implicit by operations/dumps/html already being listed?

It's not implicit but should be archived as well, thanks for pointing it out.

LSobanski added a comment.Dec 5 2023, 2:14 PM
In T343707#9382795, @jcrespo wrote:

I migrated:

  • operations/software/mediabackups

Question: what is the right default for the protected/main branch? "Allow maintainers to push and merge"? I assume we want to encourage everybody to use the MR workflow, but for SREs repos I see a point of emergency force push ? It would be nice to document a suggestion, even if only within SRE.

cc @thcipriani for awareness.

The default setting is to allow maintainers to push and merge and set force push to off. All SREs are owners of /repos/sre so in an emergency situation would be able to enable force push for the main branch of a repository if needed - would that be sufficient from your point of view (with additional documentation on how to do it)?

jcrespo added a comment.Dec 5 2023, 4:55 PM
In T343707#9383060, @LSobanski wrote:

The default setting is to allow maintainers to push and merge and set force push to off. All SREs are owners of /repos/sre so in an emergency situation would be able to enable force push for the main branch of a repository if needed - would that be sufficient from your point of view (with additional documentation on how to do it)?

Yes, the migration process was well documented except that option, so if it can be added to the mediawiki doc (if ok in general) or linked to SRE docs (if only ok for SRE), that would fix all my doubts. Thank you!

LSobanski added a comment.Dec 11 2023, 3:53 PM

operations/software/wmfdb

LSobanski added a comment.Dec 12 2023, 11:31 AM

operations/software/wmfmariadbpy

jcrespo added a comment.Dec 12 2023, 11:37 AM

I did operations/software/pampinus

Please note both wmfbackups and mediabackups mentioned above are still pointing to an outdated revision on diffusion.

LSobanski added a comment.Dec 14 2023, 3:17 PM
In T343707#9383933, @jcrespo wrote:
In T343707#9383060, @LSobanski wrote:

The default setting is to allow maintainers to push and merge and set force push to off. All SREs are owners of /repos/sre so in an emergency situation would be able to enable force push for the main branch of a repository if needed - would that be sufficient from your point of view (with additional documentation on how to do it)?

Yes, the migration process was well documented except that option, so if it can be added to the mediawiki doc (if ok in general) or linked to SRE docs (if only ok for SRE), that would fix all my doubts. Thank you!

This will be addressed as part of the ongoing documentation update: T342317: Deployment pipeline (GitLab/Kokkuri/Blubber) documentation improvements

thcipriani updated the task description. (Show Details)Dec 16 2023, 12:30 AM
thcipriani added a comment.Dec 16 2023, 12:38 AM
In T343707#9308067, @LSobanski wrote:

I confirmed all from the above list can be archived:

  • operations/docker-images/debian

This one is done, I'll try to grab the rest next week.

thcipriani mentioned this in T351600: Archive Gerrit repositories "operations/software/hhvm-dev*" (20141017).Dec 21 2023, 11:54 PM
gerritbot added a comment.Dec 22 2023, 12:16 AM

Change 985058 had a related patch set uploaded (by Thcipriani; author: Thcipriani):

[integration/config@master] Archive: operations/software/hhvm_exporter

https://gerrit.wikimedia.org/r/985058

gerritbot added a project: Patch-For-Review.Dec 22 2023, 12:16 AM
gerritbot added a comment.Dec 22 2023, 12:21 AM

Change 985058 merged by jenkins-bot:

[integration/config@master] Archive: operations/software/hhvm_exporter

https://gerrit.wikimedia.org/r/985058

thcipriani added a comment.Dec 22 2023, 12:26 AM
In T343707#9308067, @LSobanski wrote:

I confirmed all from the above list can be archived:

  • operations/docker-images/debian
  • operations/dumps/html
  • operations/software/bird-lg
  • operations/software/certpy
  • operations/software/hhvm_exporter
  • operations/software/hhvm-dev
  • operations/software/hhvm-dev/third-party
  • operations/software/hhvm-dev/folly
  • operations/software/mattermost
  • operations/software/matterircd

Done!

In T343707#9340555, @LSobanski wrote:

operations/software/wmfbackups

In T343707#9396718, @LSobanski wrote:

operations/software/wmfdb

In T343707#9399259, @LSobanski wrote:

operations/software/wmfmariadbpy

Can you confirm these for me? These all seem to have fairly recent commits (for a repos-on-gerrit-tuned value of "recent")

Maintenance_bot removed a project: Patch-For-Review.Dec 22 2023, 12:30 AM
thcipriani updated the task description. (Show Details)Dec 22 2023, 12:39 AM
LSobanski added a comment.Jan 3 2024, 8:50 AM
In T343707#9340555, @LSobanski wrote:

operations/software/wmfbackups

In T343707#9396718, @LSobanski wrote:

operations/software/wmfdb

In T343707#9399259, @LSobanski wrote:

operations/software/wmfmariadbpy

Can you confirm these for me? These all seem to have fairly recent commits (for a repos-on-gerrit-tuned value of "recent")

The recent commits are related to migration so these three are OK to archive.

thcipriani added a comment.Jan 5 2024, 10:55 PM
In T343707#9431758, @LSobanski wrote:

The recent commits are related to migration so these three are OK to archive.

oh! These are migrations. Got it.

In T343707#9431758, @LSobanski wrote:
In T343707#9340555, @LSobanski wrote:

operations/software/wmfbackups

In T343707#9396718, @LSobanski wrote:

operations/software/wmfdb

In T343707#9399259, @LSobanski wrote:

operations/software/wmfmariadbpy

Can you confirm these for me? These all seem to have fairly recent commits (for a repos-on-gerrit-tuned value of "recent")

The recent commits are related to migration so these three are OK to archive.

These are mostly done. I need to update GitHub repos still. Waiting to do that until Phabricator has fetched successfully from the new "Observe" uri. I'll monitor.

gerritbot added a comment.Jan 5 2024, 11:00 PM

Change 988109 had a related patch set uploaded (by Thcipriani; author: Thcipriani):

[integration/config@master] GitLab Migration: wmfmariadbpy wmfdb wmfbackups

https://gerrit.wikimedia.org/r/988109

gerritbot added a project: Patch-For-Review.Jan 5 2024, 11:00 PM
gerritbot added a comment.Jan 5 2024, 11:05 PM

Change 988109 merged by jenkins-bot:

[integration/config@master] GitLab Migration: wmfmariadbpy wmfdb wmfbackups

https://gerrit.wikimedia.org/r/988109

Maintenance_bot removed a project: Patch-For-Review.Jan 5 2024, 11:30 PM
thcipriani added a comment.Jan 12 2024, 9:26 PM
In T343707#9438757, @thcipriani wrote:
In T343707#9431758, @LSobanski wrote:

The recent commits are related to migration so these three are OK to archive.

oh! These are migrations. Got it.

In T343707#9431758, @LSobanski wrote:
In T343707#9340555, @LSobanski wrote:

operations/software/wmfbackups

In T343707#9396718, @LSobanski wrote:

operations/software/wmfdb

In T343707#9399259, @LSobanski wrote:

operations/software/wmfmariadbpy

Can you confirm these for me? These all seem to have fairly recent commits (for a repos-on-gerrit-tuned value of "recent")

The recent commits are related to migration so these three are OK to archive.

These are mostly done. I need to update GitHub repos still. Waiting to do that until Phabricator has fetched successfully from the new "Observe" uri. I'll monitor.

Phabricator is fetching via the new observe URI:

I added a Mirror URI to these to github, so the main branch should show up there next. Once that happens, I'll switch the main branch on GitHub and these'll be done!

LSobanski added a comment.Jan 16 2024, 2:05 PM

Here's an unusual case: operations/debs/prometheus has been migrated but still has active sub-repositories on Gerrit.

LSobanski added a comment.Jan 25 2024, 9:16 AM

operations/debs/phalerts

Stashbot added a comment.Mar 21 2024, 2:25 PM

Mentioned in SAL (#wikimedia-releng) [2024-03-21T14:25:01Z] <hashar> github: deleted archived repos wikimedia/operations-software-matterircd and wikimedia/operations-software-mattermost # T343707

bd808 subscribed.Mar 24 2024, 11:45 PM

operations/debs/wikibugs seems to have never been used. Likely was intended for packaging the archived https://gerrit.wikimedia.org/g/wikimedia/bugzilla/wikibugs bot.

Stashbot added a comment.Mar 25 2024, 8:09 AM

Mentioned in SAL (#wikimedia-releng) [2024-03-25T08:09:26Z] <hashar> gerrit: Deleted operations/debs/wikibugs which was empty and never used # T343707

bd808 unsubscribed.Mar 25 2024, 5:35 PM
LSobanski moved this task from Work in Progress (Tracking tasks) to Backlog on the collaboration-services board.Apr 12 2024, 1:38 PM
Content licensed under Creative Commons Attribution-ShareAlike (CC BY-SA) 4.0 unless otherwise noted; code licensed under GNU General Public License (GPL) 2.0 or later and other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL