Page MenuHomePhabricator

Migrate active repositories in Phabricator Differential to GitLab
Open, MediumPublic

Assigned To
Authored By
EddieGP
Apr 1 2018, 9:18 PM
Referenced Files
None
Tokens
"Like" token, awarded by Jelto."Like" token, awarded by Ferdi2005."Dislike" token, awarded by valerio.bozzolan."Like" token, awarded by fgiunchedi."Like" token, awarded by Dzahn."Dislike" token, awarded by mmodell."Like" token, awarded by Samwilson.

Description

IMHO the current state is the worst possible outcome. In addition to what we already had for most (gerrit) or some (github) of our repos, we now have a third place for some others (some tools on toolforge, Scap and a few more host their code on diffusion and do code review on differential).
Especially from a new contributors perspective, this is bad. Depending on which project I want to contribute to I now have to familiarise myself with github, gerrit and/or differential. Personally, when I started contributing here about a year ago, I didn't know about any of those and wouldn't have cared which one to learn. But by now, I had to dig into differential as well as gerrit, because I wanted to contribute to different projects over time.
This is an acceptable state for a (gerrit->differential) transition period. It isn't when the transition was declined. One can argue a lot about whether gerrit or differential fits our purpose better (which was done) - but I hardly can imagine how keeping both isn't much worse than having either of them.
tl;dr: The migration gerrit->differential being declined and gerrit being supposed to stay around, we should consider to shutdown differential and migrate existing projects from there to gerrit.

(Originally posted as T119908#4096033)

See also:
T296893: Replace Diffusion integration with Gitlab integration in Striker (toolsadmin)

Related Objects

StatusSubtypeAssignedTask
OpenNone
OpenNone
StalledNone
Openthcipriani
Resolvedthcipriani
ResolvedPaladox
ResolvedMarcoAurelio
Resolveddduvall
Resolvedfaidon
ResolvedKrinkle
ResolvedKrenair
ResolvedLegoktm
ResolvedOttomata
DeclinedNone
Resolvedhashar
ResolvedMarcoAurelio
DeclinedNone
Resolvedkaldari
DeclinedNone
Resolvedbd808
Resolvedhashar

Event Timeline

There are a very large number of changes, so older changes are hidden. Show Older Changes

Not much to be done to Gerrit. The last active repositories in Phabricator Differential will be migrated to GitLab

Aklapper renamed this task from Stop using Differential for code review to Stop using Differential for code review and move to GitLab.Mar 22 2022, 2:13 PM
Aklapper renamed this task from Stop using Differential for code review and move to GitLab to Migrate active repositories in Phabricator Differential to GitLab.Mar 30 2022, 9:28 AM
Aklapper updated the task description. (Show Details)
Aklapper added a project: User-AKlapper.

(That may have been a mis-tag on my part, in that we probably don't need to actively migrate everything for this sprint.)

(That may have been a mis-tag on my part, in that we probably don't need to actively migrate everything for this sprint.)

I would like to move all of the Striker created repos as soon as I have a fix for T296893: Replace Diffusion integration with Gitlab integration in Striker (toolsadmin) so that no additional Differential repos will be created by Striker. If there is a common migration path prior to that I will happily use it. Otherwise I'll try to sync up with y'all to design some migration process when I get that far.

bd808 changed the task status from Stalled to Open.Sep 8 2022, 2:16 AM

Removing the stalled status. Striker no longer allows self-service creation of Diffusion repositories.

With T315706: Migrate existing Striker created Diffusion repos to GitLab 347 repos were migrated to gitlab.wikimedia.org as the primary host. The associated Diffusion repositories have been converted to read-only mirrors of their gitlab counterparts. Another 127 Diffusion repos originally created via Striker were either already converted to mirrors of other primary git repos or were somehow invalid for migration.

I have done parts of this in T296022 already.

All striker repos have been moved.

extension repos appear to be just mirrors.

A bunch of other active repos have been disabled. We wrote docs how to use push over https if needed.

And finally there has been one active candidate moved that poppet up after git-ssh was recently shut down for real. This was at T321350.

We are not encouraging Phabricator to be used for any new repos anymore.

And the git-ssh.wikimedia.org is completely shut down in T296022.

Anything remaining that did not have I/O deactivated would be able to push via https.

There are still active repositories in Differential unfortunately :-(

Some are listed on https://phabricator.wikimedia.org/diffusion/query/ias.VEMtSCGU/ ( Hosted in Phabricator link from https://phabricator.wikimedia.org/diffusion/

arr. ACK. "Query Overheated. This query took too long, so only some results are shown. "

arr. ACK. "Query Overheated. This query took too long, so only some results are shown. "

Same. But I have this information elsewhere.

14 repos by my count (using a script from gitlab-migration-stats)

These repos meet the following criteria:

  1. status: active
  2. they are not observing any repos; i.e., not mirrors
  3. if the name is "private n" then policy.view is not public (3 repos)
nameurlupdated_time
1phabricator-Sprint (diffusion)https://phabricator.wikimedia.org/diffusion/PHSP/phabricator-sprint.git2022-10-20 06:00:00
2Arcanist (diffusion)https://phabricator.wikimedia.org/diffusion/ARC/arcanist.git2022-10-20 06:00:00
3phabricator-extensions (diffusion)https://phabricator.wikimedia.org/diffusion/PHEX/phab-extensions.git2022-10-20 06:00:00
4phabricator-translatewiki (diffusion)https://phabricator.wikimedia.org/diffusion/PHTW/phabricator-translatewiki.git2022-10-20 06:00:00
5tools-grrrit-wm (diffusion)https://phabricator.wikimedia.org/diffusion/TGWM/tools-grrrit-wm.git2022-10-20 06:00:00
6private repo 1private repo 12022-03-03 07:00:00
7WikiSP: MediaWiki Config (diffusion)https://phabricator.wikimedia.org/diffusion/WSP/wikisp-mw-config.git2022-02-28 07:00:00
8private repo 0private repo 02022-02-22 07:00:00
9private repo 2private repo 22021-05-14 06:00:00
10phabricator-Tools (diffusion)https://phabricator.wikimedia.org/diffusion/PHTO/phabricator-tools.git2021-03-09 07:00:00
11Phabricator (diffusion)https://phabricator.wikimedia.org/diffusion/PHAB/phabricator.git2020-02-11 07:00:00
123d2png (diffusion)https://phabricator.wikimedia.org/diffusion/TDTP/3d2png.git2019-07-04 06:00:00
13SCAP Extension Framework (diffusion)https://phabricator.wikimedia.org/diffusion/SCAPEXT/scap-ext.git2019-07-04 06:00:00
14extension-2ColConflict (diffusion)https://phabricator.wikimedia.org/diffusion/ETCC/extension-2colconflict.git2019-07-04 06:00:00

see more at: data.releng.team.

Updated time is the dateModified reported by phabricator, not the repos themselves.

Awesome!

When looking at phabricator translations (T321350) I found out the Gerrit deployment repo phabricator/deployment refers to repositories hosted on Diffusion and filed T321380 which has some of the repositories from this list. I was hit by the Query overheated and missed finding the proper SQL query to extract them all.

Dzahn raised the priority of this task from Low to Medium.Oct 25 2022, 9:48 PM

14 repos by my count (using a script from gitlab-migration-stats)

We talked about this list in today's gitlab IC meeting.

I said I will try to look at all the ones that are not Phabricator itself or internal and Brennen wants to look at the Phabricator internal ones.

So going throught this list:

nameurlupdated_time
5tools-grrrit-wm (diffusion)https://phabricator.wikimedia.org/diffusion/TGWM/tools-grrrit-wm.git2022-10-20 06:00:00
7WikiSP: MediaWiki Config (diffusion)https://phabricator.wikimedia.org/diffusion/WSP/wikisp-mw-config.git2022-02-28 07:00:00
123d2png (diffusion)https://phabricator.wikimedia.org/diffusion/TDTP/3d2png.git2019-07-04 06:00:00
13SCAP Extension Framework (diffusion)https://phabricator.wikimedia.org/diffusion/SCAPEXT/scap-ext.git2019-07-04 06:00:00
14extension-2ColConflict (diffusion)https://phabricator.wikimedia.org/diffusion/ETCC/extension-2colconflict.git2019-07-04 06:00:00

3d2png is meanwhile already not active anymore. It says "has moved to Gerrit" and inactive.

  • tools-grrrit-wm - inactive
  • WikiSP: MediaWiki Config - deactivated
  • 3d2png - inactive
  • SCAP Extension Framework - deactivated
  • extension-2ColConflict - inactive
  • SCAP Extension Framework - active (scap itself recently moved to gitlab so this should also move, right?)

I just deactivated it. I only have vague memories about what it is.

Hello @Galahad we see you are using this repository: https://phabricator.wikimedia.org/source/wikisp-mw-config/ on Phabricator Diffusion. We are currently going through the very few remaining repos on Phabricator and would like to encourage moving repos to either our new self-hosted Gitlab or Gerrit. How would you feel about that? We would of course help to import the data. Best, Daniel

  • SCAP Extension Framework - active (scap itself recently moved to gitlab so this should also move, right?)

I just deactivated it. I only have vague memories about what it is.

Thanks! I think this means if we subtract the "part of Phabricator itself / internal / secret" ones from the original list of 14 then there is literally only 1 active one left that has a maintainer outside SRE or releng. And I pinged the user right above.

Hello @Galahad we see you are using this repository: https://phabricator.wikimedia.org/source/wikisp-mw-config/ on Phabricator Diffusion. We are currently going through the very few remaining repos on Phabricator and would like to encourage moving repos to either our new self-hosted Gitlab or Gerrit. How would you feel about that? We would of course help to import the data. Best, Daniel

You already merged the repos onto GitLab a few months ago. :)

You already merged the repos onto GitLab a few months ago. :)

Oh man, I know the name sounded familiar :) Sorry about the double ping. I just failed to set this to inactive then I guess. I'll fix that right now.

thank you :)

Mentioned in SAL (#wikimedia-operations) [2022-11-30T21:41:54Z] <mutante> phabricator - deactivating diffusion repo wikisp-mw-config T191182

14 repos by my count (using a script from gitlab-migration-stats)

@brennen We are now down to only things starting with phabricator-* (or Arcanist which is also part of it) and the 3 private repos.

nameurlupdated_time
1phabricator-Sprint (diffusion)https://phabricator.wikimedia.org/diffusion/PHSP/phabricator-sprint.git2022-10-20 06:00:00
2Arcanist (diffusion)https://phabricator.wikimedia.org/diffusion/ARC/arcanist.git2022-10-20 06:00:00
3phabricator-extensions (diffusion)https://phabricator.wikimedia.org/diffusion/PHEX/phab-extensions.git2022-10-20 06:00:00
4phabricator-translatewiki (diffusion)https://phabricator.wikimedia.org/diffusion/PHTW/phabricator-translatewiki.git2022-10-20 06:00:00
6private repo 1private repo 12022-03-03 07:00:00
8private repo 0private repo 02022-02-22 07:00:00
9private repo 2private repo 22021-05-14 06:00:00
10phabricator-Tools (diffusion)https://phabricator.wikimedia.org/diffusion/PHTO/phabricator-tools.git2021-03-09 07:00:00
11Phabricator (diffusion)https://phabricator.wikimedia.org/diffusion/PHAB/phabricator.git2020-02-11 07:00:00

@thcipriani Got anything about the private repos?

Without having changed anything yet, the current status looks like this. 4 out of 6 are already "Publishing Disabled". It's just that that is a separate status from being "Active". Separate button in the admin UI. So only 2 out of these are actually active with enabled publishing:

phabricator-extensions - Active
phabricator-translatewiki - Active

phabricator-Tools - Publishing Disabled
Phabricator - Publishing Disabled
phabricator-Sprint - Publishing Disabled
Arcanist - Publishing Disabled

phabricator-extensions - Active

Guess that's a question for RelEng regarding deployments etc

phabricator-translatewiki - Active

cf. T321350: Phabricator translations can no more be received due to decommissioning of git-ssh

At this point let's check one more time what "Active" really means. Because it does not mean it's not viewable when inactive and also does not mean necessarily it has no URIs to clone from and "publishing enabled" is a separate switch.

So https://secure.phabricator.com/book/phabricator/article/diffusion_managing/ says

Basics: Deactivate Repository

Repositories can be deactivated. Deactivating a repository has these effects:

    the repository will no longer be updated;
    users will no longer be able to clone/fetch/checkout the repository;
    users will no longer be able to push to the repository; and
    the repository will be hidden from view in default queries.
Dzahn changed the task status from Open to In Progress.Dec 2 2022, 11:02 PM

This one is now just waiting for the migration of the "phab-extensions" https://phabricator.wikimedia.org/source/phab-extensions/ repo to either Gerrit or Gitlab.

(cc: @brennen who wanted to look at that)

After that I expect this to be resolved.

@thcipriani using the "02-active-diffusion-repos.py" script from https://gitlab.wikimedia.org/repos/releng/gitlab-migration-stats (that's what you used, right?) I now get 0 results. Are we done?

It's just that in the UI I still see one secret repo and the steward scripts repo .. and the "query overheated" message.

Dzahn changed the task status from In Progress to Open.Mar 23 2023, 5:43 PM
Dzahn reassigned this task from Dzahn to thcipriani.
Dzahn added a subscriber: Dzahn.

Hi Tyler, just assigning to ask whether you can run that script one more time and/or have knowledge of the one remaning secret repo. Feel free to reassign of course.

Change 907947 had a related patch set uploaded (by Dzahn; author: Dzahn):

[operations/puppet@production] deployment_server: remove iegreview with source phabricator

https://gerrit.wikimedia.org/r/907947

Change 907947 merged by Dzahn:

[operations/puppet@production] deployment_server: remove iegreview with source phabricator

https://gerrit.wikimedia.org/r/907947