Page MenuHomePhabricator

Migrate active repositories in Phabricator Differential to GitLab
Closed, ResolvedPublic

Assigned To
Authored By
EddieGP
Apr 1 2018, 9:18 PM
Referenced Files
None
Tokens
"Party Time" token, awarded by thcipriani."Party Time" token, awarded by fnegri."Like" token, awarded by Jelto."Like" token, awarded by Ferdi2005."Dislike" token, awarded by valerio.bozzolan."Like" token, awarded by fgiunchedi."Like" token, awarded by Dzahn."Dislike" token, awarded by mmodell."Like" token, awarded by Samwilson.

Description

IMHO the current state is the worst possible outcome. In addition to what we already had for most (gerrit) or some (github) of our repos, we now have a third place for some others (some tools on toolforge, Scap and a few more host their code on diffusion and do code review on differential).
Especially from a new contributors perspective, this is bad. Depending on which project I want to contribute to I now have to familiarise myself with github, gerrit and/or differential. Personally, when I started contributing here about a year ago, I didn't know about any of those and wouldn't have cared which one to learn. But by now, I had to dig into differential as well as gerrit, because I wanted to contribute to different projects over time.
This is an acceptable state for a (gerrit->differential) transition period. It isn't when the transition was declined. One can argue a lot about whether gerrit or differential fits our purpose better (which was done) - but I hardly can imagine how keeping both isn't much worse than having either of them.
tl;dr: The migration gerrit->differential being declined and gerrit being supposed to stay around, we should consider to shutdown differential and migrate existing projects from there to gerrit.

(Originally posted as T119908#4096033)

See also:
T296893: Replace Diffusion integration with Gitlab integration in Striker (toolsadmin)

Related Objects

StatusSubtypeAssignedTask
OpenNone
Open Aklapper
Resolved Aklapper
Resolved Aklapper
Resolved Aklapper
Resolvedthcipriani
ResolvedPaladox
ResolvedMarcoAurelio
Resolveddduvall
Resolvedfaidon
ResolvedKrinkle
ResolvedKrenair
ResolvedLegoktm
ResolvedOttomata
DeclinedNone
Resolvedhashar
ResolvedMarcoAurelio
DeclinedNone
Resolvedkaldari
DeclinedNone
Resolvedbd808
Resolvedhashar
Resolvedbrennen
Resolvedbrennen
Resolved Aklapper
ResolvedNone
Resolvedthcipriani
Resolvedthcipriani

Event Timeline

There are a very large number of changes, so older changes are hidden. Show Older Changes

I have done parts of this in T296022 already.

All striker repos have been moved.

extension repos appear to be just mirrors.

A bunch of other active repos have been disabled. We wrote docs how to use push over https if needed.

And finally there has been one active candidate moved that poppet up after git-ssh was recently shut down for real. This was at T321350.

We are not encouraging Phabricator to be used for any new repos anymore.

And the git-ssh.wikimedia.org is completely shut down in T296022.

Anything remaining that did not have I/O deactivated would be able to push via https.

There are still active repositories in Differential unfortunately :-(

Some are listed on https://phabricator.wikimedia.org/diffusion/query/ias.VEMtSCGU/ ( Hosted in Phabricator link from https://phabricator.wikimedia.org/diffusion/

arr. ACK. "Query Overheated. This query took too long, so only some results are shown. "

arr. ACK. "Query Overheated. This query took too long, so only some results are shown. "

Same. But I have this information elsewhere.

14 repos by my count (using a script from gitlab-migration-stats)

These repos meet the following criteria:

  1. status: active
  2. they are not observing any repos; i.e., not mirrors
  3. if the name is "private n" then policy.view is not public (3 repos)
nameurlupdated_time
1phabricator-Sprint (diffusion)https://phabricator.wikimedia.org/diffusion/PHSP/phabricator-sprint.git2022-10-20 06:00:00
2Arcanist (diffusion)https://phabricator.wikimedia.org/diffusion/ARC/arcanist.git2022-10-20 06:00:00
3phabricator-extensions (diffusion)https://phabricator.wikimedia.org/diffusion/PHEX/phab-extensions.git2022-10-20 06:00:00
4phabricator-translatewiki (diffusion)https://phabricator.wikimedia.org/diffusion/PHTW/phabricator-translatewiki.git2022-10-20 06:00:00
5tools-grrrit-wm (diffusion)https://phabricator.wikimedia.org/diffusion/TGWM/tools-grrrit-wm.git2022-10-20 06:00:00
6private repo 1private repo 12022-03-03 07:00:00
7WikiSP: MediaWiki Config (diffusion)https://phabricator.wikimedia.org/diffusion/WSP/wikisp-mw-config.git2022-02-28 07:00:00
8private repo 0private repo 02022-02-22 07:00:00
9private repo 2private repo 22021-05-14 06:00:00
10phabricator-Tools (diffusion)https://phabricator.wikimedia.org/diffusion/PHTO/phabricator-tools.git2021-03-09 07:00:00
11Phabricator (diffusion)https://phabricator.wikimedia.org/diffusion/PHAB/phabricator.git2020-02-11 07:00:00
123d2png (diffusion)https://phabricator.wikimedia.org/diffusion/TDTP/3d2png.git2019-07-04 06:00:00
13SCAP Extension Framework (diffusion)https://phabricator.wikimedia.org/diffusion/SCAPEXT/scap-ext.git2019-07-04 06:00:00
14extension-2ColConflict (diffusion)https://phabricator.wikimedia.org/diffusion/ETCC/extension-2colconflict.git2019-07-04 06:00:00

see more at: data.releng.team.

Updated time is the dateModified reported by phabricator, not the repos themselves.

Awesome!

When looking at phabricator translations (T321350) I found out the Gerrit deployment repo phabricator/deployment refers to repositories hosted on Diffusion and filed T321380 which has some of the repositories from this list. I was hit by the Query overheated and missed finding the proper SQL query to extract them all.

Dzahn raised the priority of this task from Low to Medium.Oct 25 2022, 9:48 PM

14 repos by my count (using a script from gitlab-migration-stats)

We talked about this list in today's gitlab IC meeting.

I said I will try to look at all the ones that are not Phabricator itself or internal and Brennen wants to look at the Phabricator internal ones.

So going throught this list:

nameurlupdated_time
5tools-grrrit-wm (diffusion)https://phabricator.wikimedia.org/diffusion/TGWM/tools-grrrit-wm.git2022-10-20 06:00:00
7WikiSP: MediaWiki Config (diffusion)https://phabricator.wikimedia.org/diffusion/WSP/wikisp-mw-config.git2022-02-28 07:00:00
123d2png (diffusion)https://phabricator.wikimedia.org/diffusion/TDTP/3d2png.git2019-07-04 06:00:00
13SCAP Extension Framework (diffusion)https://phabricator.wikimedia.org/diffusion/SCAPEXT/scap-ext.git2019-07-04 06:00:00
14extension-2ColConflict (diffusion)https://phabricator.wikimedia.org/diffusion/ETCC/extension-2colconflict.git2019-07-04 06:00:00

3d2png is meanwhile already not active anymore. It says "has moved to Gerrit" and inactive.

  • tools-grrrit-wm - inactive
  • WikiSP: MediaWiki Config - deactivated
  • 3d2png - inactive
  • SCAP Extension Framework - deactivated
  • extension-2ColConflict - inactive
  • SCAP Extension Framework - active (scap itself recently moved to gitlab so this should also move, right?)

I just deactivated it. I only have vague memories about what it is.

Hello @Galahad we see you are using this repository: https://phabricator.wikimedia.org/source/wikisp-mw-config/ on Phabricator Diffusion. We are currently going through the very few remaining repos on Phabricator and would like to encourage moving repos to either our new self-hosted Gitlab or Gerrit. How would you feel about that? We would of course help to import the data. Best, Daniel

  • SCAP Extension Framework - active (scap itself recently moved to gitlab so this should also move, right?)

I just deactivated it. I only have vague memories about what it is.

Thanks! I think this means if we subtract the "part of Phabricator itself / internal / secret" ones from the original list of 14 then there is literally only 1 active one left that has a maintainer outside SRE or releng. And I pinged the user right above.

Hello @Galahad we see you are using this repository: https://phabricator.wikimedia.org/source/wikisp-mw-config/ on Phabricator Diffusion. We are currently going through the very few remaining repos on Phabricator and would like to encourage moving repos to either our new self-hosted Gitlab or Gerrit. How would you feel about that? We would of course help to import the data. Best, Daniel

You already merged the repos onto GitLab a few months ago. :)

You already merged the repos onto GitLab a few months ago. :)

Oh man, I know the name sounded familiar :) Sorry about the double ping. I just failed to set this to inactive then I guess. I'll fix that right now.

thank you :)

Mentioned in SAL (#wikimedia-operations) [2022-11-30T21:41:54Z] <mutante> phabricator - deactivating diffusion repo wikisp-mw-config T191182

14 repos by my count (using a script from gitlab-migration-stats)

@brennen We are now down to only things starting with phabricator-* (or Arcanist which is also part of it) and the 3 private repos.

nameurlupdated_time
1phabricator-Sprint (diffusion)https://phabricator.wikimedia.org/diffusion/PHSP/phabricator-sprint.git2022-10-20 06:00:00
2Arcanist (diffusion)https://phabricator.wikimedia.org/diffusion/ARC/arcanist.git2022-10-20 06:00:00
3phabricator-extensions (diffusion)https://phabricator.wikimedia.org/diffusion/PHEX/phab-extensions.git2022-10-20 06:00:00
4phabricator-translatewiki (diffusion)https://phabricator.wikimedia.org/diffusion/PHTW/phabricator-translatewiki.git2022-10-20 06:00:00
6private repo 1private repo 12022-03-03 07:00:00
8private repo 0private repo 02022-02-22 07:00:00
9private repo 2private repo 22021-05-14 06:00:00
10phabricator-Tools (diffusion)https://phabricator.wikimedia.org/diffusion/PHTO/phabricator-tools.git2021-03-09 07:00:00
11Phabricator (diffusion)https://phabricator.wikimedia.org/diffusion/PHAB/phabricator.git2020-02-11 07:00:00

@thcipriani Got anything about the private repos?

Without having changed anything yet, the current status looks like this. 4 out of 6 are already "Publishing Disabled". It's just that that is a separate status from being "Active". Separate button in the admin UI. So only 2 out of these are actually active with enabled publishing:

phabricator-extensions - Active
phabricator-translatewiki - Active

phabricator-Tools - Publishing Disabled
Phabricator - Publishing Disabled
phabricator-Sprint - Publishing Disabled
Arcanist - Publishing Disabled

phabricator-extensions - Active

Guess that's a question for RelEng regarding deployments etc

phabricator-translatewiki - Active

cf. T321350: Phabricator translations can no more be received due to decommissioning of git-ssh

At this point let's check one more time what "Active" really means. Because it does not mean it's not viewable when inactive and also does not mean necessarily it has no URIs to clone from and "publishing enabled" is a separate switch.

So https://secure.phabricator.com/book/phabricator/article/diffusion_managing/ says

Basics: Deactivate Repository

Repositories can be deactivated. Deactivating a repository has these effects:

    the repository will no longer be updated;
    users will no longer be able to clone/fetch/checkout the repository;
    users will no longer be able to push to the repository; and
    the repository will be hidden from view in default queries.
Dzahn changed the task status from Open to In Progress.Dec 2 2022, 11:02 PM

This one is now just waiting for the migration of the "phab-extensions" https://phabricator.wikimedia.org/source/phab-extensions/ repo to either Gerrit or Gitlab.

(cc: @brennen who wanted to look at that)

After that I expect this to be resolved.

@thcipriani using the "02-active-diffusion-repos.py" script from https://gitlab.wikimedia.org/repos/releng/gitlab-migration-stats (that's what you used, right?) I now get 0 results. Are we done?

It's just that in the UI I still see one secret repo and the steward scripts repo .. and the "query overheated" message.

Dzahn changed the task status from In Progress to Open.Mar 23 2023, 5:43 PM
Dzahn reassigned this task from Dzahn to thcipriani.
Dzahn added a subscriber: Dzahn.

Hi Tyler, just assigning to ask whether you can run that script one more time and/or have knowledge of the one remaning secret repo. Feel free to reassign of course.

Change 907947 had a related patch set uploaded (by Dzahn; author: Dzahn):

[operations/puppet@production] deployment_server: remove iegreview with source phabricator

https://gerrit.wikimedia.org/r/907947

Change 907947 merged by Dzahn:

[operations/puppet@production] deployment_server: remove iegreview with source phabricator

https://gerrit.wikimedia.org/r/907947

I think stewards have (one of the) last repos in Diffusion, because it is a private one. FYI, I filled T346937: GitLab Private Repository Request for: Wikimedia Stewards and T346924 to facilitate migrating that repository.

Per T347577#9284700 last bullet point we have 10 more non-public read/write repos in Diffusion to tackle in order to make Diffusion not host any code repository canonically. (Not directly related to using Differential as a review tool, still IMO in scope here.)

Acting as a Phabricator admin, I unlocked most non-public read&write Git repositories in Diffusion via sudo /srv/phab/phabricator/bin/policy unlock in order to investigate. When a code repository was empty, I made it viewable by public. That means we are left with:

  • R2344: not empty but had only one initial boilerplate commit by @Kearabilwe 5.5 years ago and can very likely be archived
  • R2115: to be handled by Release-Engineering-Team
  • R2992: sent an email to @Ameisenigel to consider making it public done
  • R3019: to be handled by WMDE in T349847

@Aklapper Let's just archive R2344 (I can't see it though).

R2344:
With boilerplate content containing terms like "Watches" or "Jewellery" and at least one postal address, I decided to delete R2344 instead.

aklapper@phab1004:~$     sudo /srv/phab/phabricator/bin/remove destroy R2344
Destroying PhabricatorRepository R2344...
Permanently destroyed 1 object.
      - Database records for repository "tool-wikiintent" were destroyed, but
        this script does not remove working copies on disk. If you also want
        to destroy the repository working copy, manually remove
        "/srv/repos/2344/".
Aklapper claimed this task.

We now do not canonically host code in Diffusion anymore and thus do not use Differential as a code review tool anymore.