Page MenuHomePhabricator

Use Diffusion as canonical location for browsing code repos (not gitblit)
Closed, ResolvedPublic

Description

Mirroring all Gerrit repos in Phabricator with Diffusion is a goal in itself, not directly tied to Gerrit-Migration , that would allow us to deprecate Gitblit (http://git.wikimedia.org/).

We need to make sure that there is a way to turn these mirrored repositories into Phabricator local repositories, in order to allow a smooth future migration from Gerrit.

@demon says:

I'm mainly concerned about stable URL structures and preserving references when we move a repo from being hosted in Gerrit to being hosted in Phabricator.

Now that all repos are publicly available, update:

  • all template links to GitHub for core (easy, same URL structure: {{MW file}} done, ...)
  • same, for extensions (how to get callsigns?)
  • non-template links to GitHub for core (Manual:Code done, ...)
  • non-template links to GitHub for extensions

Links to gitblit will be handled with redirects, see the specific task for that.

Event Timeline

Qgil created this task.Oct 21 2014, 1:42 AM
Qgil raised the priority of this task from to Low.
Qgil updated the task description. (Show Details)
Qgil changed Security from none to None.
Qgil added subscribers: Qgil, MZMcBride, demon.
Qgil added a comment.Oct 21 2014, 1:48 AM

Enabling Diffusion and mirroring a couple of Wikimedia is simple. This had been done in the old good fab, and it could be done right now at https://phab-01.wmflabs.org/diffusion/

However, how should we mirror all the Gerrit repositories? With a script?

demon claimed this task.Oct 21 2014, 2:03 AM
demon raised the priority of this task from Low to Normal.
demon added a project: MediaWiki-Core-Team.
Qgil awarded a token.Oct 21 2014, 2:06 AM
demon added a comment.Oct 21 2014, 2:07 AM
This comment was removed by demon.
demon added a comment.Oct 21 2014, 2:41 AM

I think we'll have to do the repositories one by one with a signup process. We need to pick names for the repos on Phabricator (they don't have to be the same, and I don't think we're going to replicate our tree-ish structure), plus callsigns. Both of these are permanent once we stop mirroring and start hosting, so they need to be chosen well.

Once imported, Phabricator will keep them up to date on its own.

In T752#12599, @Chad wrote:

I think we'll have to do the repositories one by one with a signup process.

Alright, then this probably invalidates this other statement?

In T616#12534, @Qgil wrote:

We should probably check first if there are any performance issues before just plunging on ahead

Of course. Something like this should be deployed first on phab-01 or another Labs instance for testing.

If this is in fact manual work, we cannot do it twice. Unless there is a way to include all this information in the Phabricator labs Puppet rules?

In T752#12673, @Qgil wrote:
In T616#12534, @Qgil wrote:

We should probably check first if there are any performance issues before just plunging on ahead

Of course. Something like this should be deployed first on phab-01 or another Labs instance for testing.

If this is in fact manual work, we cannot do it twice. Unless there is a way to include all this information in the Phabricator labs Puppet rules?

My comment was about the use of Diffusion as a replacement for GitBlit (read-only Web browsing of repos), and that we should quickly sanity-check performance before just doing it for all repos. However, you can have lots of read-only copies of a repo, so it's not urgent to switch GitBlit off. We can do it a few dozen times (but let's not).

In T752#12673, @Qgil wrote:
In T752#12599, @Chad wrote:

I think we'll have to do the repositories one by one with a signup process.

Alright, then this probably invalidates this other statement?

No, this comment is about switching to Diffusion as the replacement for Gerrit (master server for a repo, in which it is changed). Because you can only ever have one master, this means a breaking change:

  • manually disabling write in Gerrit for the repo
  • manually(?) enabling write in Diffusion for the repo
  • getting everyone using the repo to change their remote repo configuration
  • updating all the documentation

This is both tedious and not something we can rush, but once we've got a mostly-agreed workflow process in place (including CI integration working) we will be able to do this for a few isolated repos to verify it works, and then at some point throw the switch and make everyone switch.

demon added a comment.Oct 21 2014, 6:05 PM
In T752#12673, @Qgil wrote:
In T616#12534, @Qgil wrote:

We should probably check first if there are any performance issues before just plunging on ahead

Of course. Something like this should be deployed first on phab-01 or another Labs instance for testing.

If this is in fact manual work, we cannot do it twice. Unless there is a way to include all this information in the Phabricator labs Puppet rules?

My comment was about the use of Diffusion as a replacement for GitBlit (read-only Web browsing of repos), and that we should quickly sanity-check performance before just doing it for all repos. However, you can have lots of read-only copies of a repo, so it's not urgent to switch GitBlit off. We can do it a few dozen times (but let's not).

No, we can't do it a few dozen times I think. The repositories can be renamed easily enough in Phabricator (as can their clone paths). The callsign is forever and once URLs start building on them I wouldn't want to delete them for a manual rename.

In T752#12673, @Qgil wrote:
In T752#12599, @Chad wrote:

I think we'll have to do the repositories one by one with a signup process.

Alright, then this probably invalidates this other statement?

No, this comment is about switching to Diffusion as the replacement for Gerrit (master server for a repo, in which it is changed). Because you can only ever have one master, this means a breaking change:

  • manually disabling write in Gerrit for the repo
  • manually(?) enabling write in Diffusion for the repo
  • getting everyone using the repo to change their remote repo configuration
  • updating all the documentation

This is both tedious and not something we can rush, but once we've got a mostly-agreed workflow process in place (including CI integration working) we will be able to do this for a few isolated repos to verify it works, and then at some point throw the switch and make everyone switch.

We're generally in agreement here. Except I also meant it for Gitblit replacement per above.

demon added a comment.Oct 21 2014, 6:07 PM

We could automate the process to do in batches or something but we can't do it until we're settled on names for individual repos.

Yes, it's also going to identify abandoned projects ;-)

My initial proposal for repos' names and tags here:

https://www.mediawiki.org/wiki/User:Jdforrester_(WMF)/Possible_Phabricator_repo_names

Feel free to edit mercilessly.

demon added a comment.Oct 23 2014, 3:47 PM

Importing MediaWiki now since it's the one we're sure won't be namespaced and we're clear on the callsign for. Once it's working I'll open it up for viewing.

I like that top bar search by hash (e.g. 00cf19dc65) works, seemingly everywhere.

Nemo_bis updated the task description. (Show Details)Jan 24 2015, 8:47 AM
Nemo_bis updated the task description. (Show Details)Jan 24 2015, 8:56 AM
Sitic added a subscriber: Sitic.Jul 3 2015, 6:36 PM
Restricted Application added a subscriber: Aklapper. · View Herald TranscriptAug 4 2015, 4:07 PM
greg moved this task from INBOX to Next on the Release-Engineering-Team board.Sep 24 2015, 1:53 AM
greg renamed this task from Wikimedia code repository browser in Phabricator to Use Diffusion as canonical location for browsing code repos (not gitblit).Nov 6 2015, 11:44 PM
greg updated the task description. (Show Details)
demon closed this task as Resolved.May 3 2016, 3:00 PM
greg moved this task from In Progress to Done on the Gitblit-Deprecate board.Jun 9 2016, 6:47 PM