Page MenuHomePhabricator

GitHub -> Phabricator import system
Open, LowestPublic

Description

There should be a way to (automatically) import code, issues and pull requests from Github to Phabricator.

Upstream: https://secure.phabricator.com/T10538

Event Timeline

Ricordisamoa updated the task description. (Show Details)
Ricordisamoa added a project: Phabricator.
Ricordisamoa changed Security from none to None.
Ricordisamoa added a subscriber: Ricordisamoa.
Qgil added a subscriber: Qgil.

Most of the Wikimedia presence in GitHub is just a mirror of Gerrit. I believe only the Language team has a couple of things only in GitHub at this point?

For the code review part, I don't think they need to do anything else than import the repository and move forward.

For the Issues part, I don't think it is worth migrating anything.

Having specific examples of projects that need to be migrated from GitHub would help assessing the relevance of this task.

Aklapper triaged this task as Lowest priority.Oct 20 2014, 11:45 AM
Ricordisamoa added a comment.EditedOct 20 2014, 11:55 AM

While 'official' repositories are just mirrored, many user-made repos use GitHub for actual Gerrit-Migration.
Most of my repositories could be hosted here, as well as loooots of Labs-related things: earwig/copyvios, tool-labs, Pathoschild/Wikimedia-contrib, etc.
IMHO most users chose GitHub over Gerrit for the cleaner GUI and more compact workflow... that we can now try to achieve with Phabricator!

Qgil added a comment.Oct 20 2014, 12:25 PM

I see, good point!

It seems that for the 'official' repositories we are not going to migrate the code review history. It is less clear at this point what we will do with the open changesets, but a possibility is to simply go through them repo by repo, and either merge, abandon, or ask the submitter to upload it to the new project in Phabricator.

If the links you have posted are representative of the number of pull requests and issues per project, then it looks like the numbers are 0-2 in each case, a manageable amount for manual solutions.

While the Bugzilla migration is being done all at once, the code review migration will be probably done repository by repository within a window of N months. It doesn't look like we will need to have any specific action for these GitHub projects. They will benefit from the same infrastructure, documentation, and support than all the rest.

demon added a subscriber: demon.Oct 20 2014, 2:34 PM
In T715#12127, @Qgil wrote:

I see, good point!

It seems that for the 'official' repositories we are not going to migrate the code review history. It is less clear at this point what we will do with the open changesets, but a possibility is to simply go through them repo by repo, and either merge, abandon, or ask the submitter to upload it to the new project in Phabricator.

It's Git, we can script the uploading of open changes to Phabricator from Gerrit once we've done a first pass at merge/abandoning them.

While 'official' repositories are just mirrored, many user-made repos use GitHub for actual Gerrit-Migration.

All gadgets now hosted on GitHub could however be hosted as a single repository.

Even if this task is not strictly related with Gitblit, it is about importing existing repositories to Wikimedia-Phabricator, which is the purpose of Gitblit-Deprecate. Associating accordingly.

Sipun added a subscriber: Sipun.Jun 24 2015, 7:52 AM
Bene awarded a token.Jun 30 2015, 9:27 PM
Bene added a subscriber: Bene.
Sitic added a subscriber: Sitic.Jul 1 2015, 8:00 PM
Restricted Application added a subscriber: Aklapper. · View Herald TranscriptJul 24 2015, 2:40 PM

I have looked into this task and have built the foundation for a new Phabricator application called PubSub. that processes GitHub JSON payloads. The way that it works is by creating a GitHub PubSubHubbub webhook with a callback URL that points to the application route + a project ID. (e.g. http://phab09.wmflabs.org/pubsub/event/5).

The request data is saved via conduit in a project custom field. The field just serves as a temporary bucket, but the write creates a project transaction with the entire payload as a newValue. The payload data can be then seen in a project specific view feed. (http://phab09.wmflabs.org/pubsub/5).

In order to allow Phabricator to parse the application/json content type, I had to patch a core configuration file. Unfortunately, Upstream is not going to support this change. See https://secure.phabricator.com/T9066.

I think that this method of allowing application/json requests to Phabricator opens up many integration possibilities. Any application that can publish events as JSON then has the possibility to link and or reference a Phabricator object.

As far as importing goes, a similar process could be used to just post the JSON formatted event history retrieved from the GitHub api on a per repository basis (i.e. /repos/:owner/:repo/issues/events). This would probably require a new Phabricator application schema for storage..

scfc added a subscriber: scfc.Aug 5 2015, 2:19 PM
greg added a subscriber: greg.Sep 9 2015, 3:28 PM

Just to be clear: The goal of this task is to import GH repos (and GH issues, maybe) to be used in Differential for code-review, right? (And also viewable through Diffusion, of course.)

Going with that assumption, this is a new feature (not something we have with Gerrit now) and thus not a blocker for the Gerrit-Migration project. Nor is it a blocker for the gitblit-deprecation (the gitblit-deprecation project should not be use for any and all code-hosting needs). I'll remove those projects but add Phabricator, and have already added Differential, since those two are the pieces that we'd need to interface heavily with to do such an import system.

Yes not a blocker for Gerrit-Migration. I think this task was about migrating projects from github to whatever Wikimedia uses. Now being able to use Differential to review commits for projects hosted on github that will remain there is a very useful feature as it would lessen the github pain, but would be a different task.

Qgil added a comment.Sep 9 2015, 4:59 PM
In T715#1620922, @greg wrote:

(and GH issues, maybe)

GitHub issues are a different story. I think I remember someone sharing some script for that in Phabricator upstream.

greg renamed this task from GitHub->Phabricator import system to GitHub -> Differential import system.Sep 9 2015, 5:01 PM
greg updated the task description. (Show Details)

Why are issues no longer part of the task?

greg added a comment.Sep 9 2015, 10:29 PM

Sorry, I took it out based on Quim's comment, I'm not against it being a part of this task :)

JanZerebecki added a comment.EditedSep 10 2015, 10:39 AM

Code can just be done with git tools, same as it already works with gerrit. Pull requests are better served by T173, unless you want to preserve comments from closed pull requests. So that makes Issues the issue of this issue.

JanZerebecki renamed this task from GitHub -> Differential import system to GitHub -> Maniphest import system.Sep 10 2015, 10:42 AM
JanZerebecki updated the task description. (Show Details)
JanZerebecki removed a project: Differential.

If someone is interested in the comments and history of pull requests a separate task is probably a good idea.

Ricordisamoa renamed this task from GitHub -> Maniphest import system to GitHub -> Phabricator import system.Sep 17 2015, 11:12 AM
Ricordisamoa updated the task description. (Show Details)

Back to the original scope.
T112870 has been created for Maniphest.

revi added a subscriber: revi.Sep 17 2015, 11:22 AM

For pull requests is T173 what you had in mind?
How does git push not support what you mean with import code?

bd808 added a subscriber: bd808.Nov 29 2015, 9:22 PM
Luke081515 moved this task from To Triage to Misc on the Phabricator board.Apr 5 2016, 4:38 PM
Aklapper updated the task description. (Show Details)Apr 29 2016, 6:52 AM
Restricted Application added a subscriber: TerraCodes. · View Herald TranscriptApr 29 2016, 6:52 AM