Page MenuHomePhabricator

Trello -> Phabricator migration script
Closed, ResolvedPublic

Description

In T36#11662, @Spage wrote:

@Qgil yes, project managers and scrum masters will have to spend time doing this. I suspect the first person who needs to migrate a lot of Trello cards (that aren't just bugzilla bugs duplicated in Trello) and who has some technical chops will hack something together, and others will improve it. Like how bingle evolved.

It's the trello_create.py script in the phabricator/tools repo, using python-phabricator, wmfphablib and reusing some of @chasemp Bugzilla migration code.

Remaining teams doing project management in Trello (see also list in T825)

Possible migration features

Need to prioritize effort vs. rewards for these:

  • create Phabricator task from Trello card with title and description
  • link to Trello card in description
  • add Trello board column to task description. (Note migrated tasks appears in the workboard's default column.)
  • convert Trello checklists
  • set Subscriber field from Trello card members and commenters.
  • add Trello label name and color to task description
  • migrate Trello comments
  • associate actions (create, comment, etc.) with actual user (instead of "trellimport")
  • set historical times on create, comments, etc. (instead of time of import)
  • migrate Trello history (edits, changes, etc.)
  • add Trello due date to task description
  • add Trello stickers to task description
  • link to Trello attachments
  • upload Trello attachments into Phabricator

Related Objects

Event Timeline

There are a very large number of changes, so older changes are hidden. Show Older Changes
Qgil raised the priority of this task from to Normal.Oct 23 2014, 5:10 AM
Qgil updated the task description. (Show Details)
Qgil added a project: Project-Management.
Qgil changed Security from none to None.
Qgil added subscribers: Quiddity, Spage, Liuxinyu970226 and 5 others.
Qgil added a comment.Oct 29 2014, 4:27 PM

Another project has migrated 40.000 Trac/Trello tickets, and they are sharing their scripts:

https://secure.phabricator.com/T3179#81309

https://secure.phabricator.com/P1407

Anybody willing to get in touch with them?

This is the blocker for UX and Design research to port our planning boards. Just a heads up.

I'm looking at this, notes at Trello to Phabricator wiki page.

cburroughs' scripts have issues: "This script uses a trello enterprise export", some missing undocumented pieces like conf/trello-scrub.yaml , etc. It might be better to start from our BZ migration code and borrow stuff from cburroughs' work. I'll talk to @chasemp

@Jaredzimmerman-WMF points out checkbox lists in Trello as something that's beyond BZ import. I'm sure there's more.

Qgil raised the priority of this task from Normal to High.Nov 27 2014, 9:25 AM

@Spage, thank you for stepping in! @chasemp is now... well, now he is enjoying the long Thanksgiving weekend, but right after that his main focus is the RT migration. Still, I hope he has some time to help you reusing his import script.

Trello to Phabricator is an interesting wiki page. Should we bring it to the Main namespace and link it properly from the corresponding Phabricator pages? It could serve as landing page for anything related t this topic, pretty much like the /versus_Bugzilla page.

Qgil moved this task from Backlog to Doing on the Project-Management board.Nov 27 2014, 9:26 AM
In T821#790802, @Qgil wrote:

Still, I hope he has some time to help you reusing his import script.

Or explore the code at https://git.wikimedia.org/tree/phabricator%2Ftools.git

In T821#790338, @Spage wrote:

cburroughs' scripts have issues: "This script uses a trello enterprise export"

This part is not a problem. Apparently "Trello Enterprise" is now called Trello Business Class. That says:

"Business Class gives you easy and powerful bulk data exports in CSV and JSON."

so presumably that is what the script needs (regardless of whether cburroughs' code is used, the Business Class export may prove useful).

WMF has the Business Class service currently (see https://trello.com/wikimediafoundation).

I have a business class account if anyone needs an export to play with

Prtksxna removed a subscriber: Prtksxna.Dec 2 2014, 5:36 AM
Qgil updated the task description. (Show Details)Dec 2 2014, 7:58 AM

If the script being discussed doesn't work out for whatever reason, the Trello API is fairly straightforward. I put together a basic Trello API wrapper for Bingle/Bugello which may be useful, though it would require some augmentation for this usecase (as I understand the API, shouldn't be too much effort): https://github.com/wikimedia/bingle/blob/master/lib/trello.py

Qgil assigned this task to Spage.Dec 4 2014, 10:43 AM

Just making clear that @Spage is working on this. If you can help him, please help him.

Qgil added a comment.Dec 10 2014, 12:42 PM

@Spage, can you post an update here?

There is an increasing interest in Trello migrations (i.e. Jaredzimmerman-WMF is happy to migrate the Design boards as soon as the script is ready). Mingle actually stepped in first with P129. And on the other hand, according to @mattflaschen the Collaboration team seems to be happy copying existing Trello cards to Phabricator manually for now... (source, board)

Spage added a subscriber: Gilles.
In T821#790802, @Qgil wrote:

Trello to Phabricator is an interesting wiki page. Should we bring it to the Main namespace

Done, https://www.mediawiki.org/wiki/Phabricator/Trello_to_Phabricator

and link it properly from the corresponding Phabricator pages? It could serve as landing page for anything related t this topic, pretty much like the /versus_Bugzilla page.

In T821#836350, @Qgil wrote:

@Spage, can you post an update here?

Sure, and it's on that page. I have some simple code in a "trabulous" branch of phabricator/tools

There is an increasing interest in Trello migrations (i.e. Jaredzimmerman-WMF is happy to migrate the Design boards as soon as the script is ready).

But @Jaredzimmerman-WMF wants attachments, comments, etc. As you note, Collaboration-Team-Triage was happy just with basic import of Title and Description, for which I used the conduit createtask form.

Mingle actually stepped in first with P129.

Yup, @Gilles code looks excellent.

And on the other hand, according to @mattflaschen the Collaboration team seems to be happy copying existing Trello cards to Phabricator manually for now... (source, board)

Currently, my trello_create.py just improves the copy and paste of Title and Description, appending a link to the trello card. See sample at T76818. Next step is to adapt some of P129's code.

Qgil added a comment.Dec 22 2014, 10:36 AM

Attachments and comments are clear requirements for a migration script, yes. About checklists, what Phabricator can offer is described in T1141.

Have you decided which route is best to take to support this. We have not one but three scripts that could be adapted for this. Is "This script uses a trello enterprise export" the only problem of cburroughs' one (the only one that has been used to migrate from Trello to Phabricator)? I think Jared mentioned that we can use an available Trello license, somewhere.

Qgil added a comment.Dec 30 2014, 9:27 AM

I have merged T36: Migrate Trello data to Phabricator here because migrating all the Trello data is not a goal in itself. At least the Collaboration team is considering to take this chance and make a clean room, cloning only the Trello cards involved in new sprints. (A similar approach is being taken by Fundraising Tech and Mingle.)

The real blocker is the Trello migration script that at least the Design team requires to migrate their tasks.

Mattflaschen-WMF added a comment.EditedDec 30 2014, 8:06 PM
In T821#948647, @Qgil wrote:

At least the Collaboration team is considering to take this chance and make a clean room, cloning only the Trello cards involved in new sprints.

We're doing that as a stopgap since neither we nor anyone else has yet written/adapted a migration script. I would prefer to migrate everything (I believe this is the general Collaboration-Team-Triage consensus as well).

In T821#939498, @Qgil wrote:

Is "This script uses a trello enterprise export" the only problem of cburroughs' one (the only one that has been used to migrate from Trello to Phabricator)?

This is not a problem at all. We have Enterprise export. See https://phabricator.wikimedia.org/T821#800192

In T821#939498, @Qgil wrote:

Attachments and comments are clear requirements for a migration script, yes.

Yup, those are what I'll work on after basic task creation.

Have you decided which route is best to take to support this.

  1. Someone exports a Trello board as JSON.
  2. Dev runs trello_create.py script in phabricator/tools repo.
  3. trello_create.py makes Phab API calls to create tasks, comments, and attachments from data in the JSON export, reusing code from the bugzilla_create and mingleterminator import scripts.
Spage updated the task description. (Show Details)Jan 6 2015, 7:38 PM
Qgil moved this task from To Triage to Doing on the Phabricator board.Jan 15 2015, 7:45 AM
Spage updated the task description. (Show Details)Jan 15 2015, 10:19 PM
Spage updated the task description. (Show Details)Jan 15 2015, 10:27 PM
Spage updated the task description. (Show Details)Jan 16 2015, 12:50 AM
Spage updated the task description. (Show Details)Jan 16 2015, 9:33 PM
Spage added projects: Research, Zero-Archive.

This handles username mappings, and detects and reports Trello labels. Still no comments, history, or attachments. See https://gerrit.wikimedia.org/r/188287

I need to migrate the tasks in the Flow backlog Trello board's Send to Phabricator - Collaboration-Team board column. Either by adding a`--column` command-line argument or just edit the JSON output. Before doing that I need to write out a Trello UID -> Phabricator task PHID map file so that an improved script could later be run that would handle comments, history, and attachments without duplicating cards.

Spage moved this task from Backlog to Doing on the ECT-February-2015 board.Feb 3 2015, 3:25 AM
Spage moved this task from Backlog to Doing on the ECT-January-2015 board.Feb 3 2015, 5:09 PM

Change 179057 had a related patch set uploaded (by Spage):
Import Trello card JSON into Phabricator

https://gerrit.wikimedia.org/r/179057

Patch-For-Review

Spage closed this task as Resolved.EditedFeb 20 2015, 8:30 PM

I used the code in https://gerrit.wikimedia.org/r/#/c/179057/ to migrate the cards in the Trello Flow backlog board's 'Send to Phabricator' column to Phabricator #§Collaboration-Team. E.g. https://trello.com/c/dTRhYBdY became T90082 , with checklist, subscribers, and label.

I'm declaring victory on this. I'm happy to help other teams run it in its current state. Migrating comments, attachments,due dates, correct author, correct timestamp, etc. is A Small Matter of Programming :-) ; someone can borrow ideas from export_trello.py, bugzilla_create.py, mingleterminator.py, etc. to make it happen.

Spage renamed this task from Trello migration script to Trello -> Phabricator migration script.Feb 20 2015, 8:37 PM
Spage updated the task description. (Show Details)
Mattflaschen-WMF reopened this task as Open.Feb 24 2015, 4:43 AM

I'm not asking for more features, but shouldn't we wait until it's merged before marking Resolved?

Change 179057 merged by Rush:
Import Trello card JSON into Phabricator

https://gerrit.wikimedia.org/r/179057

I haven't tested this at all as I don't have any trello data, but the string of dependencies leading up to https://gerrit.wikimedia.org/r/#/c/179057/ have been merged.

Mattflaschen-WMF closed this task as Resolved.Feb 26 2015, 12:31 AM
Spage added a comment.EditedFeb 27 2015, 1:51 AM

I ran the scripts to migrate the Trello "Gather backlog" board to Phabricator Gather project, one column at a time. Seemed to work, see log.

Gerrit 193316 has the updated username mappings and comment tweaks.

The latest version of the migration script links to all the Trello attachments in the task description, for example T92059.

Spage updated the task description. (Show Details)Mar 10 2015, 6:11 AM
Yurik removed a project: Zero-Archive.
Gilles removed a subscriber: Gilles.Apr 23 2015, 9:38 AM
Restricted Application added a subscriber: scfc. · View Herald TranscriptOct 6 2015, 9:12 AM