Bugzilla to Maniphest import script (tracking)
Closed, ResolvedPublic

Description

This is a tracking bug for the Bugzilla -> Maniphest (Phabricator issue tracker) import script so we can keep track of subtasks that are specifically about the automated import.

Elements of a Bugzilla bug report: see https://www.mediawiki.org/wiki/Phabricator/versus_Bugzilla#Bugzilla_data_migrated

Details

Reference
fl423

Related Objects

StatusAssignedTask
ResolvedQgil
ResolvedQgil
ResolvedQgil
Resolvedchasemp
ResolvedAklapper
Declinedchasemp
Resolvedchasemp
Resolved mmodell
Resolvedchasemp
DeclinedAklapper
Resolvedchasemp
Resolved mmodell
ResolvedAklapper
ResolvedAklapper
Declined mmodell
Resolvedchasemp
DeclinedQgil
Resolvedchasemp
InvalidQgil
Resolvedchasemp
Resolvedchasemp
Resolvedchasemp
Resolvedchasemp
DuplicateNone
Resolvedchasemp
Resolvedchasemp
Resolvedchasemp
There are a very large number of changes, so older changes are hidden. Show Older Changes

aklapper wrote on 2014-07-09 14:58:12 (UTC)

Andre, shouldn't all the tasks linked from the description become blockers of this task?

Yes, but being confused by the overuse of dependencies I will set that in one go once the list above is complete, instead of clicking around, comparing missing numbers, and trying to not mix up again the fine difference between "blocking tasks" and "blocked tasks"... :)

mattflaschen wrote on 2014-07-26 02:41:44 (UTC)

In T423#20, @Aklapper wrote:

There might be a chicken and egg problem on importing User accounts and Tickets, and what to create first.

Can someone explain this? If the script takes the approach of importing users (rather than just faking it by posting everything by import-script), I can't think of any problems that would be caused by importing users first.

Importing Bugzilla tickets then Bugzilla users would not work.

henzen wrote on 2014-07-30 06:21:19 (UTC)

Greets,

A common feature with bugz deployments is the use of top level classifications... Commonly used to separate/group bugs into companies or departments, etc, allowing you to then drill down to products, components, etc.

We use it extensively across our group of companies.

I don't see Classification being addressed as part of the import process in this task. I can see ourselves and very likely others embracing your import process to migrate from bugz to phab, and classification is a vital aspect which should not be ignored.

Regards,
Henry

aklapper wrote on 2014-07-30 10:42:20 (UTC)

In T423#45, @henzen wrote:

I don't see Classification being addressed as part of the import process in this task.

We do not have classifications enabled in Wikimedia Bugzilla so it does not affect or interest us. Anybody will be free to pick up the migration code and enhance it to suit their needs and Bugzilla configurations, but this is out of scope for Wikimedia.

noisy wrote on 2014-08-18 21:16:09 (UTC)

Is there a plan to make this migration script public? It would be very useful for other people who want to migrate from bugzilla to phabricator.

Rush wrote on 2014-08-18 21:41:35 (UTC)

@noisy

keep an eye on https://gerrit.wikimedia.org/r/#/admin/projects/phabricator/tools

eventually it will look like real tools and not just tinkering :)

but yes everything will be public

mattflaschen wrote on 2014-08-19 05:59:46 (UTC)

In T423#54, @Rush wrote:

Direct link: https://git.wikimedia.org/tree/phabricator%2Ftools.git/

but yes everything will be public

Could you also add a license to that repo?

aklapper wrote on 2014-08-22 20:58:40 (UTC)

For the records, we ran into the problem of invalid byte sequences when exporting certain tickets from Bugzilla via its XML RPC API, resulting in script crashing and token invalidation. This is tracked in https://bugzilla.wikimedia.org/show_bug.cgi?id=69747

aklapper wrote on 2014-08-31 19:31:52 (UTC)

In T423#58, @Aklapper wrote:

For the records, we ran into the problem of invalid byte sequences

And for the records of the records, I worked around that problem.

aklapper wrote on 2014-09-07 20:58:19 (UTC)

  • Phab API does not provide a way to set dependencies between tasks. @mmodell might hack together an API class so Chase's script could access edges from the API. Should this be upstreamed?
  • Realized today that script does not tackle yet that accounts in Bugzilla can be 1) disabled (no login possible and no bugmail delivered) and 2) have bugmail delivery disabled only. Related Bugzilla DB SQL queries to identify them:
    • 1) SELECT login_name FROM profiles WHERE disabledtext!='';
    • 2) SELECT login_name FROM profiles WHERE disable_mail=1;

aklapper wrote on 2014-09-08 13:22:55 (UTC)

In T423#62, @Aklapper wrote:

Related Bugzilla DB SQL queries

Requested output in https://rt.wikimedia.org/Ticket/Display.html?id=8304

Aklapper moved this task from Backlog to Doing on the Bugzilla-Migration board.Sep 18 2014, 3:29 PM
Aklapper assigned this task to chasemp.Sep 18 2014, 3:36 PM
Aklapper set Security to None.
Aklapper updated the task description. (Show Details)Sep 22 2014, 1:05 PM
Aklapper updated the task description. (Show Details)Sep 23 2014, 10:48 PM
Qgil added a subscriber: Qgil.Sep 26 2014, 3:24 PM

As discussed in previous meetings, we need to document what will happen to these values in Bugzilla reports

  • Reported: YYYY-MM-DD HH:MM UTC by Author
  • Modified: YYYY-MM-DD HH:MM UTC by Author

The dates of the comments could be kept in the fab migration, and we need to confirm that we will keep them in the Bugzilla migration as well.

Qgil added a comment.Sep 27 2014, 11:07 AM

The description of this task has been copied and formatted as table at https://www.mediawiki.org/wiki/Phabricator/versus_Bugzilla#.C2.A0Bugzilla_data_migrated

Maybe you want to delete the description here and point to the wiki page instead, just like we did with the timeline?

mattflaschen wrote on 2014-07-26 02:41:44 (UTC)

@Aklapper wrote:

There might be a chicken and egg problem on importing User accounts and Tickets, and what to create first.

Can someone explain this?

The migration of user accounts is covered in T419. Please see there (and ask there if something is unclear). Thank you!

flimport added a subscriber: scfc.Oct 7 2014, 3:01 AM
Aklapper updated the task description. (Show Details)Oct 8 2014, 6:24 PM
He7d3r added a subscriber: He7d3r.Oct 8 2014, 6:49 PM
In T259#3323, @flimport wrote:

aklapper wrote on 2014-09-07 20:58:19 (UTC)

  • Phab API does not provide a way to set dependencies between tasks. @mmodell might hack together an API class so Chase's script could access edges from the API. Should this be upstreamed?

Yes, I think so. Setting dependencies seems like general-purpose functionality that should be in the Phabricator API. Is there an upstream bug for this?

chasemp added a comment.EditedOct 10 2014, 1:10 AM

yes, it's not a matter of they don't want to do it. It's a matter of internal architecture that is in flux

https://secure.phabricator.com/T5873

This comment was removed by RandomDSdevel.
Aklapper updated the task description. (Show Details)Oct 14 2014, 10:27 AM
Aklapper updated the task description. (Show Details)Oct 14 2014, 11:02 AM
Qgil added a comment.Oct 30 2014, 9:05 PM

Let's mark "blocker freeze" for the Bugzilla migration script. I have just added the tasks that were reported in the context of Bugzilla-Preview. If there are any new blockers proposed, they will be evaluated in our Phabricator team meetings.

Jdforrester-WMF changed the status of subtask T694: Print URL, Version, etc only when they contain data from Unknown Status to Resolved.Nov 5 2014, 5:50 PM

Talked to Chase: There are some problems with some attachments marked as "obsolete" when trying to import them (attachments marked as "private" are excluded anyway). Hence we will not import attachments marked as "obsolete" in Bugzilla (Bugzilla hides them by default in the "Attachment" table of a report, providing a "Show Obsolete" link).

Note to myself: https://www.mediawiki.org/wiki/Phabricator/versus_Bugzilla needs updating for this; currently says "We will migrate all the public Bugzilla reports, including their attachments." plus the "Attachments" item in the "Bugzilla data migrated" table plus the "History of changes" item under "Missing data".

chasemp closed this task as Resolved.Nov 23 2014, 10:58 PM

well, we're here so closing.

Qgil added a comment.Nov 23 2014, 11:03 PM

@chasemp , you're so humble. At this point, I feel this import script of yours is almost a member of the team. :) Thank you very much for your work on it. I really have no idea how we would have got here without this script, and of course without you.