Page MenuHomePhabricator

Create projects for Pywikibot's compat and core versions, and assign the corresponding tasks to each
Closed, ResolvedPublic

Description

We need to create tag projects for "PyWikiBot-Compat-(1.0)" and "PyWikiBot-Core-(2.0)", and right after the migration of Bugzilla reports we need to assign them the tasks belonging to each.

Previous description, kept for the record:

From https://www.mediawiki.org/wiki/Topic:S648xvfba8003tio

On 14 November 2014 10:43 PM, John Vandenberg wrote:

I see that version is not going to be migrated. I can appreciate it may not have be especially useful for MediaWiki bug reports that are 10 yeas old.

For pywikibot, which is being migrated for free, we have two different git repositories: compat (1.0) and core (2.0), and the version field in Bugzilla represents one repository or the other. I and others have spent a lot of time (think many hundreds of manhours) sorting through old bugs imported from sourceforge.net to determine whether they are v1 or v2 bugs. Or if the version is unspecified, it means someone hasnt yet determined which version the bug report applied to.

IMO the version field needs to be migrated to a proper field in Phabricator; not dumped in the description field of pywikibot bugs. It would be heartbreaking to see all bugs from the two versions merged back together. It is the common case to filter bug searches by version=v1 or version=v2, as two codebases are very different now, and even backporting bugfixes from v2 into v1 would adversely affect its stability. But v1 bugs are very useful for the many continuing users of pywikibot v1, if only to learn that a problem encountered is already known, who else has encountered it, but also there are patches, workaround, etc available in old bug reports.

Event Timeline

valhallasw updated the task description. (Show Details)
valhallasw raised the priority of this task from to Needs Triage.
valhallasw added a project: Bugzilla-Migration.
valhallasw changed Security from none to None.
valhallasw renamed this task from Version field must be migrated to Migrate "version" field.
valhallasw added a subscriber: valhallasw.

As a workaround, adding different 'version' products might be an option, i.e. adding 'Pywikibot-ComponentName' /and/ 'Pywikibot-2.0' as tags.

JAnD added a subscriber: JAnD.Nov 14 2014, 10:29 PM
revi added a subscriber: revi.Nov 14 2014, 10:36 PM
jayvdb added a subscriber: jayvdb.Nov 14 2014, 11:03 PM

I and others have spent a lot of time (think many hundreds of manhours) sorting through old bugs imported from sourceforge.net to determine whether they are v1 or v2 bugs.

That information will not get lost (but dumped into the description field). Also note that you can edit the initial description of tasks in Phabricator, in contrast to Bugzilla.

Or if the version is unspecified, it means someone hasnt yet determined which version the bug report applied to.

For the records, 37% of the Pywikibot tickets in Bugzilla have "Version: unspecified".

IMO the version field needs to be migrated to a proper field in Phabricator; not dumped in the description field of pywikibot bugs. It would be heartbreaking to see all bugs from the two versions merged back together.

The current names of the Version field in Bugzilla are "compat (1.0)" and "core (2.0)".
Phabricator only allows "Contains words" as a search criterion (and not e.g. "exactly this string with its whitespaces"). I wonder whether we should rename the two Pywikibot "Version" field values to "compat-(1.0)" "core-(2.0)" instead so they become "one word", before importing the tickets from Bugzilla next Friday?

In my humble opinion, that should sufficiently allow you to still filter the tasks. Let me know what you think.

I wonder whether we should rename the two Pywikibot "Version" field values to "compat-(1.0)" "core-(2.0)" instead so they become "one word".

Sigh, or not, as searching for "compat-(1.0)" does not list this ticket. I'm still slightly disappointed by Phabricator's search. Related: T679 and T95. Please let me investigate a bit more in the next days.

Alright, let's give Phab some time: https://phabricator.wikimedia.org/daemon/

<^d> I think the PhabricatorSearchWorker will catch up once PhabricatorRepositoryCommitHeraldWorker is empty

I and others have spent a lot of time (think many hundreds of manhours) sorting through old bugs imported from sourceforge.net to determine whether they are v1 or v2 bugs.

That information will not get lost (but dumped into the description field). Also note that you can edit the initial description of tasks in Phabricator, in contrast to Bugzilla.

You use that word 'information' ...

When you take a field, with three values, and merge it into another free text field ... in information theory you have lost information. I can show you a mathematical proof for that. :P How much information you have lost depends on the amount of structure added to the combined field.

Lots of information is being lost, but will hopefully be retained in the old bugzilla, provided it is kept around indefinitely. e.g. who set the version field is not even being migrated, and others.

Or if the version is unspecified, it means someone hasnt yet determined which version the bug report applied to.

For the records, 37% of the Pywikibot tickets in Bugzilla have "Version: unspecified".

If you look in bugzilla at only tickets 2014 onwards, there are only 27 open with version 'unspecified', and most of those actually apply to both versions (but it is unlikely nobody will do the work on v1, and the ticket will be closed when it is in v2, as effectively it is a v1 WONTFIX).

IMO the version field needs to be migrated to a proper field in Phabricator; not dumped in the description field of pywikibot bugs. It would be heartbreaking to see all bugs from the two versions merged back together.

The current names of the Version field in Bugzilla are "compat (1.0)" and "core (2.0)".
Phabricator only allows "Contains words" as a search criterion (and not e.g. "exactly this string with its whitespaces"). I wonder whether we should rename the two Pywikibot "Version" field values to "compat-(1.0)" "core-(2.0)" instead so they become "one word", before importing the tickets from Bugzilla next Friday?

In my humble opinion, that should sufficiently allow you to still filter the tasks. Let me know what you think.

I did a search for "compat-(1.0)" and it returns "No data" - it doesnt find your comment. Does that 'contains word' search only look at the 'Description' field?

I did a search for "pywikibot" and it didnt find *this* thing (T1282), despite "pywikibot" being in the 'Description' field...??

In T1282#22425, @jayvdb wrote:

I did a search for "compat-(1.0)" and it returns "No data" - it doesnt find your comment. Does that 'contains word' search only look at the 'Description' field?
I did a search for "pywikibot" and it didnt find *this* thing (T1282), despite "pywikibot" being in the 'Description' field...??

Please see my last comment above.

Qgil added a subscriber: Qgil.Nov 15 2014, 12:55 AM

As a workaround, adding different 'version' products might be an option, i.e. adding 'Pywikibot-ComponentName' /and/ 'Pywikibot-2.0' as tags.

I think this is the way to go. Today there are

964 PyWikiBot bugs today for a total of 73k don't justify adding a Version custom field for everybody. Now, PyWikiBot's case is special because Compat and Core are effectively two different products. In that case, having a tag for each makes total sense. Take into account that tags in Phabricator accomplish a very similar function than dropdown menus in Bugzilla, with the difference that they don't clutter the UI for creating and updating tasks. You can achieve the same results through search queries.

If for some reason there is a problem querying "Version: Compat (1.0)" and "Version: Core (2.0)", then we could edit the version values in Bugzilla before the migration, to convert them in a value that can be searched. If for some strange reason not even this works, then I will be happy tagging those compat & core tasks manually based on the values stored in old-bugzilla.wikimedia.org.

In T1282#22434, @Qgil wrote:

As a workaround, adding different 'version' products might be an option, i.e. adding 'Pywikibot-ComponentName' /and/ 'Pywikibot-2.0' as tags.

I think this is the way to go.

How will people report the version in new tasks? 'tags' doesnt appear on https://phabricator.wikimedia.org/maniphest/task/create/
Can tags be added to that form

964 PyWikiBot bugs today for a total of 73k don't justify adding a Version custom field for everybody.

Yea, totally understand that.

Now, PyWikiBot's case is special because Compat and Core are effectively two different products

We could just have two different products for core and compat, as Phab seems to allow multiple products in most forms. What are the drawbacks of that approach?

Qgil added a comment.EditedNov 15 2014, 1:15 AM

Yes, by 'tags' I mean 'projects that look like tags'. They are introduced in the "Projects" field. In other words, we are talking about the same. :) See https://www.mediawiki.org/wiki/Phabricator/Requesting_a_new_project#Type_of_project

According to the list of PyWikiBot components, you are going to get 20 projects with a blue suitcase type of label and a title like "Pywikibot-archivebot.py", "* Pywikibot-documentation", etc.

In addition to these, you could have two projects with a white tag type of label and the titles "Pywikibot-Compat-(1.0)" and "PyWikiBot-Core-(2.0)" that you could combine with any of the projects above. Yes, Phabricator tasks can be associated with several projects, see for instance T188, T749...

ok. projects/products/tags - all the same thing. got it.

Sounds like a good plan, except for that part about you manually tagging ~1000 tasks.

XZise added a subscriber: XZise.Nov 15 2014, 9:48 AM

Well the version is still in the description so a bot could still go through them after migration.

Also having tags we could in theory define one bug for both versions and as soon as one version is fixed that version could be removed from the tags. Although in hindsight this doesn't sound as great as I thought because the information is not that visible that a part of the bug was already fixed.

Qgil renamed this task from Migrate "version" field to Create projects for PyWikiBot's compat and core versions, and assigned the corresponding tasks to each.Nov 15 2014, 7:29 PM
Qgil updated the task description. (Show Details)
Qgil triaged this task as Normal priority.
Qgil claimed this task.
Qgil added a comment.Nov 15 2014, 7:34 PM

See the new description reflecting the current status of the discussion,

We can create the tag projects already now; one thing less. Do you really want to go for these labels?

"PyWikiBot-Compat-(1.0)" and "PyWikiBot-Core-(2.0)"

Do we really need to keep both compat/core and 1.0/2.0? At least from an external view it looks better if they only refer to compat/core or version numbers, but I really don't have any strong opinion.

Sounds like a good plan, except for that part about you manually tagging ~1000 tasks.

Agree, take this as some kind of effective reminder to myself that the search query should work. :) If not, we are talking only about 600 tasks... (no need to tag the "Unspecified", right?)

Qgil renamed this task from Create projects for PyWikiBot's compat and core versions, and assigned the corresponding tasks to each to Create projects for PyWikiBot's compat and core versions, and assign the corresponding tasks to each.
In T1282#22523, @Qgil wrote:

See the new description reflecting the current status of the discussion,

We can create the tag projects already now; one thing less. Do you really want to go for these labels?

"PyWikiBot-Compat-(1.0)" and "PyWikiBot-Core-(2.0)"

Do we really need to keep both compat/core and 1.0/2.0? At least from an external view it looks better if they only refer to compat/core or version numbers, but I really don't have any strong opinion.

Those names are indeed ugly. I'd prefer 'Pywikibot-core' for the new codebase, and either 'Pywikibot-compat' or 'Pywikipedia' for the old codebase.

No need for versions, as Pywikibot-core will hopefully have shipped a version 3 by this time next year, and all users have happily migrated from compat to a released version of core , and no longer care about whatever bugs they may have once raised against the compat branch.

jayvdb renamed this task from Create projects for PyWikiBot's compat and core versions, and assign the corresponding tasks to each to Create projects for Pywikibot's compat and core versions, and assign the corresponding tasks to each.Nov 16 2014, 2:12 AM

The current names of the Version field in Bugzilla are "compat (1.0)" and "core (2.0)".
Phabricator only allows "Contains words" as a search criterion (and not e.g. "exactly this string with its whitespaces"). I wonder whether we should rename the two Pywikibot "Version" field values to "compat-(1.0)" "core-(2.0)" instead so they become "one word", before importing the tickets from Bugzilla next Friday?

In my humble opinion, that should sufficiently allow you to still filter the tasks. Let me know what you think.

Apart from setting up projects for versions, my proposal above to replace the space by a hyphen in Pywikibot's two Bugzilla Version field entries would make searching for tasks with this version information easier (plus Phab's search index has caught up now so you can see that it would work).

Shall I do this and rename these two values in Bugzilla? Feedback very welcome.

I'd suggest to just drop the numbers and use 'core' and 'compat'. Renaming them in BZ sounds reasonable.

Qgil reassigned this task from Qgil to Aklapper.Nov 17 2014, 3:29 PM
Qgil added projects: Pywikibot-compat, Pywikibot.

Tag projects created: Pywikibot & Pywikibot-compat

Renaming versions already in Bugzilla makes total sense. @Aklapper, please remove Bugzilla-Migration when you have done this. We can keep this task open in both tag projects until the migrated tasks have been effectively assigned to these tags.

The search has been tested! Good, it looks like I will not need to do manual copy&paste. ;)

Can we not TitleCase "PyWikiBot"? It's just "Pywikibot" :)

Qgil added a comment.Nov 17 2014, 6:43 PM

My bad. Fixed.

JAnD added a comment.Nov 18 2014, 7:57 AM

I'd suggest to just drop the numbers and use 'core' and 'compat'. Renaming them in BZ sounds reasonable.

No, not good idea, because of similar first two letetrs.
When there were 'trunk' and 'rewrite' versions, they were different enough

In T1282#22667, @Qgil wrote:

Renaming versions already in Bugzilla makes total sense.

Done.

Aklapper removed Aklapper as the assignee of this task.Nov 19 2014, 5:37 PM

We can keep this task open in both tag projects until the migrated tasks have been effectively assigned to these tags.

Unassigning myself for the time being but I or anybody else can do that after migration when there's time.

Actually per pep8 (coding convention of python) name of packages like pywikibot should be all lowercase, not even Pywikibot

Qgil added a comment.Nov 19 2014, 9:41 PM

Well, at least the current project name is consistent with there references at https://www.mediawiki.org/wiki/Manual:Pywikibot

I just changed all references to "Pywikibot" in that page (except Illegal ones, like page title)

and It got reverted, I don't want to be a grammar Nazi but for the record: https://www.python.org/dev/peps/pep-0008#package-and-module-names

This task is not the place to change the project name (Im sure the migration team have better things to do), and a product name is not changed by simply editing one wiki page.
We had patches for both pywikibot and Pywikibot ( https://gerrit.wikimedia.org/r/#/c/147017/ and https://gerrit.wikimedia.org/r/#/c/146405/ ), where you put forward your opinion; Xqt picked 'Pywikibot'. If you dont like that, use the mailing list or a talk page. But I suggest that you consider this first. http://stackoverflow.com/questions/12338189/why-did-some-very-likely-pep8-aware-developers-capitalize-their-package-names

Qgil added a comment.Nov 19 2014, 10:21 PM

Ok, the great news is that you can edit these project names too. Up to you. :)

I forgot about the patches (I didn't know they got merged), I won't continue on using pywikibot. and if anyone would like to use it, opening a discussion in pywikipedia-l is recommended.

Just noting that the bugs appear to be in "Pywikibot-General" , and dont appear to have been tagged (yet) with core vs compat.
e.g. https://old-bugzilla.wikimedia.org/show_bug.cgi?id=73661 is https://phabricator.wikimedia.org/T75661 xzise's email hasnt been linked in that one.
e.g. https://old-bugzilla.wikimedia.org/show_bug.cgi?id=73628 is https://phabricator.wikimedia.org/T75628 my email hasnt been linked in that one.

Qgil added a comment.Nov 24 2014, 8:28 AM

At least yesterday night the Bugzilla migrated content was still being indexed. Before tagging anything, we need to make sure that we get the same number of tasks in Bugzilla and Phabricator for both queries.

Aklapper claimed this task.Nov 27 2014, 4:20 PM
Aklapper closed this task as Resolved.Nov 27 2014, 4:26 PM

Search indexing in Phab has finished.
124 for compat-(1.0); 512 for core-(2.0) in BZ when searching via Version field (query).
133 for compat-(1.0), 522 for core-(2.0) in Phab when searching for "contains words".

Cough. At least not less.
A hyphen splits a string into tokens in Phab. I should have used underscore instead I guess. :(
So a comment like "Is this still the case? Is this in 1.0 (compat) or 2.0 (core)?" in 57159 is also listed in Phab search results.
I'm not a fan of Phab's search (yet, see T75854) and T76152 also made this more complicated than expected requiring me to edit in chunks.

So the workaround was to export Bugzilla's search results to CSV, add 2000 to each number, construct an explicit Phab URL for 1.0 and 2.0 by using the "Task IDs" field as a URL parameter, and then mass-edit in chunks.

Verification:
126 task in pywikibot-compat (minus 1282 and 57156), 521 in pywikibot-core (minus 1315, 75704, 70986, 75754, 75723, 75868, 75934, 76141, 76144) in Phab when searching for "in all projects".
Fits.

Done now; sorry for the bug mail noise (please filter your Phab notification mail on that).

Qgil added a comment.Nov 27 2014, 8:37 PM

Wow, thank you for the explanation. And for the end result, of course.

Qgil awarded a token.Nov 27 2014, 8:38 PM