Page MenuHomePhabricator

Create maintenance script for migrating property terms to new schema
Closed, ResolvedPublic

Description

Use PropertyTermStoreWriter from T219295 when it is good to depend on it

Details

Related Gerrit Patches:
mediawiki/extensions/Wikibase : masterUse property id based limits in rebuildPropertyTerms.php
mediawiki/extensions/Wikibase : masterAdd a rebuilder script for the terms store

Event Timeline

Restricted Application added a subscriber: Aklapper. · View Herald TranscriptApr 2 2019, 4:20 PM
alaa_wmde updated the task description. (Show Details)Apr 2 2019, 4:24 PM
JeroenDeDauw updated the task description. (Show Details)Apr 10 2019, 7:30 AM

Don't think it is a good idea to modify the existing RebuildTermSqlIndex code. We can just create a new script. The existing code has things in there we don't need, and still having the wb_terms specific thing around might be useful to various users.

JeroenDeDauw removed JeroenDeDauw as the assignee of this task.Apr 14 2019, 5:39 PM
JeroenDeDauw updated the task description. (Show Details)
JeroenDeDauw added a subscriber: JeroenDeDauw.

Change 503331 had a related patch set uploaded (by Jforrester; owner: Jeroen De Dauw):
[mediawiki/extensions/Wikibase@master] Add a rebuilder script for the terms store

https://gerrit.wikimedia.org/r/503331

Change 503331 merged by jenkins-bot:
[mediawiki/extensions/Wikibase@master] Add a rebuilder script for the terms store

https://gerrit.wikimedia.org/r/503331

AFAIK the script we currently have (https://gerrit.wikimedia.org/r/c/mediawiki/extensions/Wikibase/+/505670) is sufficient for this task. It has continuation based on page id rather than property id. I figure that won't fly for items but likely is OK for properties. Do we need continuation at all for properties?

There is some issues though. Some properties result in an error, and on re-run many of them do.

When I throw the exception I get this: https://pastebin.com/ejFn1pnT

*Somehow* a duplicate entry is getting inserted. I added an extra test for this but the test is not catching the issue. https://github.com/wmde/doctrine-term-store/pull/7 Probably this is some kind of evil encoding issue.

JeroenDeDauw added a comment.EditedApr 23 2019, 1:03 AM

I'm calling it a day. Current guess is that the tables are not created right because we are not using this setting in mediawiki/doctrine-connection.

(https://www.doctrine-project.org/projects/doctrine-dbal/en/2.9/reference/configuration.html)

Or maybe something is wrong here https://github.com/wmde/doctrine-term-store/blob/master/src/PackagePrivate/Doctrine/NormalizedStore.php#L100-L105

One thing left in the maintenance script itself: support property id ranges. I will work on this today

Change 507954 had a related patch set uploaded (by Jeroen De Dauw; owner: Jeroen De Dauw):
[mediawiki/extensions/Wikibase@master] Use property id based limits in rebuildPropertyTerms.php

https://gerrit.wikimedia.org/r/507954

Change 507954 abandoned by Jeroen De Dauw:
Use property id based limits in rebuildPropertyTerms.php

https://gerrit.wikimedia.org/r/507954

alaa_wmde closed this task as Resolved.May 22 2019, 10:26 AM