Page MenuHomePhabricator

Create maintenance script for migrating property terms to new schema
Closed, ResolvedPublic

Description

Use PropertyTermStoreWriter from T219295 when it is good to depend on it

Event Timeline

Don't think it is a good idea to modify the existing RebuildTermSqlIndex code. We can just create a new script. The existing code has things in there we don't need, and still having the wb_terms specific thing around might be useful to various users.

JeroenDeDauw updated the task description. (Show Details)
JeroenDeDauw added a subscriber: JeroenDeDauw.

Change 503331 had a related patch set uploaded (by Jforrester; owner: Jeroen De Dauw):
[mediawiki/extensions/Wikibase@master] Add a rebuilder script for the terms store

https://gerrit.wikimedia.org/r/503331

Change 503331 merged by jenkins-bot:
[mediawiki/extensions/Wikibase@master] Add a rebuilder script for the terms store

https://gerrit.wikimedia.org/r/503331

AFAIK the script we currently have (https://gerrit.wikimedia.org/r/c/mediawiki/extensions/Wikibase/+/505670) is sufficient for this task. It has continuation based on page id rather than property id. I figure that won't fly for items but likely is OK for properties. Do we need continuation at all for properties?

There is some issues though. Some properties result in an error, and on re-run many of them do.

When I throw the exception I get this: https://pastebin.com/ejFn1pnT

*Somehow* a duplicate entry is getting inserted. I added an extra test for this but the test is not catching the issue. https://github.com/wmde/doctrine-term-store/pull/7 Probably this is some kind of evil encoding issue.

I'm calling it a day. Current guess is that the tables are not created right because we are not using this setting in mediawiki/doctrine-connection.

(https://www.doctrine-project.org/projects/doctrine-dbal/en/2.9/reference/configuration.html)

Or maybe something is wrong here https://github.com/wmde/doctrine-term-store/blob/master/src/PackagePrivate/Doctrine/NormalizedStore.php#L100-L105

One thing left in the maintenance script itself: support property id ranges. I will work on this today

Change 507954 had a related patch set uploaded (by Jeroen De Dauw; owner: Jeroen De Dauw):
[mediawiki/extensions/Wikibase@master] Use property id based limits in rebuildPropertyTerms.php

https://gerrit.wikimedia.org/r/507954

Change 507954 abandoned by Jeroen De Dauw:
Use property id based limits in rebuildPropertyTerms.php

https://gerrit.wikimedia.org/r/507954