We put the data type in a new extension. In retrospect perhaps we should have done that right away.

Thanks @TheDJ! Fixed with https://github.com/JeroenDeDauw/Maps/commit/eab4fcaa969b7b284aaa1fc530f2b53a3fcf6f6a

I think there is wisdom in "less is more" and that talk pages do not make sense for all wikis. A clean way to disable them would be nice.

In theory it could call wfLoadExtension()

It'd be nice to replace mediawiki/mw-extension-registry-helper provides with something simpler while retaining automatic enabling of dependencies. Can an extension somehow enable other extensions that it depends on when it is enabled? (We'd make this optional so you can define things manually/explicitly for WMF wikis.)

Also seeing this issue

Is this on some roadmap?

+1

https://github.com/wmde/WikibaseDataModelServices/pull/219

Ready for review: https://gerrit.wikimedia.org/r/c/mediawiki/extensions/Wikibase/+/507954

One thing left in the maintenance script itself: support property id ranges. I will work on this today

I think something along these lines makes sense, please comment here: https://github.com/wmde/doctrine-term-store/pull/8/files

I am also not happy with this task, as again it specifies a solution, not an outcome. I much rather have "avoid expensive cleanup during the request to the degree this is possible" as acceptance criteria in the story.

We talked about this (two weeks ago?) and concluded there likely is no need to introduce anything in wikibase/term-store. I'm pretty annoyed with this task now since it specifies a solution rather than a problem that needs to be solved.

WikibaseImport contains a limited number of items and properties

https://www.mediawiki.org/wiki/Manual:Job_queue

This means we have to go with the "smart update using diff" approach, since otherwise we do not know which terms have been removed. Not clear to me it will make sense to do the cleanup in post-request, we might end up only delaying a few % of the cost. I suggest to first make it work on write and then see if we can gain a lot by moving stuff to a job.

I'm calling it a day. Current guess is that the tables are not created right because we are not using this setting in mediawiki/doctrine-connection.

There is some issues though. Some properties result in an error, and on re-run many of them do.

With https://gerrit.wikimedia.org/r/c/mediawiki/extensions/Wikibase/+/501992/ the rebuilding works using the Doctrine Term Store.

@alaa_wmde what is the status of this?

AFAIK the script we currently have (https://gerrit.wikimedia.org/r/c/mediawiki/extensions/Wikibase/+/505670) is sufficient for this task. It has continuation based on page id rather than property id. I figure that won't fly for items but likely is OK for properties. Do we need continuation at all for properties?

https://github.com/wmde/WikibaseDataModelServices/pull/216

We already did some of this while working on the property script: https://gerrit.wikimedia.org/r/c/mediawiki/extensions/Wikibase/+/505679

Yesterday while thinking about design stuff I randomly realized that we might not need a script like this. Can't we just use https://github.com/Wikidata/WikibaseImport to important a bunch of real entities? If that is too slow, then perhaps we can use https://github.com/JeroenDeDauw/Replicator to import JSON dumps.

https://github.com/wmde/WikibaseDataModelServices/pull/214

https://gerrit.wikimedia.org/r/c/mediawiki/extensions/Wikibase/+/503331

Don't think it is a good idea to modify the existing RebuildTermSqlIndex code. We can just create a new script. The existing code has things in there we don't need, and still having the wb_terms specific thing around might be useful to various users.

@alaa_wmde is this done?

First step: https://gerrit.wikimedia.org/r/c/mediawiki/extensions/Wikibase/+/502781

@alaa_wmde @Ladsgroup please to do the review. I can do the MW vendor bit but can't do the review myself of course.

"delete everything" means deleting all terms for an item/property in the item/property_terms table, rather than just those that actually need to be removed.

We won't be doing this as per https://phabricator.wikimedia.org/T220150

We figured we go with delete and insert everything. Task description updated to reflect this.

While not the best fro a design or flexibility perspective, I suspect the most pragmatic approach here is to just create a MW maintenance script in Wikibase (Repo?).

@alaa_wmde are you working on this? If so, please link the stuff you have so far.

Also part of this: https://github.com/wmde/wikibase-term-store/pull/7

First step https://gerrit.wikimedia.org/r/c/mediawiki/extensions/Wikibase/+/501992

Setting this up turned out to be easier than expected. The library has running tests on TravisCI and can transform Mysqli and SQLite (PDO) based MW Database objects.