Page MenuHomePhabricator

Wire up read logic for normalized storage in Wikibase
Closed, ResolvedPublic

Description

WikibaseRepo and WikibaseClient should use the new implementations of PrefetchingTermLookup (T225001) and PropertyLabelResolver (T225003) depending on the migration settings.

Details

Related Gerrit Patches:
mediawiki/extensions/Wikibase : masterUse CachedDatabasePropertyLabelResolver depending on property terms migratin stage.
mediawiki/extensions/Wikibase : masterRename some Dispatching* services for clarification
mediawiki/extensions/WikibaseCirrusSearch : masterTests: Change DataAccessSettings constructor call
mediawiki/extensions/Wikibase : masterAdd client setting to read from normalized terms schema
mediawiki/extensions/WikibaseMediaInfo : masterUpdate DataAccessSettings constructor call
mediawiki/extensions/Wikibase : masterAdd PrefetchingPropertyTermLookup in data access
mediawiki/extensions/Wikibase : masterAdd default to ByTypeDispatchingPrefetchingTermLookup

Event Timeline

Change 516612 had a related patch set uploaded (by Lucas Werkmeister (WMDE); owner: Lucas Werkmeister (WMDE)):
[mediawiki/extensions/Wikibase@master] Rename some Dispatching* services for clarification

https://gerrit.wikimedia.org/r/516612

It looks like the current BufferingTermLookup is set up in two places:

  • In data-access/src/PerRepositoryServiceWiring.php, which is eventually used by MultipleRepositoryAwareWikibaseServices and used if useEntitySourceBasedFederation is false.
  • In data-access/src/SingleEntitySourceServices.php, which is eventually used by MultipleEntitySourceServices and used if useEntitySourceBasedFederation is true.

I have no idea why we seem to have two completely separate federated service setups. useEntitySourceBasedFederation defaults to false, but in production is set to true for Commons and Test Wikidata, as well as any beta installation, and since production Wikidata shouldn’t use federation as far as I’m aware, I guess we’re actually using entity source-based federation everywhere? Whatever that means.

I figure we still need to wire up the new logic in both places, but this looks like something that should be cleaned up, or at least documented better…

Change 516621 had a related patch set uploaded (by Lucas Werkmeister (WMDE); owner: Lucas Werkmeister (WMDE)):
[mediawiki/extensions/Wikibase@master] Add default to ByTypeDispatchingPrefetchingTermLookup

https://gerrit.wikimedia.org/r/516621

Change 516622 had a related patch set uploaded (by Lucas Werkmeister (WMDE); owner: Lucas Werkmeister (WMDE)):
[mediawiki/extensions/Wikibase@master] Add PrefetchingPropertyTermLookup in data access

https://gerrit.wikimedia.org/r/516622

Change 516621 merged by jenkins-bot:
[mediawiki/extensions/Wikibase@master] Add default to ByTypeDispatchingPrefetchingTermLookup

https://gerrit.wikimedia.org/r/516621

Change 516622 merged by jenkins-bot:
[mediawiki/extensions/Wikibase@master] Add PrefetchingPropertyTermLookup in data access

https://gerrit.wikimedia.org/r/516622

Moving back to address the remaining parts of using PropertyLabelResolver

alaa_wmde removed Lucas_Werkmeister_WMDE as the assignee of this task.Jun 12 2019, 3:15 PM

Well, this also still needs to be wired up so that DataAccessSettings::useNormalizedPropertyTerms() can actually return true. I’m still working on that part, so re-claiming the task.

Change 516790 had a related patch set uploaded (by Lucas Werkmeister (WMDE); owner: Lucas Werkmeister (WMDE)):
[mediawiki/extensions/Wikibase@master] Add client setting to read from normalized terms schema

https://gerrit.wikimedia.org/r/516790

I'm wiring up ProrpertyLabelResolver (T225003) as part of this now

PropertyLabelResolver is actually used in client, this means it has to access db using DBAccessBase, which means that has to be pushed down to all our Database implementations too and used there instead of ILoadBalancer.

PropertyLabelResolver is actually used in client, this means it has to access db using DBAccessBase, which means that has to be pushed down to all our Database implementations too and used there instead of ILoadBalancer.

Maybe we don't need that yet. Seems like we only need to get the right LoadBalancer instance and that should be enough. Let's see

Change 517148 had a related patch set uploaded (by Alaa Sarhan; owner: Alaa Sarhan):
[mediawiki/extensions/Wikibase@master] WIP. Use CachedDatabasePropertyLabelResolver depending on property terms migratin stage.

https://gerrit.wikimedia.org/r/517148

alaa_wmde added a comment.EditedJun 14 2019, 11:06 PM

PropertyLabelResolver is actually used in client, this means it has to access db using DBAccessBase, which means that has to be pushed down to all our Database implementations too and used there instead of ILoadBalancer.

Maybe we don't need that yet. Seems like we only need to get the right LoadBalancer instance and that should be enough. Let's see

I'm talking nonsense .. of course we have to support federation if the old TermSqlIndex did

Change 517430 had a related patch set uploaded (by Hoo man; owner: Hoo man):
[mediawiki/extensions/WikibaseCirrusSearch@master] Tests: Change DataAccessSettings constructor call

https://gerrit.wikimedia.org/r/517430

Change 517432 had a related patch set uploaded (by Hoo man; owner: Hoo man):
[mediawiki/extensions/WikibaseMediaInfo@master] Update DataAccessSettings constructor call

https://gerrit.wikimedia.org/r/517432

Change 517430 merged by jenkins-bot:
[mediawiki/extensions/WikibaseCirrusSearch@master] Tests: Change DataAccessSettings constructor call

https://gerrit.wikimedia.org/r/517430

Change 517432 merged by jenkins-bot:
[mediawiki/extensions/WikibaseMediaInfo@master] Update DataAccessSettings constructor call

https://gerrit.wikimedia.org/r/517432

Change 516790 merged by jenkins-bot:
[mediawiki/extensions/Wikibase@master] Add client setting to read from normalized terms schema

https://gerrit.wikimedia.org/r/516790

Addshore moved this task from incoming to in progress on the Wikidata board.Jun 21 2019, 11:25 PM
alaa_wmde closed this task as Resolved.Jun 26 2019, 6:32 AM
alaa_wmde moved this task from In Review to Done on the Wikidata wb_terms Trailblazing board.

Change 517148 merged by jenkins-bot:
[mediawiki/extensions/Wikibase@master] Use CachedDatabasePropertyLabelResolver depending on property terms migratin stage.

https://gerrit.wikimedia.org/r/517148