Page MenuHomePhabricator

Allow accessing Wikibase entities from multiple (Wikibase) databases
Open, HighPublic

Description

Following the discussion in T211800, assuming for now a single entity type can only be provided by single Wikibase instance, make it possible for wikibase to access entities provided by multiple sources.
For now, only direct database database access is considered.

The general idea would be to define "sources" that are providing entities of particular type. Services providing access to entity data will be defined per source (i,e. a particular service is configured to access the database of given source/instance/repository). Wikibase will determine which source-specific service to use to access data of particular entity based on the entity-type vs source configuration.

Entity ID prefixes are not considered for now, as long as there is no technical need (state January 2019) to have a single entity type provided by multiple Wikibase instances/sources./databases.

Event Timeline

There are a very large number of changes, so older changes are hidden. Show Older Changes
Jheald added a subscriber: Jheald.Tue, Jan 29, 3:15 PM

Why are we baking in the assumption that only one Wikibase instance can be associated with a particular entity type?

As posted in T211800 there is a consistent request by Commons users to be able to systematically create local Q-items associated with Commons categories, as well as some categories being sitelinked to Q-items on Wikidata.

Change 486093 merged by jenkins-bot:
[mediawiki/extensions/Wikibase@master] Added SingleEntitySourceServices and MultipleEntitySourceServices

https://gerrit.wikimedia.org/r/486093

Change 487144 had a related patch set uploaded (by WMDE-leszek; owner: WMDE-leszek):
[mediawiki/extensions/Wikibase@master] Added tests for MultipleEntitySourceServices' EntityStoreWatcher behaviour

https://gerrit.wikimedia.org/r/487144

@Jheald work tracked in this ticket is about making the existing single-source-per-entity-type implementation in Wikibase actually work, less entangled and also easier extensible to the "multiple sources per type" scenario in the future.
WMDE is going to work on the latter in 2019 (for non-Wikimedia Wikibase instances use case), and WMF's Structured Data on Commons people, I am certain, will take care of the need of the Commons community you've mentioned (thanks for your input in T211800, by the way).

This task focuses on improving the existing implementation, which has the said limitation in. It is not going to make Wikibase less available for the further extension.
I am going to rephrase the task description a bit. I can see how it can be misleading. Apologies for that.

WMDE-leszek updated the task description. (Show Details)Wed, Jan 30, 4:48 PM

Change 487148 had a related patch set uploaded (by WMDE-leszek; owner: WMDE-leszek):
[mediawiki/extensions/Wikibase@master] Added test for MultipleEntitySourceServices::getEntityRevisionLookup

https://gerrit.wikimedia.org/r/487148

Change 487032 had a related patch set uploaded (by WMDE-leszek; owner: Addshore):
[mediawiki/extensions/Wikibase@master] Followup docs & tests for *EntitySourceServices

https://gerrit.wikimedia.org/r/487032

Change 487032 merged by jenkins-bot:
[mediawiki/extensions/Wikibase@master] Followup docs & tests for *EntitySourceServices

https://gerrit.wikimedia.org/r/487032

Change 487369 had a related patch set uploaded (by WMDE-leszek; owner: WMDE-leszek):
[mediawiki/extensions/Wikibase@master] Added useEntitySourceBasedFederation flag to DataAccessSettings

https://gerrit.wikimedia.org/r/487369

Change 487372 had a related patch set uploaded (by WMDE-leszek; owner: WMDE-leszek):
[mediawiki/extensions/Wikibase@master] Added dummy EntitySource class exploding whenever used

https://gerrit.wikimedia.org/r/487372

Change 487373 had a related patch set uploaded (by WMDE-leszek; owner: WMDE-leszek):
[mediawiki/extensions/Wikibase@master] Added EntitySource to SqlEntityInfoBuilder

https://gerrit.wikimedia.org/r/487373

Change 487494 had a related patch set uploaded (by WMDE-leszek; owner: WMDE-leszek):
[mediawiki/extensions/Wikibase@master] Added EntitySource to TermSqlIndex

https://gerrit.wikimedia.org/r/487494

Change 487144 merged by jenkins-bot:
[mediawiki/extensions/Wikibase@master] Added tests for MultipleEntitySourceServices' EntityStoreWatcher behaviour

https://gerrit.wikimedia.org/r/487144

Change 485827 had a related patch set uploaded (by WMDE-leszek; owner: WMDE-leszek):
[mediawiki/extensions/Wikibase@master] Added ByTypeDispatchingPrefetchingTermLookup

https://gerrit.wikimedia.org/r/485827

Change 487607 had a related patch set uploaded (by WMDE-leszek; owner: WMDE-leszek):
[mediawiki/extensions/Wikibase@master] Added EntitySource to WikiPageEntityMetaDataLookup

https://gerrit.wikimedia.org/r/487607

Change 487636 had a related patch set uploaded (by WMDE-leszek; owner: WMDE-leszek):
[mediawiki/extensions/WikibaseMediaInfo@master] Temporarily skip tests that call WikiPageEntityMetaDataLookup's constructor

https://gerrit.wikimedia.org/r/487636

Change 485829 had a related patch set uploaded (by WMDE-leszek; owner: WMDE-leszek):
[mediawiki/extensions/Wikibase@master] Added ByTypeDispatchingEntityPrefetcher

https://gerrit.wikimedia.org/r/485829

Change 487645 had a related patch set uploaded (by WMDE-leszek; owner: WMDE-leszek):
[mediawiki/extensions/Wikibase@master] Added EntitySource to PropertyInfoTable

https://gerrit.wikimedia.org/r/487645

Change 485855 had a related patch set uploaded (by WMDE-leszek; owner: WMDE-leszek):
[mediawiki/extensions/Wikibase@master] Added PropertyInfoLookup to *EntitySourceServices

https://gerrit.wikimedia.org/r/485855

Change 485862 had a related patch set uploaded (by WMDE-leszek; owner: WMDE-leszek):
[mediawiki/extensions/Wikibase@master] Use *EntitySourceServices as a drop-in replacement for repo-based WikibaseServices

https://gerrit.wikimedia.org/r/485862

Change 487148 merged by jenkins-bot:
[mediawiki/extensions/Wikibase@master] Added test for MultipleEntitySourceServices::getEntityRevisionLookup

https://gerrit.wikimedia.org/r/487148

Change 485671 merged by jenkins-bot:
[mediawiki/extensions/Wikibase@master] Added ByTypeDispatchingEntityInfoBuilder

https://gerrit.wikimedia.org/r/485671

Change 487369 merged by jenkins-bot:
[mediawiki/extensions/Wikibase@master] Added useEntitySourceBasedFederation flag to DataAccessSettings

https://gerrit.wikimedia.org/r/487369

Change 487372 merged by jenkins-bot:
[mediawiki/extensions/Wikibase@master] Added dummy EntitySource class exploding whenever used

https://gerrit.wikimedia.org/r/487372

Change 487373 merged by jenkins-bot:
[mediawiki/extensions/Wikibase@master] Added EntitySource to SqlEntityInfoBuilder

https://gerrit.wikimedia.org/r/487373

Change 485672 merged by jenkins-bot:
[mediawiki/extensions/Wikibase@master] Added EntityInfoBuilder to *EntitySourceServices

https://gerrit.wikimedia.org/r/485672

Change 487967 had a related patch set uploaded (by WMDE-leszek; owner: WMDE-leszek):
[mediawiki/extensions/Wikibase@master] Added WikibaseClient::getItemTermIndex

https://gerrit.wikimedia.org/r/487967

Change 487494 merged by jenkins-bot:
[mediawiki/extensions/Wikibase@master] Added EntitySource to TermSqlIndex

https://gerrit.wikimedia.org/r/487494

Change 488008 had a related patch set uploaded (by WMDE-leszek; owner: WMDE-leszek):
[mediawiki/extensions/WikibaseMediaInfo@master] Adjust WikiPageEntityMetaDataLookup constructor call and re-enabled the test

https://gerrit.wikimedia.org/r/488008

Change 485690 merged by jenkins-bot:
[mediawiki/extensions/Wikibase@master] Added TermSearchInteractoryFactory to *EntitySourceServices

https://gerrit.wikimedia.org/r/485690

Change 485827 merged by jenkins-bot:
[mediawiki/extensions/Wikibase@master] Added ByTypeDispatchingPrefetchingTermLookup

https://gerrit.wikimedia.org/r/485827

Change 485828 merged by jenkins-bot:
[mediawiki/extensions/Wikibase@master] Added PrefetchingTermLookup to *EntitySourceServices

https://gerrit.wikimedia.org/r/485828

Change 487636 merged by jenkins-bot:
[mediawiki/extensions/WikibaseMediaInfo@master] Temporarily skip tests that call WikiPageEntityMetaDataLookup's constructor

https://gerrit.wikimedia.org/r/487636

Change 487607 merged by jenkins-bot:
[mediawiki/extensions/Wikibase@master] Added EntitySource to WikiPageEntityMetaDataLookup

https://gerrit.wikimedia.org/r/487607

Change 485829 merged by jenkins-bot:
[mediawiki/extensions/Wikibase@master] Added ByTypeDispatchingEntityPrefetcher

https://gerrit.wikimedia.org/r/485829

Change 488040 had a related patch set uploaded (by WMDE-leszek; owner: WMDE-leszek):
[mediawiki/extensions/Wikibase@master] Added concept base URI to Entity Source and Definitions

https://gerrit.wikimedia.org/r/488040

Change 488008 merged by jenkins-bot:
[mediawiki/extensions/WikibaseMediaInfo@master] Adjust WikiPageEntityMetaDataLookup constructor call and re-enabled the test

https://gerrit.wikimedia.org/r/488008

Change 488050 had a related patch set uploaded (by WMDE-leszek; owner: WMDE-leszek):
[mediawiki/extensions/Wikibase@master] Use concept base URIs from source definitions when using source config instead of repository prefix one

https://gerrit.wikimedia.org/r/488050

Change 485830 merged by jenkins-bot:
[mediawiki/extensions/Wikibase@master] Added EntityPrefetcher to *EntitySourceServices

https://gerrit.wikimedia.org/r/485830

Change 487645 merged by jenkins-bot:
[mediawiki/extensions/Wikibase@master] Added EntitySource to PropertyInfoTable

https://gerrit.wikimedia.org/r/487645

Change 485855 merged by jenkins-bot:
[mediawiki/extensions/Wikibase@master] Added PropertyInfoLookup to *EntitySourceServices

https://gerrit.wikimedia.org/r/485855

Change 485861 merged by jenkins-bot:
[mediawiki/extensions/Wikibase@master] MultipleEntitySourceServices implements WikibaseServices interface

https://gerrit.wikimedia.org/r/485861

Change 485862 merged by jenkins-bot:
[mediawiki/extensions/Wikibase@master] Use MultipleEntitySourceServices as a drop-in replacement for repo-based WikibaseServices

https://gerrit.wikimedia.org/r/485862

Change 488092 had a related patch set uploaded (by WMDE-leszek; owner: WMDE-leszek):
[mediawiki/extensions/Wikibase@master] Take EntitySources into account in EntityContentFactory::getTitleForId

https://gerrit.wikimedia.org/r/488092

Change 487967 merged by jenkins-bot:
[mediawiki/extensions/Wikibase@master] Added WikibaseClient::getItemTermIndex

https://gerrit.wikimedia.org/r/487967

Change 488040 merged by jenkins-bot:
[mediawiki/extensions/Wikibase@master] Added concept base URI to Entity Source and Definitions

https://gerrit.wikimedia.org/r/488040

Change 488050 merged by jenkins-bot:
[mediawiki/extensions/Wikibase@master] Use concept base URIs from source definitions when using source config instead of repository prefix one

https://gerrit.wikimedia.org/r/488050

Change 488092 merged by jenkins-bot:
[mediawiki/extensions/Wikibase@master] Take EntitySources into account in EntityContentFactory::getTitleForId

https://gerrit.wikimedia.org/r/488092

Change 488500 had a related patch set uploaded (by WMDE-leszek; owner: WMDE-leszek):
[mediawiki/extensions/Wikibase@master] Add interwiki prefix to Entity Source

https://gerrit.wikimedia.org/r/488500

Change 488924 had a related patch set uploaded (by WMDE-leszek; owner: WMDE-leszek):
[mediawiki/extensions/Wikibase@master] Use prefix of the entity source when generating Title objects in EntityContentFactory

https://gerrit.wikimedia.org/r/488924

Change 488966 had a related patch set uploaded (by WMDE-leszek; owner: WMDE-leszek):
[mediawiki/extensions/Wikibase@master] Added EntitySource to WikiPageEntityStore

https://gerrit.wikimedia.org/r/488966

Change 489385 had a related patch set uploaded (by WMDE-leszek; owner: WMDE-leszek):
[mediawiki/extensions/Wikibase@master] wbsearchentities: Show name of entity source name in "repository" part of API response

https://gerrit.wikimedia.org/r/489385

Change 489388 had a related patch set uploaded (by WMDE-leszek; owner: WMDE-leszek):
[mediawiki/extensions/Wikibase@master] Do not use ForeignEntityValidator when using entity-source based access to other Wikibase instance entities

https://gerrit.wikimedia.org/r/489388

Change 489447 had a related patch set uploaded (by WMDE-leszek; owner: WMDE-leszek):
[mediawiki/extensions/Wikibase@master] Made RdfVocabulary work with EntitySources

https://gerrit.wikimedia.org/r/489447

Change 489449 had a related patch set uploaded (by WMDE-leszek; owner: WMDE-leszek):
[mediawiki/extensions/Wikibase@master] Fixed entity-source-based-access branch of WikibaseRepo::getEntityTypeToRepositoryMapping

https://gerrit.wikimedia.org/r/489449

Change 489454 had a related patch set uploaded (by WMDE-leszek; owner: WMDE-leszek):
[mediawiki/extensions/Wikibase@master] Made RepoLinker aware of entity sources

https://gerrit.wikimedia.org/r/489454

Change 488500 merged by jenkins-bot:
[mediawiki/extensions/Wikibase@master] Add interwiki prefix to Entity Source

https://gerrit.wikimedia.org/r/488500

Change 488924 merged by jenkins-bot:
[mediawiki/extensions/Wikibase@master] Use prefix of the entity source when generating Title objects in EntityContentFactory

https://gerrit.wikimedia.org/r/488924

Change 488966 merged by jenkins-bot:
[mediawiki/extensions/Wikibase@master] Added EntitySource to WikiPageEntityStore

https://gerrit.wikimedia.org/r/488966

Change 489385 merged by jenkins-bot:
[mediawiki/extensions/Wikibase@master] wbsearchentities: Show name of entity source name in "repository" part of API response

https://gerrit.wikimedia.org/r/489385

Change 489698 had a related patch set uploaded (by Addshore; owner: Addshore):
[operations/mediawiki-config@master] Wikibase.php, add conditional setting of useEntitySourceBasedFederation

https://gerrit.wikimedia.org/r/489698

Change 489701 had a related patch set uploaded (by Addshore; owner: Addshore):
[operations/mediawiki-config@master] BETA: wmgUseEntitySourceBasedFederation true

https://gerrit.wikimedia.org/r/489701

Change 489698 merged by jenkins-bot:
[operations/mediawiki-config@master] Wikibase.php, add conditional setting of useEntitySourceBasedFederation

https://gerrit.wikimedia.org/r/489698

Change 489388 merged by jenkins-bot:
[mediawiki/extensions/Wikibase@master] Do not use ForeignEntityValidator when using entity-source based access to other Wikibase instance entities

https://gerrit.wikimedia.org/r/489388

Change 489449 merged by jenkins-bot:
[mediawiki/extensions/Wikibase@master] Fixed entity-source-based-access branch of WikibaseRepo::getEntityTypeToRepositoryMapping

https://gerrit.wikimedia.org/r/489449

Change 489701 merged by jenkins-bot:
[operations/mediawiki-config@master] BETA: wmgUseEntitySourceBasedFederation true

https://gerrit.wikimedia.org/r/489701

The EntitySource stuff is now turned on on beta

Change 489771 had a related patch set uploaded (by WMDE-leszek; owner: WMDE-leszek):
[mediawiki/extensions/WikibaseQualityConstraints@master] Temporarily skip tests creating RdfVocabulary instance

https://gerrit.wikimedia.org/r/489771

Change 489454 merged by jenkins-bot:
[mediawiki/extensions/Wikibase@master] Made RepoLinker aware of entity sources

https://gerrit.wikimedia.org/r/489454

Change 490060 had a related patch set uploaded (by WMDE-leszek; owner: WMDE-leszek):
[mediawiki/extensions/Wikibase@master] Added a dedicated config setting for entity sources, and a service processing it

https://gerrit.wikimedia.org/r/490060

Change 490104 had a related patch set uploaded (by WMDE-leszek; owner: WMDE-leszek):
[operations/mediawiki-config@master] Added wmgWikibaseEntitySources setting for defining Wikibase "entity sources"

https://gerrit.wikimedia.org/r/490104

Change 490108 had a related patch set uploaded (by WMDE-leszek; owner: WMDE-leszek):
[operations/mediawiki-config@master] DNM Define Wikibase "entity sources" on beta commons

https://gerrit.wikimedia.org/r/490108

Change 490145 had a related patch set uploaded (by WMDE-leszek; owner: WMDE-leszek):
[mediawiki/extensions/Wikibase@master] Moved out logic on creating entity source definitions from backwards compatible settings out of WikibaseRepo/Client

https://gerrit.wikimedia.org/r/490145

Change 490633 had a related patch set uploaded (by WMDE-leszek; owner: WMDE-leszek):
[operations/mediawiki-config@master] Added wmgWikibaseRepoLocalEntitySourceName to define the "local" source of Wikibase Repo

https://gerrit.wikimedia.org/r/490633

Change 490634 had a related patch set uploaded (by WMDE-leszek; owner: WMDE-leszek):
[operations/mediawiki-config@master] Added wmgUseEntitySourceBasedFederation setting to switch the federation mechanism

https://gerrit.wikimedia.org/r/490634

Change 490658 had a related patch set uploaded (by WMDE-leszek; owner: WMDE-leszek):
[operations/mediawiki-config@master] DNM Use "entity source based federation" on Beta Commons

https://gerrit.wikimedia.org/r/490658

Change 490658 abandoned by WMDE-leszek:
DNM Use "entity source based federation" on Beta Commons

Reason:
not needed, done already

https://gerrit.wikimedia.org/r/490658

Change 490634 abandoned by WMDE-leszek:
Added wmgUseEntitySourceBasedFederation setting to switch the federation mechanism

https://gerrit.wikimedia.org/r/490634

Change 491534 had a related patch set uploaded (by WMDE-leszek; owner: WMDE-leszek):
[mediawiki/extensions/Wikibase@master] Documented entity-source-related config options.

https://gerrit.wikimedia.org/r/491534

Discovered T216686 which might be caused by this endeavour.

Change 492263 had a related patch set uploaded (by WMDE-leszek; owner: WMDE-leszek):
[mediawiki/extensions/Wikibase@master] Fixed WikibaseRepo::getLocalRepoWikiPageMetaDataAccessor when using "entity source based federation"

https://gerrit.wikimedia.org/r/492263

Change 492263 merged by jenkins-bot:
[mediawiki/extensions/Wikibase@master] Fixed WikibaseRepo::getLocalRepoWikiPageMetaDataAccessor when using "entity source based federation"

https://gerrit.wikimedia.org/r/492263