Page MenuHomePhabricator

Allow accessing Wikibase entities from multiple (Wikibase) databases
Closed, ResolvedPublic

Description

Following the discussion in T211800, assuming for now a single entity type can only be provided by single Wikibase instance, make it possible for wikibase to access entities provided by multiple sources.
For now, only direct database database access is considered.

The general idea would be to define "sources" that are providing entities of particular type. Services providing access to entity data will be defined per source (i,e. a particular service is configured to access the database of given source/instance/repository). Wikibase will determine which source-specific service to use to access data of particular entity based on the entity-type vs source configuration.

Entity ID prefixes are not considered for now, as long as there is no technical need (state January 2019) to have a single entity type provided by multiple Wikibase instances/sources./databases.


Update January 2020:

The implementation of "entity-source-based federation" is almost completed (remaining changes have related patches on gerrit) Wikibase Repo "component" (enabled instead of "repository-prefix-based" one using configuration setting), and also close to complete in the Wikibase Client "component".
After finishing the implementation, the remaining step to complete this work would be to remove the "use entity source based federation" setting and disable the possibility to use "repository-prefix-based federation". Consequently all code paths related to "repository-prefix-based federation" would be dropped.

The known remaining steps are listed below (list not complete as of 2020-01-15, being continuously updated) - note the list is broken in the ordered "steps", order of items in the "step" might be arbitrary

...

A follow up to this task removing all of the stuff that this task replaces can be found at T241975: entitysources: Remove old MultiRepository & PerRepository Service containers and config

Details

Related Gerrit Patches:
mediawiki/extensions/Wikibase : masterRemoved now-redundant tests cases in RepoHooksTest
mediawiki/extensions/Wikibase : masterAdded RdfDumpGenerator test cases for entity source based access mode
mediawiki/extensions/Wikibase : masterRemoved now-redundant tests cases in TermStoreWriter tests
mediawiki/extensions/Wikibase : masterRemoved now-redundant tests cases in DataAccessSnakFormatterOutputFormatTest
mediawiki/extensions/Wikibase : masterRemoved useEntitySourceBasedFederation config option
mediawiki/extensions/Wikibase : masterAdjusted SearchEntities API class to use concept uri data from entity sources
mediawiki/extensions/Wikibase : masterReorganized slightly ValidatorBuildersTest
mediawiki/extensions/Wikibase : masterDo not prefix IDs in HtmlPageLinkRendererBeginHookHandler when not needed
mediawiki/extensions/Wikibase : masterMake it impossible to write to non local database in PropertyInfoTable
mediawiki/extensions/Wikibase : masterAdjusted RepoLinker to use concept uri data from entity sources
mediawiki/extensions/Wikibase : masterrebuildTermSqlIndex: Adjusted constructor to use right "entity source"
mediawiki/extensions/Wikibase : masterMade WikibaseClient::getRepoItemUriParser use data from entity source definitions, when "entity source based federation" in use
mediawiki/extensions/Wikibase : wmf/1.35.0-wmf.16EntitySourceDefitions::getEntityTypeToSourceMapping fix for sub entities
mediawiki/extensions/Wikibase : masterTest EntitySourceDefitions::getEntityTypeToSourceMapping with subs
mediawiki/extensions/Wikibase : masterEntitySourceDefitions::getEntityTypeToSourceMapping fix for sub entities
mediawiki/extensions/Wikibase : masterPrefix the value namespace in the RDF output
mediawiki/extensions/Wikibase : masterWikibaseRepo: (conditionally) use entity source definitions instead of conceptBaseUri setting
mediawiki/extensions/Wikibase : masterConstructing GenericServices: use entity source defintions when applicable
mediawiki/extensions/Wikibase : masterUse (conditionally) EntitySourceDefinitions in WikibaseRepo::getEnabledEntityTypes
mediawiki/extensions/Wikibase : masterWikibaseRepo::newValidatorBuilders: do not rely on RepositoryDefinitions when the "entity source based federation" is in use.
mediawiki/extensions/Wikibase : masterUse (conditionally) EntitySourceDefinitions in WikibaseRepo::getLocalEntityTypes
mediawiki/extensions/Wikibase : masterMoved out logic on creating entity source definitions from backwards compatible settings out of WikibaseRepo/Client
mediawiki/extensions/WikibaseQualityConstraints : masterAdjust constructor calls in SparqlHelperTest
mediawiki/extensions/WikibaseQualityConstraints : masterTemporarily skip some tests due to on-going changes to Wikibase
mediawiki/extensions/WikibaseLexeme : masterAdjust integration test clean up to have test pass also when EntitySources based federation is in use
mediawiki/extensions/Wikibase : masterdocs, update federation docs
mediawiki/extensions/Wikibase : masterdocs, update services.md for current state
operations/mediawiki-config : masterAdded wmgWikibaseEntitySources setting for defining Wikibase "entity sources"
mediawiki/extensions/WikibaseQualityConstraints : masterRestored the skipped test after changes to Wikibase
mediawiki/extensions/Wikibase : masterPrefix statement, and reference namespaces in RDF output
mediawiki/extensions/Wikibase : masterAdded prefixes to document (data) URIs in the RDF output
mediawiki/extensions/WikibaseQualityConstraints : masterTemporarily skip a test
mediawiki/extensions/Wikibase : masterAdded a service generating canonical URLs of "document" pages for defined Entity Sources
mediawiki/extensions/WikibaseQualityConstraints : masterAdjusted calls to RdfVocabulary methods
mediawiki/extensions/Wikibase : masterPrefix RDF (turtle) namespaces with configurable prefixes instead of using a source/repository suffix
mediawiki/extensions/WikibaseQualityConstraints : masterTemporarily skip tests instantiating RdfVocabulary object from Wikibase
mediawiki/extensions/Wikibase : masterAdd RDF namespace prefixes to the entity source config
mediawiki/extensions/WikibaseMediaInfo : masterUpdated MediaInfoRdfBuilder after changes to Wikibase's RdfVocabulary
mediawiki/extensions/WikibaseQualityConstraints : masterRestore tests for new RdfVocabulary ctor
mediawiki/extensions/Wikibase : masterAdd tests for RDF mapping for the properties defined in "foreign" entity sources
mediawiki/extensions/Wikibase : masterMade RdfVocabulary work with EntitySources
mediawiki/extensions/WikibaseQualityConstraints : masterTemporarily skip tests creating RdfVocabulary instance
operations/mediawiki-config : masterAdded wmgWikibaseRepoLocalEntitySourceName to define the "local" source of Wikibase Repo
operations/mediawiki-config : masterDNM Define Wikibase "entity sources" on beta commons
mediawiki/extensions/Wikibase : masterDocumented entity-source-related config options.
mediawiki/extensions/Wikibase : masterAdded a dedicated config setting for entity sources, and a service processing it
mediawiki/extensions/Wikibase : masterFixed WikibaseRepo::getLocalRepoWikiPageMetaDataAccessor when using "entity source based federation"
operations/mediawiki-config : masterAdded wmgUseEntitySourceBasedFederation setting to switch the federation mechanism
operations/mediawiki-config : masterDNM Use "entity source based federation" on Beta Commons
mediawiki/extensions/Wikibase : masterMade RepoLinker aware of entity sources
mediawiki/extensions/Wikibase : masterAdded EntitySource to WikiPageEntityStore
operations/mediawiki-config : masterBETA: wmgUseEntitySourceBasedFederation true
mediawiki/extensions/Wikibase : masterFixed entity-source-based-access branch of WikibaseRepo::getEntityTypeToRepositoryMapping
mediawiki/extensions/Wikibase : masterDo not use ForeignEntityValidator when using entity-source based access to other Wikibase instance entities
operations/mediawiki-config : masterWikibase.php, add conditional setting of useEntitySourceBasedFederation
mediawiki/extensions/Wikibase : masterwbsearchentities: Show name of entity source name in "repository" part of API response
mediawiki/extensions/Wikibase : masterUse prefix of the entity source when generating Title objects in EntityContentFactory
mediawiki/extensions/Wikibase : masterAdd interwiki prefix to Entity Source
mediawiki/extensions/Wikibase : masterTake EntitySources into account in EntityContentFactory::getTitleForId
mediawiki/extensions/Wikibase : masterUse concept base URIs from source definitions when using source config instead of repository prefix one
mediawiki/extensions/Wikibase : masterAdded concept base URI to Entity Source and Definitions
mediawiki/extensions/Wikibase : masterAdded WikibaseClient::getItemTermIndex
mediawiki/extensions/Wikibase : masterUse MultipleEntitySourceServices as a drop-in replacement for repo-based WikibaseServices
mediawiki/extensions/Wikibase : masterMultipleEntitySourceServices implements WikibaseServices interface
mediawiki/extensions/Wikibase : masterAdded PropertyInfoLookup to *EntitySourceServices
mediawiki/extensions/Wikibase : masterAdded EntitySource to PropertyInfoTable
mediawiki/extensions/Wikibase : masterAdded EntityPrefetcher to *EntitySourceServices
mediawiki/extensions/WikibaseMediaInfo : masterAdjust WikiPageEntityMetaDataLookup constructor call and re-enabled the test
mediawiki/extensions/Wikibase : masterAdded EntitySource to WikiPageEntityMetaDataLookup
mediawiki/extensions/Wikibase : masterAdded ByTypeDispatchingEntityPrefetcher
mediawiki/extensions/Wikibase : masterAdded PrefetchingTermLookup to *EntitySourceServices
mediawiki/extensions/WikibaseMediaInfo : masterTemporarily skip tests that call WikiPageEntityMetaDataLookup's constructor
mediawiki/extensions/Wikibase : masterAdded TermSearchInteractoryFactory to *EntitySourceServices
mediawiki/extensions/Wikibase : masterAdded ByTypeDispatchingPrefetchingTermLookup
mediawiki/extensions/Wikibase : masterAdded EntitySource to TermSqlIndex
mediawiki/extensions/Wikibase : masterAdded EntityInfoBuilder to *EntitySourceServices
mediawiki/extensions/Wikibase : masterAdded EntitySource to SqlEntityInfoBuilder
mediawiki/extensions/Wikibase : masterAdded dummy EntitySource class exploding whenever used
mediawiki/extensions/Wikibase : masterAdded useEntitySourceBasedFederation flag to DataAccessSettings
mediawiki/extensions/Wikibase : masterAdded ByTypeDispatchingEntityInfoBuilder
mediawiki/extensions/Wikibase : masterAdded test for MultipleEntitySourceServices::getEntityRevisionLookup
mediawiki/extensions/Wikibase : masterAdded tests for MultipleEntitySourceServices' EntityStoreWatcher behaviour
mediawiki/extensions/Wikibase : masterFollowup docs & tests for *EntitySourceServices
mediawiki/extensions/Wikibase : masterAdded SingleEntitySourceServices and MultipleEntitySourceServices
mediawiki/extensions/Wikibase : masterAdded ByTypeDispatchingEntityRevisionLookup
mediawiki/extensions/Wikibase : masterAdded EntitySource and EntitySourceDefinitions

Related Objects

StatusSubtypeAssignedTask
Declineddchen
OpenNone
OpenNone
DuplicateNone
OpenNone
ResolvedAbit
OpenNone
DuplicateNone
OpenNone
OpenNone
OpenNone
ResolvedRamsey-WMF
ResolvedCparle
ResolvedJdforrester-WMF
ResolvedJdforrester-WMF
OpenNone
OpenNone
ResolvedWMDE-leszek
ResolvedWMDE-leszek
ResolvedNone
ResolvedWMDE-leszek
ResolvedWMDE-leszek
ResolvedWMDE-leszek
ResolvedWMDE-leszek
ResolvedWMDE-leszek
DuplicateNone
ResolvedAddshore
ResolvedAddshore
ResolvedAddshore

Event Timeline

There are a very large number of changes, so older changes are hidden. Show Older Changes

Change 563961 merged by jenkins-bot:
[mediawiki/extensions/WikibaseQualityConstraints@master] Adjust constructor calls in SparqlHelperTest

https://gerrit.wikimedia.org/r/563961

Change 494052 had a related patch set uploaded (by WMDE-leszek; owner: WMDE-leszek):
[mediawiki/extensions/Wikibase@master] WikibaseRepo::newValidatorBuilders: do not rely on RepositoryDefinitions when the "entity source based federation" is in use.

https://gerrit.wikimedia.org/r/494052

Change 494650 had a related patch set uploaded (by WMDE-leszek; owner: WMDE-leszek):
[mediawiki/extensions/Wikibase@master] Constructing GenericServices: use entity source defintions when applicable

https://gerrit.wikimedia.org/r/494650

Change 494651 had a related patch set uploaded (by WMDE-leszek; owner: WMDE-leszek):
[mediawiki/extensions/Wikibase@master] Use (conditionally) EntitySourceDefinitions in WikibaseRepo::getEnabledEntityTypes

https://gerrit.wikimedia.org/r/494651

WMDE-leszek updated the task description. (Show Details)Jan 15 2020, 11:55 AM

@WMDE-leszek which bits of this should the campsite be looking at? / what is ready for review vs doing?

Sorry for the late reply. It took me a bit to review the current state of thing there. I've updated the task description with what I believe is the current state, and tried to outline the remaining steps.
It might make sense to break down certain "steps" from this already long task for better prioritization etc.

WMDE-leszek updated the task description. (Show Details)Jan 16 2020, 12:46 PM

Change 494051 had a related patch set uploaded (by Addshore; owner: WMDE-leszek):
[mediawiki/extensions/Wikibase@master] Use (conditionally) EntitySourceDefinitions in WikibaseRepo::getLocalEntityTypes

https://gerrit.wikimedia.org/r/494051

Change 490145 merged by jenkins-bot:
[mediawiki/extensions/Wikibase@master] Moved out logic on creating entity source definitions from backwards compatible settings out of WikibaseRepo/Client

https://gerrit.wikimedia.org/r/490145

Change 494051 merged by jenkins-bot:
[mediawiki/extensions/Wikibase@master] Use (conditionally) EntitySourceDefinitions in WikibaseRepo::getLocalEntityTypes

https://gerrit.wikimedia.org/r/494051

Addshore updated the task description. (Show Details)Jan 16 2020, 2:24 PM

Change 494052 merged by jenkins-bot:
[mediawiki/extensions/Wikibase@master] WikibaseRepo::newValidatorBuilders: do not rely on RepositoryDefinitions when the "entity source based federation" is in use.

https://gerrit.wikimedia.org/r/494052

Change 494650 merged by jenkins-bot:
[mediawiki/extensions/Wikibase@master] Constructing GenericServices: use entity source defintions when applicable

https://gerrit.wikimedia.org/r/494650

Change 494651 merged by jenkins-bot:
[mediawiki/extensions/Wikibase@master] Use (conditionally) EntitySourceDefinitions in WikibaseRepo::getEnabledEntityTypes

https://gerrit.wikimedia.org/r/494651

Change 564976 merged by jenkins-bot:
[mediawiki/extensions/Wikibase@master] WikibaseRepo: (conditionally) use entity source definitions instead of conceptBaseUri setting

https://gerrit.wikimedia.org/r/564976

Addshore updated the task description. (Show Details)Jan 16 2020, 4:43 PM
Restricted Application added a project: User-Addshore. · View Herald TranscriptJan 17 2020, 12:43 PM
Addshore updated the task description. (Show Details)Jan 17 2020, 4:48 PM

Change 533202 merged by jenkins-bot:
[mediawiki/extensions/Wikibase@master] Prefix the value namespace in the RDF output

https://gerrit.wikimedia.org/r/533202

Addshore removed Addshore as the assignee of this task.Jan 20 2020, 10:23 AM
Addshore updated the task description. (Show Details)Jan 20 2020, 11:46 AM
Addshore updated the task description. (Show Details)Jan 20 2020, 2:11 PM

Change 494055 had a related patch set uploaded (by WMDE-leszek; owner: WMDE-leszek):
[mediawiki/extensions/Wikibase@master] WIP Made WikibaseClient::getRepoItemUriParser use data from entity source definitions, when "entity source based federation" in use

https://gerrit.wikimedia.org/r/494055

Change 566720 had a related patch set uploaded (by Addshore; owner: Addshore):
[mediawiki/extensions/Wikibase@master] EntitySourceDefitions::getEntityTypeToSourceMapping fix for sub entities

https://gerrit.wikimedia.org/r/566720

Change 566721 had a related patch set uploaded (by Addshore; owner: Addshore):
[mediawiki/extensions/Wikibase@wmf/1.35.0-wmf.16] EntitySourceDefitions::getEntityTypeToSourceMapping fix for sub entities

https://gerrit.wikimedia.org/r/566721

Change 566732 had a related patch set uploaded (by Addshore; owner: Addshore):
[mediawiki/extensions/Wikibase@master] Test EntitySourceDefitions::getEntityTypeToSourceMapping with subs

https://gerrit.wikimedia.org/r/566732

Change 566720 merged by jenkins-bot:
[mediawiki/extensions/Wikibase@master] EntitySourceDefitions::getEntityTypeToSourceMapping fix for sub entities

https://gerrit.wikimedia.org/r/566720

Change 566732 merged by jenkins-bot:
[mediawiki/extensions/Wikibase@master] Test EntitySourceDefitions::getEntityTypeToSourceMapping with subs

https://gerrit.wikimedia.org/r/566732

Change 566721 merged by jenkins-bot:
[mediawiki/extensions/Wikibase@wmf/1.35.0-wmf.16] EntitySourceDefitions::getEntityTypeToSourceMapping fix for sub entities

https://gerrit.wikimedia.org/r/566721

Mentioned in SAL (#wikimedia-operations) [2020-01-23T16:02:40Z] <ladsgroup@deploy1001> Synchronized php-1.35.0-wmf.16/extensions/Wikibase/data-access/src/EntitySourceDefinitions.php: [[gerrit:566721|EntitySourceDefitions::getEntityTypeToSourceMapping fix for sub entities (T242415 T214557)]] (duration: 01m 08s)

Change 494055 merged by jenkins-bot:
[mediawiki/extensions/Wikibase@master] Made WikibaseClient::getRepoItemUriParser use data from entity source definitions, when "entity source based federation" in use

https://gerrit.wikimedia.org/r/494055

WMDE-leszek updated the task description. (Show Details)Mon, Jan 27, 1:59 PM

Change 569255 had a related patch set uploaded (by WMDE-leszek; owner: WMDE-leszek):
[mediawiki/extensions/Wikibase@master] Removed useEntitySourceBasedFederation config option

https://gerrit.wikimedia.org/r/569255

Change 569325 had a related patch set uploaded (by WMDE-leszek; owner: WMDE-leszek):
[mediawiki/extensions/Wikibase@master] Adjusted RepoLinker to use concept uri data from entity sources

https://gerrit.wikimedia.org/r/569325

Change 569326 had a related patch set uploaded (by WMDE-leszek; owner: WMDE-leszek):
[mediawiki/extensions/Wikibase@master] Adjusted SearchEntities API class to use concept uri data from entity sources

https://gerrit.wikimedia.org/r/569326

Change 569327 had a related patch set uploaded (by WMDE-leszek; owner: WMDE-leszek):
[mediawiki/extensions/Wikibase@master] Added RdfDumpGenerator test cases for entity source based access mode

https://gerrit.wikimedia.org/r/569327

Change 569328 had a related patch set uploaded (by WMDE-leszek; owner: WMDE-leszek):
[mediawiki/extensions/Wikibase@master] Reorganized slightly ValidatorBuildersTest

https://gerrit.wikimedia.org/r/569328

Change 569329 had a related patch set uploaded (by WMDE-leszek; owner: WMDE-leszek):
[mediawiki/extensions/Wikibase@master] rebuildTermSqlIndix: Adjusted constructor to use right "entity source"

https://gerrit.wikimedia.org/r/569329

Change 569330 had a related patch set uploaded (by WMDE-leszek; owner: WMDE-leszek):
[mediawiki/extensions/Wikibase@master] Removed now-redundant tests cases in RepoHooksTest

https://gerrit.wikimedia.org/r/569330

Change 569331 had a related patch set uploaded (by WMDE-leszek; owner: WMDE-leszek):
[mediawiki/extensions/Wikibase@master] Removed now-redundant tests cases in DataAccessSnakFormatterOutputFormatTest

https://gerrit.wikimedia.org/r/569331

Change 569332 had a related patch set uploaded (by WMDE-leszek; owner: WMDE-leszek):
[mediawiki/extensions/Wikibase@master] Make it impossible to write to non local database in PropertyInfoTable

https://gerrit.wikimedia.org/r/569332

Change 569329 merged by jenkins-bot:
[mediawiki/extensions/Wikibase@master] rebuildTermSqlIndex: Adjusted constructor to use right "entity source"

https://gerrit.wikimedia.org/r/569329

Change 569325 merged by jenkins-bot:
[mediawiki/extensions/Wikibase@master] Adjusted RepoLinker to use concept uri data from entity sources

https://gerrit.wikimedia.org/r/569325

Change 569332 merged by jenkins-bot:
[mediawiki/extensions/Wikibase@master] Make it impossible to write to non local database in PropertyInfoTable

https://gerrit.wikimedia.org/r/569332

Change 570879 had a related patch set uploaded (by WMDE-leszek; owner: WMDE-leszek):
[mediawiki/extensions/Wikibase@master] Do not prefix IDs in HtmlPageLinkRendererBeginHookHandler when not needed

https://gerrit.wikimedia.org/r/570879

Entity source based federation (implemented in this ticket) is now deployed and enabled everywhere.
Now just to remove the old stuff.

Except for https://phabricator.wikimedia.org/T241971 making it the wikibase default

Change 570879 merged by jenkins-bot:
[mediawiki/extensions/Wikibase@master] Do not prefix IDs in HtmlPageLinkRendererBeginHookHandler when not needed

https://gerrit.wikimedia.org/r/570879

Change 569328 merged by jenkins-bot:
[mediawiki/extensions/Wikibase@master] Reorganized slightly ValidatorBuildersTest

https://gerrit.wikimedia.org/r/569328

Change 569326 merged by jenkins-bot:
[mediawiki/extensions/Wikibase@master] Adjusted SearchEntities API class to use concept uri data from entity sources

https://gerrit.wikimedia.org/r/569326

WMDE-leszek closed this task as Resolved.Tue, Feb 18, 6:16 PM
WMDE-leszek claimed this task.
WMDE-leszek updated the task description. (Show Details)

Change 569255 merged by jenkins-bot:
[mediawiki/extensions/Wikibase@master] Removed useEntitySourceBasedFederation config option

https://gerrit.wikimedia.org/r/569255

Change 569331 merged by jenkins-bot:
[mediawiki/extensions/Wikibase@master] Removed now-redundant tests cases in DataAccessSnakFormatterOutputFormatTest

https://gerrit.wikimedia.org/r/569331

Change 573607 had a related patch set uploaded (by Addshore; owner: Addshore):
[mediawiki/extensions/Wikibase@master] Removed now-redundant tests cases in TermStoreWriter tests

https://gerrit.wikimedia.org/r/573607

Change 573607 merged by jenkins-bot:
[mediawiki/extensions/Wikibase@master] Removed now-redundant tests cases in TermStoreWriter tests

https://gerrit.wikimedia.org/r/573607

Change 569327 merged by jenkins-bot:
[mediawiki/extensions/Wikibase@master] Added RdfDumpGenerator test cases for entity source based access mode

https://gerrit.wikimedia.org/r/569327

Change 569330 merged by jenkins-bot:
[mediawiki/extensions/Wikibase@master] Removed now-redundant tests cases in RepoHooksTest

https://gerrit.wikimedia.org/r/569330