Page MenuHomePhabricator

Allow accessing Wikibase entities from multiple (Wikibase) databases
Closed, ResolvedPublic

Description

Following the discussion in T211800, assuming for now a single entity type can only be provided by single Wikibase instance, make it possible for wikibase to access entities provided by multiple sources.
For now, only direct database database access is considered.

The general idea would be to define "sources" that are providing entities of particular type. Services providing access to entity data will be defined per source (i,e. a particular service is configured to access the database of given source/instance/repository). Wikibase will determine which source-specific service to use to access data of particular entity based on the entity-type vs source configuration.

Entity ID prefixes are not considered for now, as long as there is no technical need (state January 2019) to have a single entity type provided by multiple Wikibase instances/sources./databases.


Update January 2020:

The implementation of "entity-source-based federation" is almost completed (remaining changes have related patches on gerrit) Wikibase Repo "component" (enabled instead of "repository-prefix-based" one using configuration setting), and also close to complete in the Wikibase Client "component".
After finishing the implementation, the remaining step to complete this work would be to remove the "use entity source based federation" setting and disable the possibility to use "repository-prefix-based federation". Consequently all code paths related to "repository-prefix-based federation" would be dropped.

The known remaining steps are listed below (list not complete as of 2020-01-15, being continuously updated) - note the list is broken in the ordered "steps", order of items in the "step" might be arbitrary

...

A follow up to this task removing all of the stuff that this task replaces can be found at T241975: entitysources: Remove old MultiRepository & PerRepository Service containers and config

Details

ProjectBranchLines +/-Subject
mediawiki/extensions/Wikibasemaster+14 -136
mediawiki/extensions/Wikibasemaster+197 -0
mediawiki/extensions/Wikibasemaster+10 -436
mediawiki/extensions/Wikibasemaster+0 -37
mediawiki/extensions/Wikibasemaster+5 -291
mediawiki/extensions/Wikibasemaster+26 -12
mediawiki/extensions/Wikibasemaster+55 -90
mediawiki/extensions/Wikibasemaster+330 -14
mediawiki/extensions/Wikibasemaster+42 -0
mediawiki/extensions/Wikibasemaster+28 -10
mediawiki/extensions/Wikibasemaster+1 -1
mediawiki/extensions/Wikibasemaster+68 -4
mediawiki/extensions/Wikibasewmf/1.35.0-wmf.16+6 -0
mediawiki/extensions/Wikibasemaster+26 -4
mediawiki/extensions/Wikibasemaster+6 -0
mediawiki/extensions/Wikibasemaster+118 -46
mediawiki/extensions/Wikibasemaster+4 -0
mediawiki/extensions/Wikibasemaster+22 -4
mediawiki/extensions/Wikibasemaster+91 -1
mediawiki/extensions/Wikibasemaster+12 -7
mediawiki/extensions/Wikibasemaster+66 -0
mediawiki/extensions/Wikibasemaster+747 -183
mediawiki/extensions/WikibaseQualityConstraintsmaster+3 -26
mediawiki/extensions/WikibaseQualityConstraintsmaster+24 -0
mediawiki/extensions/WikibaseLexememaster+41 -7
mediawiki/extensions/Wikibasemaster+45 -20
mediawiki/extensions/Wikibasemaster+141 -51
operations/mediawiki-configmaster+8 -0
mediawiki/extensions/WikibaseQualityConstraintsmaster+1 -3
mediawiki/extensions/Wikibasemaster+30 -9
mediawiki/extensions/Wikibasemaster+149 -56
mediawiki/extensions/WikibaseQualityConstraintsmaster+2 -0
mediawiki/extensions/Wikibasemaster+98 -0
mediawiki/extensions/WikibaseQualityConstraintsmaster+175 -57
mediawiki/extensions/Wikibasemaster+629 -151
mediawiki/extensions/WikibaseQualityConstraintsmaster+26 -1
mediawiki/extensions/Wikibasemaster+316 -36
mediawiki/extensions/WikibaseMediaInfomaster+9 -5
mediawiki/extensions/WikibaseQualityConstraintsmaster+6 -20
mediawiki/extensions/Wikibasemaster+3 K -4
mediawiki/extensions/Wikibasemaster+526 -59
mediawiki/extensions/WikibaseQualityConstraintsmaster+22 -0
operations/mediawiki-configmaster+5 -0
operations/mediawiki-configmaster+24 -0
mediawiki/extensions/Wikibasemaster+10 -0
mediawiki/extensions/Wikibasemaster+413 -3
mediawiki/extensions/Wikibasemaster+1 -1
operations/mediawiki-configmaster+6 -0
operations/mediawiki-configmaster+5 -0
mediawiki/extensions/Wikibasemaster+123 -7
mediawiki/extensions/Wikibasemaster+680 -14
operations/mediawiki-configmaster+3 -0
mediawiki/extensions/Wikibasemaster+2 -2
mediawiki/extensions/Wikibasemaster+129 -2
operations/mediawiki-configmaster+8 -0
mediawiki/extensions/Wikibasemaster+15 -3
mediawiki/extensions/Wikibasemaster+15 -5
mediawiki/extensions/Wikibasemaster+108 -39
mediawiki/extensions/Wikibasemaster+124 -3
mediawiki/extensions/Wikibasemaster+196 -14
mediawiki/extensions/Wikibasemaster+136 -57
mediawiki/extensions/Wikibasemaster+46 -1
mediawiki/extensions/Wikibasemaster+400 -48
mediawiki/extensions/Wikibasemaster+55 -2
mediawiki/extensions/Wikibasemaster+85 -1
mediawiki/extensions/Wikibasemaster+350 -22
mediawiki/extensions/Wikibasemaster+71 -0
mediawiki/extensions/WikibaseMediaInfomaster+12 -2
mediawiki/extensions/Wikibasemaster+419 -19
mediawiki/extensions/Wikibasemaster+283 -1
mediawiki/extensions/Wikibasemaster+90 -2
mediawiki/extensions/WikibaseMediaInfomaster+2 -0
mediawiki/extensions/Wikibasemaster+103 -0
mediawiki/extensions/Wikibasemaster+384 -1
mediawiki/extensions/Wikibasemaster+876 -24
mediawiki/extensions/Wikibasemaster+86 -3
mediawiki/extensions/Wikibasemaster+218 -6
mediawiki/extensions/Wikibasemaster+99 -0
mediawiki/extensions/Wikibasemaster+36 -13
mediawiki/extensions/Wikibasemaster+151 -0
mediawiki/extensions/Wikibasemaster+29 -0
mediawiki/extensions/Wikibasemaster+88 -0
mediawiki/extensions/Wikibasemaster+165 -1
mediawiki/extensions/Wikibasemaster+268 -0
mediawiki/extensions/Wikibasemaster+216 -0
mediawiki/extensions/Wikibasemaster+364 -0
Show related patches Customize query in gerrit

Related Objects

StatusSubtypeAssignedTask
Declineddchen
OpenNone
OpenNone
DuplicateNone
OpenNone
OpenNone
DuplicateNone
OpenNone
OpenNone
OpenNone
ResolvedRamsey-WMF
ResolvedCparle
ResolvedJdforrester-WMF
ResolvedJdforrester-WMF
OpenNone
OpenNone
ResolvedWMDE-leszek
ResolvedWMDE-leszek
ResolvedNone
ResolvedWMDE-leszek
ResolvedWMDE-leszek
ResolvedWMDE-leszek
ResolvedWMDE-leszek
ResolvedWMDE-leszek
DuplicateNone
ResolvedAddshore
ResolvedAddshore
ResolvedAddshore

Event Timeline

There are a very large number of changes, so older changes are hidden. Show Older Changes

Change 494052 had a related patch set uploaded (by WMDE-leszek; owner: WMDE-leszek):
[mediawiki/extensions/Wikibase@master] WikibaseRepo::newValidatorBuilders: do not rely on RepositoryDefinitions when the "entity source based federation" is in use.

https://gerrit.wikimedia.org/r/494052

Change 494650 had a related patch set uploaded (by WMDE-leszek; owner: WMDE-leszek):
[mediawiki/extensions/Wikibase@master] Constructing GenericServices: use entity source defintions when applicable

https://gerrit.wikimedia.org/r/494650

Change 494651 had a related patch set uploaded (by WMDE-leszek; owner: WMDE-leszek):
[mediawiki/extensions/Wikibase@master] Use (conditionally) EntitySourceDefinitions in WikibaseRepo::getEnabledEntityTypes

https://gerrit.wikimedia.org/r/494651

WMDE-leszek updated the task description. (Show Details)Jan 15 2020, 11:55 AM

@WMDE-leszek which bits of this should the campsite be looking at? / what is ready for review vs doing?

Sorry for the late reply. It took me a bit to review the current state of thing there. I've updated the task description with what I believe is the current state, and tried to outline the remaining steps.
It might make sense to break down certain "steps" from this already long task for better prioritization etc.

WMDE-leszek updated the task description. (Show Details)Jan 16 2020, 12:46 PM

Change 494051 had a related patch set uploaded (by Addshore; owner: WMDE-leszek):
[mediawiki/extensions/Wikibase@master] Use (conditionally) EntitySourceDefinitions in WikibaseRepo::getLocalEntityTypes

https://gerrit.wikimedia.org/r/494051

Change 490145 merged by jenkins-bot:
[mediawiki/extensions/Wikibase@master] Moved out logic on creating entity source definitions from backwards compatible settings out of WikibaseRepo/Client

https://gerrit.wikimedia.org/r/490145

Change 494051 merged by jenkins-bot:
[mediawiki/extensions/Wikibase@master] Use (conditionally) EntitySourceDefinitions in WikibaseRepo::getLocalEntityTypes

https://gerrit.wikimedia.org/r/494051

Addshore updated the task description. (Show Details)Jan 16 2020, 2:24 PM

Change 494052 merged by jenkins-bot:
[mediawiki/extensions/Wikibase@master] WikibaseRepo::newValidatorBuilders: do not rely on RepositoryDefinitions when the "entity source based federation" is in use.

https://gerrit.wikimedia.org/r/494052

Change 494650 merged by jenkins-bot:
[mediawiki/extensions/Wikibase@master] Constructing GenericServices: use entity source defintions when applicable

https://gerrit.wikimedia.org/r/494650

Change 494651 merged by jenkins-bot:
[mediawiki/extensions/Wikibase@master] Use (conditionally) EntitySourceDefinitions in WikibaseRepo::getEnabledEntityTypes

https://gerrit.wikimedia.org/r/494651

Change 564976 merged by jenkins-bot:
[mediawiki/extensions/Wikibase@master] WikibaseRepo: (conditionally) use entity source definitions instead of conceptBaseUri setting

https://gerrit.wikimedia.org/r/564976

Addshore updated the task description. (Show Details)Jan 16 2020, 4:43 PM
Restricted Application added a project: User-Addshore. · View Herald TranscriptJan 17 2020, 12:43 PM
Addshore updated the task description. (Show Details)Jan 17 2020, 4:48 PM

Change 533202 merged by jenkins-bot:
[mediawiki/extensions/Wikibase@master] Prefix the value namespace in the RDF output

https://gerrit.wikimedia.org/r/533202

Addshore updated the task description. (Show Details)Jan 20 2020, 11:46 AM
Addshore updated the task description. (Show Details)Jan 20 2020, 2:11 PM

Change 494055 had a related patch set uploaded (by WMDE-leszek; owner: WMDE-leszek):
[mediawiki/extensions/Wikibase@master] WIP Made WikibaseClient::getRepoItemUriParser use data from entity source definitions, when "entity source based federation" in use

https://gerrit.wikimedia.org/r/494055

Change 566720 had a related patch set uploaded (by Addshore; owner: Addshore):
[mediawiki/extensions/Wikibase@master] EntitySourceDefitions::getEntityTypeToSourceMapping fix for sub entities

https://gerrit.wikimedia.org/r/566720

Change 566721 had a related patch set uploaded (by Addshore; owner: Addshore):
[mediawiki/extensions/Wikibase@wmf/1.35.0-wmf.16] EntitySourceDefitions::getEntityTypeToSourceMapping fix for sub entities

https://gerrit.wikimedia.org/r/566721

Change 566732 had a related patch set uploaded (by Addshore; owner: Addshore):
[mediawiki/extensions/Wikibase@master] Test EntitySourceDefitions::getEntityTypeToSourceMapping with subs

https://gerrit.wikimedia.org/r/566732

Change 566720 merged by jenkins-bot:
[mediawiki/extensions/Wikibase@master] EntitySourceDefitions::getEntityTypeToSourceMapping fix for sub entities

https://gerrit.wikimedia.org/r/566720

Change 566732 merged by jenkins-bot:
[mediawiki/extensions/Wikibase@master] Test EntitySourceDefitions::getEntityTypeToSourceMapping with subs

https://gerrit.wikimedia.org/r/566732

Change 566721 merged by jenkins-bot:
[mediawiki/extensions/Wikibase@wmf/1.35.0-wmf.16] EntitySourceDefitions::getEntityTypeToSourceMapping fix for sub entities

https://gerrit.wikimedia.org/r/566721

Mentioned in SAL (#wikimedia-operations) [2020-01-23T16:02:40Z] <ladsgroup@deploy1001> Synchronized php-1.35.0-wmf.16/extensions/Wikibase/data-access/src/EntitySourceDefinitions.php: [[gerrit:566721|EntitySourceDefitions::getEntityTypeToSourceMapping fix for sub entities (T242415 T214557)]] (duration: 01m 08s)

Change 494055 merged by jenkins-bot:
[mediawiki/extensions/Wikibase@master] Made WikibaseClient::getRepoItemUriParser use data from entity source definitions, when "entity source based federation" in use

https://gerrit.wikimedia.org/r/494055

WMDE-leszek updated the task description. (Show Details)Jan 27 2020, 1:59 PM

Change 569255 had a related patch set uploaded (by WMDE-leszek; owner: WMDE-leszek):
[mediawiki/extensions/Wikibase@master] Removed useEntitySourceBasedFederation config option

https://gerrit.wikimedia.org/r/569255

Change 569325 had a related patch set uploaded (by WMDE-leszek; owner: WMDE-leszek):
[mediawiki/extensions/Wikibase@master] Adjusted RepoLinker to use concept uri data from entity sources

https://gerrit.wikimedia.org/r/569325

Change 569326 had a related patch set uploaded (by WMDE-leszek; owner: WMDE-leszek):
[mediawiki/extensions/Wikibase@master] Adjusted SearchEntities API class to use concept uri data from entity sources

https://gerrit.wikimedia.org/r/569326

Change 569327 had a related patch set uploaded (by WMDE-leszek; owner: WMDE-leszek):
[mediawiki/extensions/Wikibase@master] Added RdfDumpGenerator test cases for entity source based access mode

https://gerrit.wikimedia.org/r/569327

Change 569328 had a related patch set uploaded (by WMDE-leszek; owner: WMDE-leszek):
[mediawiki/extensions/Wikibase@master] Reorganized slightly ValidatorBuildersTest

https://gerrit.wikimedia.org/r/569328

Change 569329 had a related patch set uploaded (by WMDE-leszek; owner: WMDE-leszek):
[mediawiki/extensions/Wikibase@master] rebuildTermSqlIndix: Adjusted constructor to use right "entity source"

https://gerrit.wikimedia.org/r/569329

Change 569330 had a related patch set uploaded (by WMDE-leszek; owner: WMDE-leszek):
[mediawiki/extensions/Wikibase@master] Removed now-redundant tests cases in RepoHooksTest

https://gerrit.wikimedia.org/r/569330

Change 569331 had a related patch set uploaded (by WMDE-leszek; owner: WMDE-leszek):
[mediawiki/extensions/Wikibase@master] Removed now-redundant tests cases in DataAccessSnakFormatterOutputFormatTest

https://gerrit.wikimedia.org/r/569331

Change 569332 had a related patch set uploaded (by WMDE-leszek; owner: WMDE-leszek):
[mediawiki/extensions/Wikibase@master] Make it impossible to write to non local database in PropertyInfoTable

https://gerrit.wikimedia.org/r/569332

Change 569329 merged by jenkins-bot:
[mediawiki/extensions/Wikibase@master] rebuildTermSqlIndex: Adjusted constructor to use right "entity source"

https://gerrit.wikimedia.org/r/569329

Change 569325 merged by jenkins-bot:
[mediawiki/extensions/Wikibase@master] Adjusted RepoLinker to use concept uri data from entity sources

https://gerrit.wikimedia.org/r/569325

Change 569332 merged by jenkins-bot:
[mediawiki/extensions/Wikibase@master] Make it impossible to write to non local database in PropertyInfoTable

https://gerrit.wikimedia.org/r/569332

Change 570879 had a related patch set uploaded (by WMDE-leszek; owner: WMDE-leszek):
[mediawiki/extensions/Wikibase@master] Do not prefix IDs in HtmlPageLinkRendererBeginHookHandler when not needed

https://gerrit.wikimedia.org/r/570879

Entity source based federation (implemented in this ticket) is now deployed and enabled everywhere.
Now just to remove the old stuff.

Except for https://phabricator.wikimedia.org/T241971 making it the wikibase default

Change 570879 merged by jenkins-bot:
[mediawiki/extensions/Wikibase@master] Do not prefix IDs in HtmlPageLinkRendererBeginHookHandler when not needed

https://gerrit.wikimedia.org/r/570879

Change 569328 merged by jenkins-bot:
[mediawiki/extensions/Wikibase@master] Reorganized slightly ValidatorBuildersTest

https://gerrit.wikimedia.org/r/569328

Change 569326 merged by jenkins-bot:
[mediawiki/extensions/Wikibase@master] Adjusted SearchEntities API class to use concept uri data from entity sources

https://gerrit.wikimedia.org/r/569326

WMDE-leszek closed this task as Resolved.Feb 18 2020, 6:16 PM
WMDE-leszek claimed this task.
WMDE-leszek updated the task description. (Show Details)

Change 569255 merged by jenkins-bot:
[mediawiki/extensions/Wikibase@master] Removed useEntitySourceBasedFederation config option

https://gerrit.wikimedia.org/r/569255

Change 569331 merged by jenkins-bot:
[mediawiki/extensions/Wikibase@master] Removed now-redundant tests cases in DataAccessSnakFormatterOutputFormatTest

https://gerrit.wikimedia.org/r/569331

Change 573607 had a related patch set uploaded (by Addshore; owner: Addshore):
[mediawiki/extensions/Wikibase@master] Removed now-redundant tests cases in TermStoreWriter tests

https://gerrit.wikimedia.org/r/573607

Change 573607 merged by jenkins-bot:
[mediawiki/extensions/Wikibase@master] Removed now-redundant tests cases in TermStoreWriter tests

https://gerrit.wikimedia.org/r/573607

Change 569327 merged by jenkins-bot:
[mediawiki/extensions/Wikibase@master] Added RdfDumpGenerator test cases for entity source based access mode

https://gerrit.wikimedia.org/r/569327

Change 569330 merged by jenkins-bot:
[mediawiki/extensions/Wikibase@master] Removed now-redundant tests cases in RepoHooksTest

https://gerrit.wikimedia.org/r/569330