Page MenuHomePhabricator

entitysources: Directly create entitySources config for WMF production wikis
Closed, ResolvedPublic

Description

"entitySource" configuration setting is the default way of defining what wikis provides which entity types.
Legacy "repositories" setting is going to be removed after all its uses are adjusted to use a dedicated setting.

Related Objects

Event Timeline

There are a very large number of changes, so older changes are hidden. Show Older Changes
Maintenance_bot moved this task from incoming to in progress on the Wikidata board.Jun 3 2020, 7:15 AM
Addshore claimed this task.Jun 15 2020, 2:26 PM
Restricted Application added a project: User-Addshore. ยท View Herald TranscriptJun 15 2020, 2:26 PM

Mentioned in SAL (#wikimedia-operations) [2020-06-15T18:28:55Z] <addshore@deploy1001> Synchronized wmf-config/InitialiseSettings.php: [[gerrit:605584]] T254315 test wikidata: Use the database name in the Wikibase entity source config (duration: 00m 58s)

Addshore updated the task description. (Show Details)Jul 1 2020, 11:38 AM
Addshore updated the task description. (Show Details)

Change 569258 merged by jenkins-bot:
[operations/mediawiki-config@master] Wikidata: Define entity sources configuration

https://gerrit.wikimedia.org/r/c/operations/mediawiki-config/ /569258

Addshore updated the task description. (Show Details)Jul 1 2020, 11:46 AM

Mentioned in SAL (#wikimedia-operations) [2020-07-01T11:46:23Z] <addshore@deploy1001> Synchronized wmf-config/InitialiseSettings.php: T254315 Wikidata: Define entity sources configuration [[gerrit:569258]] (duration: 01m 06s)

Addshore updated the task description. (Show Details)Jul 1 2020, 1:16 PM

Change 569259 merged by jenkins-bot:
[operations/mediawiki-config@master] Wikidata client wikis: Define entity sources configuration

https://gerrit.wikimedia.org/r/c/operations/mediawiki-config/ /569259

Mentioned in SAL (#wikimedia-operations) [2020-07-01T18:41:39Z] <addshore@deploy1001> sync-file aborted: T254315 Wikidata client wikis: Define entity sources configuration [[gerrit:569259]] (duration: 00m 38s)

Mentioned in SAL (#wikimedia-operations) [2020-07-01T18:43:40Z] <addshore@deploy1001> Synchronized wmf-config: REVERT T254315 Wikidata client wikis: Define entity sources configuration [[gerrit:569259]] (duration: 01m 04s)

Change 608839 had a related patch set uploaded (by Addshore; owner: Addshore):
[operations/mediawiki-config@master] Revert "Revert "Wikidata client wikis: Define entity sources configuration""

https://gerrit.wikimedia.org/r/c/operations/mediawiki-config/ /608839

It looks like the order of changes both on gerrit and in this description were slightly wrong and the commonswiki entity sources stuff needs to happen before the wikidata client patch.
This also likely means that the wikidata client patch will need even more poking, but we can figure that out once the first one gets in.

This difference in order will likely mean that the 3 tickets filed above can be closed, as they will not happen, but I'll leave that for when the config change is actually done.

Addshore updated the task description. (Show Details)Jul 1 2020, 7:26 PM

Change 569260 merged by jenkins-bot:
[operations/mediawiki-config@master] Commons: Define entity sources configuration

https://gerrit.wikimedia.org/r/569260

Mentioned in SAL (#wikimedia-operations) [2020-07-06T18:38:40Z] <addshore@deploy1001> Synchronized wmf-config/InitialiseSettings.php: T256906 T256907 T256909 T254315 [[gerrit:569260]] Commons: Define entity sources configuration (duration: 00m 56s)

Change 608839 merged by jenkins-bot:
[operations/mediawiki-config@master] Wikidata client wikis: Define entity sources configuration (take 2)

https://gerrit.wikimedia.org/r/608839

Mentioned in SAL (#wikimedia-operations) [2020-07-06T18:45:14Z] <addshore@deploy1001> Synchronized wmf-config: T254315 Wikidata client wikis: Define entity sources configuration (take 2) [[gerrit:608839]] (duration: 00m 58s)

Mentioned in SAL (#wikimedia-operations) [2020-07-06T18:47:03Z] <addshore@deploy1001> Synchronized dblists/wikidataclient.dblist: T254315 Wikidata client wikis: Define entity sources configuration (take 2) [[gerrit:608839]] (duration: 00m 56s)

Change 609643 had a related patch set uploaded (by Lucas Werkmeister (WMDE); owner: Lucas Werkmeister (WMDE)):
[operations/mediawiki-config@master] Revert "Wikidata client wikis: Define entity sources configuration (take 2)"

https://gerrit.wikimedia.org/r/609643

Change 609643 merged by jenkins-bot:
[operations/mediawiki-config@master] Revert "Wikidata client wikis: Define entity sources configuration (take 2)"

https://gerrit.wikimedia.org/r/609643

Mentioned in SAL (#wikimedia-operations) [2020-07-07T07:23:13Z] <lucaswerkmeister-wmde@deploy1001> Synchronized dblists/wikidataclient.dblist: Config: [[gerrit:609643|Revert "Wikidata client wikis: Define entity sources configuration (take 2)" (T254315, T257266)]] (duration: 00m 56s)

Mentioned in SAL (#wikimedia-operations) [2020-07-07T07:24:23Z] <lucaswerkmeister-wmde@deploy1001> Synchronized wmf-config/: Config: [[gerrit:609643|Revert "Wikidata client wikis: Define entity sources configuration (take 2)" (T254315, T257266)]] (duration: 00m 56s)

Change 609644 had a related patch set uploaded (by Lucas Werkmeister (WMDE); owner: Lucas Werkmeister (WMDE)):
[operations/mediawiki-config@master] Revert "Commons: Define entity sources configuration"

https://gerrit.wikimedia.org/r/609644

Change 609644 merged by jenkins-bot:
[operations/mediawiki-config@master] Revert "Commons: Define entity sources configuration"

https://gerrit.wikimedia.org/r/609644

Mentioned in SAL (#wikimedia-operations) [2020-07-07T07:27:45Z] <lucaswerkmeister-wmde@deploy1001> Synchronized wmf-config/InitialiseSettings.php: Config: [[gerrit:609644|Revert "Commons: Define entity sources configuration" (T256906, T256907, T256909, T254315, T257266)]] (duration: 00m 53s)

Mentioned in SAL (#wikimedia-operations) [2020-07-07T07:31:05Z] <lucaswerkmeister-wmde@deploy1001> Synchronized wmf-config/InitialiseSettings.php: Config: [[gerrit:609644|Revert "Commons: Define entity sources configuration" (T256906, T256907, T256909, T254315, T257266)]] (forgot to git rebase so the last sync was a no-op) (duration: 00m 56s)

Change 609987 had a related patch set uploaded (by Addshore; owner: Addshore):
[operations/mediawiki-config@master] Commons: Define entity sources configuration (take 2)

https://gerrit.wikimedia.org/r/609987

Change 609988 had a related patch set uploaded (by Addshore; owner: Addshore):
[operations/mediawiki-config@master] Wikidata client wikis: Define entity sources configuration (take 3)

https://gerrit.wikimedia.org/r/609988

Addshore reopened this task as Open.Jul 7 2020, 8:00 AM

Everything was reverted so re opening

Addshore updated the task description. (Show Details)Jul 7 2020, 8:00 AM

While investigating one of the issues highlighted in T257266 I found the following:

addshore@deploy1001:~$ mwscript shell.php --wiki test2wiki
Psy Shell v0.10.4 (PHP 7.2.31-1+0~20200514.41+debian9~1.gbpe2a56b+wmf1 โ€” cli) by Justin Hileman
>>>
>>> Wikibase\Client\WikibaseClient::getDefaultInstance()->getStore()->getSiteLinkLookup()->getItemIdForLink( 'test2wiki', 'February' );
=> Wikibase\DataModel\Entity\ItemId {#4975}
>>> Wikibase\Client\WikibaseClient::getDefaultInstance()->getStore()->getSiteLinkLookup()->getItemIdForLink( 'testcommonswiki', 'FooLinked' );
=> Wikibase\DataModel\Entity\ItemId {#4956}
>>>
>>>
addshore@deploy1001:~$ mwscript shell.php --wiki testcommonswiki
Psy Shell v0.10.4 (PHP 7.2.31-1+0~20200514.41+debian9~1.gbpe2a56b+wmf1 โ€” cli) by Justin Hileman
>>>
>>> nt\WikiWikibase\Client\WikibaseClient::getDefaultInstance()->getStore()->getSiteLinkLookup()->getItemIdForLink( 'test2wiki', 'February' );
=> null
>>> Wikibase\Client\WikibaseClient::getDefaultInstance()->getStore()->getSiteLinkLookup()->getItemIdForLink( 'testcommonswiki', 'FooLinked' );
=> null

This suggests that testcommons client code can not access the testwikidata repo dbs for things such as the site link lookup...
This would explain sitelinks (and other things) breaking when we rolled forward with this change on real commons

Addshore added a comment.EditedJul 14 2020, 3:30 PM

And it looks like getDatabaseDomainNameOfLocalRepo is used in to make the sqlstore.
For testcommons & commons his will be the local wikibase.
This should NOT be used to access the site store... as the local wikibase does not hold the data about items (which is where the sitelinks are).
This is likely the cause of issues found in T257266

Related to / caused in T243186: entitysources: Client: adjust WikibaseClient::getStore

Test commons with the new entity source based config (pointing to itself)

addshore@deploy1001:~$ mwscript shell.php --wiki testcommonswiki
Psy Shell v0.10.4 (PHP 7.2.31-1+0~20200514.41+debian9~1.gbpe2a56b+wmf1 โ€” cli) by Justin Hileman
>>> Wikibase\Client\WikibaseClient::getDefaultInstance()->getDatabaseDomainNameOfLocalRepo()
=> "testcommonswiki"

Real commons with the old entity source based config... (pointing at wikidata /other repo)

addshore@deploy1001:~$ mwscript shell.php --wiki commonswiki
Psy Shell v0.10.4 (PHP 7.2.31-1+0~20200514.41+debian9~1.gbpe2a56b+wmf1 โ€” cli) by Justin Hileman
>>> Wikibase\Client\WikibaseClient::getDefaultInstance()->getDatabaseDomainNameOfLocalRepo()
=> "wikidatawiki"
>>>

Change 612663 had a related patch set uploaded (by Addshore; owner: Addshore):
[mediawiki/extensions/Wikibase@master] Improve docs for 'localEntitySourceName'

https://gerrit.wikimedia.org/r/612663

Addshore updated the task description. (Show Details)Jul 14 2020, 9:55 PM

Change 612666 had a related patch set uploaded (by Addshore; owner: Addshore):
[operations/mediawiki-config@master] Wikibase: Split localEntitySourceName config for repo and client

https://gerrit.wikimedia.org/r/612666

Change 612667 had a related patch set uploaded (by Addshore; owner: Addshore):
[operations/mediawiki-config@master] Wikibase labs: All client "local" entity sources are wikidata

https://gerrit.wikimedia.org/r/612667

Change 612668 had a related patch set uploaded (by Addshore; owner: Addshore):
[operations/mediawiki-config@master] Wikibase test: Client local entity sources are always testwikidata

https://gerrit.wikimedia.org/r/612668

Change 612669 had a related patch set uploaded (by Addshore; owner: Addshore):
[operations/mediawiki-config@master] Wikidata test: Split client db lists.

https://gerrit.wikimedia.org/r/612669

Change 612670 had a related patch set uploaded (by Addshore; owner: Addshore):
[operations/mediawiki-config@master] Wikibase: remove wmgWikibaseLocalEntitySourceName

https://gerrit.wikimedia.org/r/612670

Change 612666 merged by jenkins-bot:
[operations/mediawiki-config@master] Wikibase: Split localEntitySourceName config for repo and client

https://gerrit.wikimedia.org/r/612666

Change 612667 merged by jenkins-bot:
[operations/mediawiki-config@master] Wikibase labs: All client "local" entity sources are wikidata

https://gerrit.wikimedia.org/r/612667

Mentioned in SAL (#wikimedia-operations) [2020-07-15T11:08:53Z] <addshore@deploy1001> Synchronized wmf-config/Wikibase.php: [[gerrit:612666]] Wikibase: Split localEntitySourceName config for repo and client T254315 (duration: 01m 16s)

Mentioned in SAL (#wikimedia-operations) [2020-07-15T11:11:57Z] <addshore@deploy1001> Synchronized wmf-config/InitialiseSettings-labs.php: LABS [[gerrit:612667]] Wikibase labs: All client "local" entity sources are wikidata T254315 (duration: 01m 04s)

Change 612669 merged by jenkins-bot:
[operations/mediawiki-config@master] Wikidata test: Split client db lists.

https://gerrit.wikimedia.org/r/612669

Mentioned in SAL (#wikimedia-operations) [2020-07-15T11:26:20Z] <addshore@deploy1001> Synchronized dblists/wikidataclient.dblist: T254315 [[gerrit:612669]] Wikidata test: Split client db lists. PT1/2 (duration: 01m 05s)

Mentioned in SAL (#wikimedia-operations) [2020-07-15T11:27:49Z] <addshore@deploy1001> Synchronized wmf-config: T254315 [[gerrit:612669]] Wikidata test: Split client db lists. PT2/2 (duration: 01m 06s)

Change 612668 merged by jenkins-bot:
[operations/mediawiki-config@master] Wikibase test: Client local entity sources are always testwikidata

https://gerrit.wikimedia.org/r/612668

Mentioned in SAL (#wikimedia-operations) [2020-07-15T11:36:08Z] <addshore@deploy1001> Synchronized wmf-config/InitialiseSettings.php: [[gerrit:612668]] Wikibase test: Client local entity sources are always testwikidata T254315 (duration: 01m 05s)

Change 609987 merged by jenkins-bot:
[operations/mediawiki-config@master] Commons: Define entity sources configuration (take 2)

https://gerrit.wikimedia.org/r/609987

Mentioned in SAL (#wikimedia-operations) [2020-07-15T11:57:46Z] <addshore@deploy1001> Synchronized wmf-config/InitialiseSettings.php: [[gerrit:609987]] Commons: Define entity sources configuration (take 2) T254315 (duration: 01m 03s)

Addshore updated the task description. (Show Details)Jul 15 2020, 12:04 PM

Managed to deploy the bulk of the new config changes, including setting entity sources on commons repo and client.
Scheduled the 2 remaining patches for the backport windows tomorrow EU time (then this ticket will be done).
I believe this now also unblocked the sdc commons rdf dumps?

Change 612663 merged by jenkins-bot:
[mediawiki/extensions/Wikibase@master] Improve docs for 'localEntitySourceName'

https://gerrit.wikimedia.org/r/612663

Change 612725 had a related patch set uploaded (by Addshore; owner: Addshore):
[mediawiki/extensions/Wikibase@REL1_35] Improve docs for 'localEntitySourceName'

https://gerrit.wikimedia.org/r/612725

Change 612725 merged by jenkins-bot:
[mediawiki/extensions/Wikibase@REL1_35] Improve docs for 'localEntitySourceName'

https://gerrit.wikimedia.org/r/612725

Change 609988 merged by jenkins-bot:
[operations/mediawiki-config@master] Wikidata client wikis: Define entity sources configuration (take 3)

https://gerrit.wikimedia.org/r/609988

Mentioned in SAL (#wikimedia-operations) [2020-07-16T11:18:11Z] <addshore@deploy1001> Synchronized wmf-config/InitialiseSettings.php: T254315 T257266 [[gerrit:609988]] Wikidata client wikis: Define entity sources configuration (take 3) (duration: 01m 08s)

Change 612670 merged by jenkins-bot:
[operations/mediawiki-config@master] Wikibase: remove wmgWikibaseLocalEntitySourceName

https://gerrit.wikimedia.org/r/612670

Mentioned in SAL (#wikimedia-operations) [2020-07-16T11:23:38Z] <addshore@deploy1001> Synchronized wmf-config/Wikibase.php: T254315 [[gerrit:612670]] Wikibase: remove wmgWikibaseLocalEntitySourceName (duration: 01m 05s)

Addshore updated the task description. (Show Details)

This config is now deployed on ALL sites that need it :)
I'll leave it in the verification column for now to confirm no bugs appear!