Page MenuHomePhabricator
Create Task
Maniphest T242087

entitysources: Directly create entitySources config for WMF sites instead of using compat layer
Closed, ResolvedPublic
Actions

Description

Right now we are using entitySources and the entitySource services in production.
However we currently give Wikibase the legacy configuration, and it munges this into the entitySources configuration.

We should instead directly write the entitySources config, this will allow us to remove the back compat in time.

Steps:

Deployment checking code helpers https://phabricator.wikimedia.org/P10913

Details

SubjectRepoBranchLines +/-
Wikibase: added a config to simpler configuration of "entity sources"operations/mediawiki-configmaster+23 -0
Wikibase: Generate "entity source" config array based on source namesoperations/mediawiki-configmaster+278 -2
Wikibase: Removed no longer used wmgWikibaseEntitySources settingoperations/mediawiki-configmaster+0 -147
Wikidata/Wikibase: use entity source Wikibase setting for all wikibase-enabled wikisoperations/mediawiki-configmaster+2 -19
Beta cluster: remove custom wmgWikibaseClientRepositories settingsoperations/mediawiki-configmaster+0 -16
Beta cluster: use entity source Wikibase setting for all wikibase-enabled wikisoperations/mediawiki-configmaster+2 -2
Beta commons: Define entity sources configurationoperations/mediawiki-configmaster+25 -0
Beta wikidata: Define entity sources configurationoperations/mediawiki-configmaster+19 -0
Beta wikidata: Define entity sources configurationoperations/mediawiki-configmaster+19 -0
Wikibase: added config variables to configure entity sourcesoperations/mediawiki-configmaster+18 -0
Customize query in gerrit

Related Objects
Search...

StatusSubtypeAssignedTask
 ResolvedAddshoreT241975 entitysources: Remove old MultiRepository & PerRepository Service containers and config
 ResolvedAddshoreT242087 entitysources: Directly create entitySources config for WMF sites instead of using compat layer
 ResolvedWMDE-leszekT241971 useEntitySourceBasedFederation true by default in Wikibase
 ResolvedAddshoreT243395 wmgUseEntitySourceBasedFederation true for Wikimedia clients (all sites)
 ResolvedLadsgroupT241972 wmgUseEntitySourceBasedFederation true for Wikidata.org
 ResolvedLadsgroupT241973 wmgUseEntitySourceBasedFederation true for test.wikidata.org
 ResolvedLadsgroupT241974 wmgUseEntitySourceBasedFederation true for beta wikidata
 ResolvedPRODUCTION ERRORAddshoreT243415 Multiple repo/includes/Rdf errors, when wmgUseEntitySourceBasedFederation true on Wikidata.org
 ResolvedAddshoreT248656 entitysources: Directly create entitySources config for beta cluster sites
 ResolvedAddshoreT248664 entitysources: Directly create entitySources config for WMF "test" wikis
 Resolved• Pablo-WMDET250181 test.wikidata.org - term store not updating
 ResolvedAddshoreT252019 Remove Wikibase assumption that a false dbname is the only way to refer to the local DB.
 ResolvedAddshoreT254315 entitysources: Directly create entitySources config for WMF production wikis
 ResolvedmatthiasmullieT255601 MediaInfo UI breaks when repoDatabase set to "testcommonswiki" instead of false
 ResolvedPRODUCTION ERRORAddshoreT256906 No namespace configured for MediaInfo entities, when deploying entity source config to wikidata clients
 ResolvedPRODUCTION ERRORAddshoreT256907 Call to a member function getDatabaseName() on null, when deploying entity source config to wikidata clients
 ResolvedPRODUCTION ERRORAddshoreT256909 Call to a member function getSourceName() on null, when deploying entity source config to wikidata clients

Event Timeline

There are a very large number of changes, so older changes are hidden. Show Older Changes
Addshore added a project: User-WMDE-leszek.Jan 7 2020, 10:13 AM
Addshore moved this task from Ready to estimate to Ready to pick up on the Wikidata-Campsite board.Jan 7 2020, 10:30 AM
Addshore moved this task from Ready to pick up to Wikidata-Campsite-Iteration-∞ (On Hold) on the Wikidata-Campsite board.Jan 7 2020, 1:30 PM
Addshore edited projects, added Wikidata-Campsite (Wikidata-Campsite-Iteration-∞ (On Hold)); removed Wikidata-Campsite.
Maintenance_bot moved this task from incoming to in progress on the Wikidata board.Jan 7 2020, 2:15 PM
Addshore triaged this task as Low priority.Jan 14 2020, 4:09 PM
Addshore renamed this task from Directly create entitySources config for WMF sites instead of using compat layer to entitysources: Directly create entitySources config for WMF sites instead of using compat layer.Jan 16 2020, 4:33 PM
gerritbot added a comment.Jan 30 2020, 5:34 PM

Change 569031 had a related patch set uploaded (by WMDE-leszek; owner: WMDE-leszek):
[operations/mediawiki-config@master] Wikibase: added config variables to configure entity sources

https://gerrit.wikimedia.org/r/569031

gerritbot added a project: Patch-For-Review.Jan 30 2020, 5:34 PM
WMDE-leszek subscribed.Jan 30 2020, 5:35 PM
Addshore moved this task from To Do (prioritised from top to bottom) to Doing on the Wikidata-Campsite (Wikidata-Campsite-Iteration-∞ (On Hold)) board.Jan 31 2020, 10:24 AM
gerritbot added a comment.Jan 31 2020, 11:54 AM

Change 569204 had a related patch set uploaded (by WMDE-leszek; owner: WMDE-leszek):
[operations/mediawiki-config@master] Beta wikidata: Define entity sources configuration

https://gerrit.wikimedia.org/r/569204

gerritbot added a comment.Jan 31 2020, 11:54 AM

Change 569205 had a related patch set uploaded (by WMDE-leszek; owner: WMDE-leszek):
[operations/mediawiki-config@master] Beta commons: Define entity sources configuration

https://gerrit.wikimedia.org/r/569205

gerritbot added a comment.Jan 31 2020, 11:54 AM

Change 569206 had a related patch set uploaded (by WMDE-leszek; owner: WMDE-leszek):
[operations/mediawiki-config@master] Beta cluster: use entity source Wikibase setting for all wikibase-enabled wikis

https://gerrit.wikimedia.org/r/569206

gerritbot added a comment.Jan 31 2020, 11:54 AM

Change 569208 had a related patch set uploaded (by WMDE-leszek; owner: WMDE-leszek):
[operations/mediawiki-config@master] Beta cluster: remove custom wmgWikibaseClientRepositories settings

https://gerrit.wikimedia.org/r/569208

gerritbot added a comment.Jan 31 2020, 11:54 AM

Change 569209 had a related patch set uploaded (by WMDE-leszek; owner: WMDE-leszek):
[operations/mediawiki-config@master] Test wikidata: Define entity sources configuration

https://gerrit.wikimedia.org/r/569209

gerritbot added a comment.Jan 31 2020, 5:59 PM

Change 569256 had a related patch set uploaded (by WMDE-leszek; owner: WMDE-leszek):
[operations/mediawiki-config@master] Test wikibase clients: Define entity sources configuration

https://gerrit.wikimedia.org/r/569256

gerritbot added a comment.Jan 31 2020, 5:59 PM

Change 569257 had a related patch set uploaded (by WMDE-leszek; owner: WMDE-leszek):
[operations/mediawiki-config@master] Test commons: Define entity sources configuration

https://gerrit.wikimedia.org/r/569257

gerritbot added a comment.Jan 31 2020, 5:59 PM

Change 569258 had a related patch set uploaded (by WMDE-leszek; owner: WMDE-leszek):
[operations/mediawiki-config@master] Wikidata: Define entity sources configuration

https://gerrit.wikimedia.org/r/569258

gerritbot added a comment.Jan 31 2020, 5:59 PM

Change 569259 had a related patch set uploaded (by WMDE-leszek; owner: WMDE-leszek):
[operations/mediawiki-config@master] Wikidata client wikis: Define entity sources configuration

https://gerrit.wikimedia.org/r/569259

gerritbot added a comment.Jan 31 2020, 5:59 PM

Change 569260 had a related patch set uploaded (by WMDE-leszek; owner: WMDE-leszek):
[operations/mediawiki-config@master] Commons: Define entity sources configuration

https://gerrit.wikimedia.org/r/569260

gerritbot added a comment.Jan 31 2020, 5:59 PM

Change 569261 had a related patch set uploaded (by WMDE-leszek; owner: WMDE-leszek):
[operations/mediawiki-config@master] Wikidata/Wikibase: use entity source Wikibase setting for all wikibase-enabled wikis

https://gerrit.wikimedia.org/r/569261

WMDE-leszek moved this task from Doing to Stalled/Waiting on the Wikidata-Campsite (Wikidata-Campsite-Iteration-∞ (On Hold)) board.Feb 1 2020, 3:40 PM
Addshore moved this task from Stalled/Waiting to Peer Review on the Wikidata-Campsite (Wikidata-Campsite-Iteration-∞ (On Hold)) board.Feb 7 2020, 8:32 AM
WMDE-leszek closed subtask T241971: useEntitySourceBasedFederation true by default in Wikibase as Resolved.Feb 10 2020, 9:30 AM
Ladsgroup moved this task from Peer Review to To Do (prioritised from top to bottom) on the Wikidata-Campsite (Wikidata-Campsite-Iteration-∞ (On Hold)) board.Feb 10 2020, 6:21 PM
Ladsgroup subscribed.

The whole chain needs manual rebase apparently.

WMDE-leszek added a comment.Feb 11 2020, 8:46 AM

I've gone through the patches and clicked "rebase" on them. Worked on all of them. Not sure why I didn't get a complaint on the need to rebase manually.

WMDE-leszek moved this task from To Do (prioritised from top to bottom) to Doing on the Wikidata-Campsite (Wikidata-Campsite-Iteration-∞ (On Hold)) board.Feb 11 2020, 8:54 AM
WMDE-leszek moved this task from Doing to Peer Review on the Wikidata-Campsite (Wikidata-Campsite-Iteration-∞ (On Hold)) board.Feb 17 2020, 8:09 AM
gerritbot added a comment.Feb 18 2020, 12:16 PM

Change 569031 merged by jenkins-bot:
[operations/mediawiki-config@master] Wikibase: added config variables to configure entity sources

https://gerrit.wikimedia.org/r/569031

Stashbot added a comment.Feb 18 2020, 12:18 PM

Mentioned in SAL (#wikimedia-operations) [2020-02-18T12:18:51Z] <ladsgroup@deploy1001> Synchronized wmf-config/InitialiseSettings.php: SWAT: [[gerrit:569031|Wikibase: added config variables to configure entity sources (T242087)]], Part I (duration: 01m 06s)

Stashbot added a comment.Feb 18 2020, 12:20 PM

Mentioned in SAL (#wikimedia-operations) [2020-02-18T12:20:27Z] <ladsgroup@deploy1001> Synchronized wmf-config/InitialiseSettings.php: SWAT: [[gerrit:569031|Wikibase: added config variables to configure entity sources (T242087)]], Part I, take II (the cache issue) (duration: 01m 04s)

Stashbot added a comment.Feb 18 2020, 12:22 PM

Mentioned in SAL (#wikimedia-operations) [2020-02-18T12:22:18Z] <ladsgroup@deploy1001> Synchronized wmf-config/Wikibase.php: SWAT: [[gerrit:569031|Wikibase: added config variables to configure entity sources (T242087)]], Part II (duration: 01m 03s)

gerritbot added a comment.Feb 18 2020, 12:31 PM

Change 569204 merged by jenkins-bot:
[operations/mediawiki-config@master] Beta wikidata: Define entity sources configuration

https://gerrit.wikimedia.org/r/569204

gerritbot added a comment.Feb 18 2020, 5:57 PM

Change 572928 had a related patch set uploaded (by WMDE-leszek; owner: WMDE-leszek):
[operations/mediawiki-config@master] Beta wikidata: Define entity sources configuration

https://gerrit.wikimedia.org/r/572928

Tarrow mentioned this in T245540: InvalidArgumentException during Wikibase initialization on Beta Wikidata.Feb 18 2020, 6:05 PM
Jakob_WMDE moved this task from Peer Review to Doing on the Wikidata-Campsite (Wikidata-Campsite-Iteration-∞ (On Hold)) board.Feb 20 2020, 10:18 AM
WMDE-leszek moved this task from Doing to Stalled/Waiting on the Wikidata-Campsite (Wikidata-Campsite-Iteration-∞ (On Hold)) board.Feb 21 2020, 3:00 PM
WMDE-leszek claimed this task.Mar 27 2020, 11:16 AM
WMDE-leszek updated the task description. (Show Details)
WMDE-leszek moved this task from Stalled/Waiting to Doing on the Wikidata-Campsite (Wikidata-Campsite-Iteration-∞ (On Hold)) board.
gerritbot added a comment.Mar 27 2020, 11:53 AM

Change 572928 merged by jenkins-bot:
[operations/mediawiki-config@master] Beta wikidata: Define entity sources configuration

https://gerrit.wikimedia.org/r/572928

gerritbot added a comment.Mar 27 2020, 12:14 PM

Change 569205 merged by jenkins-bot:
[operations/mediawiki-config@master] Beta commons: Define entity sources configuration

https://gerrit.wikimedia.org/r/569205

gerritbot added a comment.Mar 27 2020, 1:09 PM

Change 569206 merged by jenkins-bot:
[operations/mediawiki-config@master] Beta cluster: use entity source Wikibase setting for all wikibase-enabled wikis

https://gerrit.wikimedia.org/r/569206

gerritbot added a comment.Mar 31 2020, 4:32 PM

Change 569208 merged by jenkins-bot:
[operations/mediawiki-config@master] Beta cluster: remove custom wmgWikibaseClientRepositories settings

https://gerrit.wikimedia.org/r/569208

Addshore closed subtask T248656: entitysources: Directly create entitySources config for beta cluster sites as Resolved.Apr 1 2020, 8:44 AM
Addshore updated the task description. (Show Details)Apr 6 2020, 3:00 PM
Addshore closed subtask T248664: entitysources: Directly create entitySources config for WMF "test" wikis as Resolved.May 4 2020, 1:02 PM
Addshore removed WMDE-leszek as the assignee of this task.May 4 2020, 1:14 PM

Test is now all wrapped up, so I can look at moving forward with this this or next week.
SO I'll put my face on it.

Addshore updated the task description. (Show Details)May 4 2020, 1:14 PM
Addshore claimed this task.May 6 2020, 9:50 AM
Restricted Application added a project: User-Addshore. · View Herald TranscriptMay 6 2020, 9:50 AM
Addshore changed the task status from Open to Stalled.May 6 2020, 10:14 AM
Addshore removed Addshore as the assignee of this task.
Addshore moved this task from Doing to Stalled/Waiting on the Wikidata-Campsite (Wikidata-Campsite-Iteration-∞ (On Hold)) board.

While T248664: entitysources: Directly create entitySources config for WMF "test" wikis was rolled out T250181: test.wikidata.org - term store not updating was created.
After reviewing the current patches that are up, I think we need to solve the underlying issue in term storage there before moving forward with this...

WMDE-leszek changed the task status from Stalled to Open.Jun 3 2020, 6:35 AM
WMDE-leszek moved this task from Stalled/Waiting to Doing on the Wikidata-Campsite (Wikidata-Campsite-Iteration-∞ (On Hold)) board.

I believe it is no longer stalled since last week per changes in T252019 got merged, and also the config changes were adjusted to use "false" as a database name for the time being, see e.g. https://gerrit.wikimedia.org/r/c/operations/mediawiki-config/+/569258

WMDE-leszek updated the task description. (Show Details)Jun 3 2020, 6:45 AM
WMDE-leszek moved this task from Doing to Stalled/Waiting on the Wikidata-Campsite (Wikidata-Campsite-Iteration-∞ (On Hold)) board.
Addshore closed subtask T252019: Remove Wikibase assumption that a false dbname is the only way to refer to the local DB. as Resolved.Jun 11 2020, 9:06 AM
gerritbot added a comment.Jun 15 2020, 3:23 PM

Change 605608 had a related patch set uploaded (by WMDE-leszek; owner: WMDE-leszek):
[operations/mediawiki-config@master] Wikibase: added a config to simpler configuration of "entity sources"

https://gerrit.wikimedia.org/r/605608

gerritbot added a comment.Jun 15 2020, 3:23 PM

Change 605609 had a related patch set uploaded (by WMDE-leszek; owner: WMDE-leszek):
[operations/mediawiki-config@master] Wikibase: Generate "entity source" config array based on source names

https://gerrit.wikimedia.org/r/605609

gerritbot added a comment.Jun 15 2020, 3:23 PM

Change 605610 had a related patch set uploaded (by WMDE-leszek; owner: WMDE-leszek):
[operations/mediawiki-config@master] Wikibase: Removed no longer used wmgWikibaseEntitySources setting

https://gerrit.wikimedia.org/r/605610

Addshore claimed this task.Jul 6 2020, 6:57 PM
Addshore closed subtask T254315: entitysources: Directly create entitySources config for WMF production wikis as Resolved.
Addshore updated the task description. (Show Details)
Addshore moved this task from Stalled/Waiting to Doing on the Wikidata-Campsite (Wikidata-Campsite-Iteration-∞ (On Hold)) board.
Addshore moved this task from Unsorted 💣 to Back Burner 🏛️ on the User-Addshore board.
Addshore updated the task description. (Show Details)Jul 6 2020, 9:28 PM

Moving the last point to a new ticket

Addshore mentioned this in T257260: entitysources: Clean up any remainders of the legacy back/compat config in the mediawiki-config repository.Jul 6 2020, 9:33 PM
Addshore moved this task from Doing to Test (Verification) on the Wikidata-Campsite (Wikidata-Campsite-Iteration-∞ (On Hold)) board.

Created T257260: entitysources: Clean up any remainders of the legacy back/compat config in the mediawiki-config repository for the last little bit

Addshore added a comment.Jul 6 2020, 9:34 PM

Finally moving this to verification, as the final parts of this change are just cleanup and the actual criteria have been met already.

gerritbot added a comment.Jul 6 2020, 9:35 PM

Change 569261 abandoned by Addshore:
[operations/mediawiki-config@master] Wikidata/Wikibase: use entity source Wikibase setting for all wikibase-enabled wikis

Reason:
Abandoning for the reason I left in the comment

https://gerrit.wikimedia.org/r/569261

Addshore reopened subtask T254315: entitysources: Directly create entitySources config for WMF production wikis as Open.Jul 7 2020, 8:00 AM
Addshore added a comment.Jul 16 2020, 11:35 AM

All subtickets now have all of their patches deployed and all wmf prod sites that need this config now have it!
Will leave it in verification for a day to see if anything pops up

Addshore closed this task as Resolved.Jul 17 2020, 7:03 AM
Addshore closed subtask T254315: entitysources: Directly create entitySources config for WMF production wikis as Resolved.
Addshore moved this task from Test (Verification) to Done on the Wikidata-Campsite (Wikidata-Campsite-Iteration-∞ (On Hold)) board.
Content licensed under Creative Commons Attribution-ShareAlike (CC BY-SA) 4.0 unless otherwise noted; code licensed under GNU General Public License (GPL) 2.0 or later and other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL