Page MenuHomePhabricator

entitysources: Directly create entitySources config for WMF sites instead of using compat layer
Closed, ResolvedPublic

Description

Right now we are using entitySources and the entitySource services in production.
However we currently give Wikibase the legacy configuration, and it munges this into the entitySources configuration.

We should instead directly write the entitySources config, this will allow us to remove the back compat in time.

Steps:

Deployment checking code helpers https://phabricator.wikimedia.org/P10913

Related Objects

StatusSubtypeAssignedTask
OpenAddshore
ResolvedAddshore
ResolvedWMDE-leszek
ResolvedAddshore
ResolvedLadsgroup
ResolvedLadsgroup
ResolvedLadsgroup
ResolvedPRODUCTION ERRORAddshore
ResolvedAddshore
ResolvedAddshore
Resolvedโ€ข Pablo-WMDE
ResolvedAddshore
ResolvedAddshore
Resolvedmatthiasmullie
ResolvedPRODUCTION ERRORAddshore
ResolvedPRODUCTION ERRORAddshore
ResolvedPRODUCTION ERRORAddshore

Event Timeline

There are a very large number of changes, so older changes are hidden. Show Older Changes
Addshore renamed this task from Directly create entitySources config for WMF sites instead of using compat layer to entitysources: Directly create entitySources config for WMF sites instead of using compat layer.Jan 16 2020, 4:33 PM

Change 569031 had a related patch set uploaded (by WMDE-leszek; owner: WMDE-leszek):
[operations/mediawiki-config@master] Wikibase: added config variables to configure entity sources

https://gerrit.wikimedia.org/r/569031

Change 569204 had a related patch set uploaded (by WMDE-leszek; owner: WMDE-leszek):
[operations/mediawiki-config@master] Beta wikidata: Define entity sources configuration

https://gerrit.wikimedia.org/r/569204

Change 569205 had a related patch set uploaded (by WMDE-leszek; owner: WMDE-leszek):
[operations/mediawiki-config@master] Beta commons: Define entity sources configuration

https://gerrit.wikimedia.org/r/569205

Change 569206 had a related patch set uploaded (by WMDE-leszek; owner: WMDE-leszek):
[operations/mediawiki-config@master] Beta cluster: use entity source Wikibase setting for all wikibase-enabled wikis

https://gerrit.wikimedia.org/r/569206

Change 569208 had a related patch set uploaded (by WMDE-leszek; owner: WMDE-leszek):
[operations/mediawiki-config@master] Beta cluster: remove custom wmgWikibaseClientRepositories settings

https://gerrit.wikimedia.org/r/569208

Change 569209 had a related patch set uploaded (by WMDE-leszek; owner: WMDE-leszek):
[operations/mediawiki-config@master] Test wikidata: Define entity sources configuration

https://gerrit.wikimedia.org/r/569209

Change 569256 had a related patch set uploaded (by WMDE-leszek; owner: WMDE-leszek):
[operations/mediawiki-config@master] Test wikibase clients: Define entity sources configuration

https://gerrit.wikimedia.org/r/569256

Change 569257 had a related patch set uploaded (by WMDE-leszek; owner: WMDE-leszek):
[operations/mediawiki-config@master] Test commons: Define entity sources configuration

https://gerrit.wikimedia.org/r/569257

Change 569258 had a related patch set uploaded (by WMDE-leszek; owner: WMDE-leszek):
[operations/mediawiki-config@master] Wikidata: Define entity sources configuration

https://gerrit.wikimedia.org/r/569258

Change 569259 had a related patch set uploaded (by WMDE-leszek; owner: WMDE-leszek):
[operations/mediawiki-config@master] Wikidata client wikis: Define entity sources configuration

https://gerrit.wikimedia.org/r/569259

Change 569260 had a related patch set uploaded (by WMDE-leszek; owner: WMDE-leszek):
[operations/mediawiki-config@master] Commons: Define entity sources configuration

https://gerrit.wikimedia.org/r/569260

Change 569261 had a related patch set uploaded (by WMDE-leszek; owner: WMDE-leszek):
[operations/mediawiki-config@master] Wikidata/Wikibase: use entity source Wikibase setting for all wikibase-enabled wikis

https://gerrit.wikimedia.org/r/569261

I've gone through the patches and clicked "rebase" on them. Worked on all of them. Not sure why I didn't get a complaint on the need to rebase manually.

Change 569031 merged by jenkins-bot:
[operations/mediawiki-config@master] Wikibase: added config variables to configure entity sources

https://gerrit.wikimedia.org/r/569031

Mentioned in SAL (#wikimedia-operations) [2020-02-18T12:18:51Z] <ladsgroup@deploy1001> Synchronized wmf-config/InitialiseSettings.php: SWAT: [[gerrit:569031|Wikibase: added config variables to configure entity sources (T242087)]], Part I (duration: 01m 06s)

Mentioned in SAL (#wikimedia-operations) [2020-02-18T12:20:27Z] <ladsgroup@deploy1001> Synchronized wmf-config/InitialiseSettings.php: SWAT: [[gerrit:569031|Wikibase: added config variables to configure entity sources (T242087)]], Part I, take II (the cache issue) (duration: 01m 04s)

Mentioned in SAL (#wikimedia-operations) [2020-02-18T12:22:18Z] <ladsgroup@deploy1001> Synchronized wmf-config/Wikibase.php: SWAT: [[gerrit:569031|Wikibase: added config variables to configure entity sources (T242087)]], Part II (duration: 01m 03s)

Change 569204 merged by jenkins-bot:
[operations/mediawiki-config@master] Beta wikidata: Define entity sources configuration

https://gerrit.wikimedia.org/r/569204

Change 572928 had a related patch set uploaded (by WMDE-leszek; owner: WMDE-leszek):
[operations/mediawiki-config@master] Beta wikidata: Define entity sources configuration

https://gerrit.wikimedia.org/r/572928

Change 572928 merged by jenkins-bot:
[operations/mediawiki-config@master] Beta wikidata: Define entity sources configuration

https://gerrit.wikimedia.org/r/572928

Change 569205 merged by jenkins-bot:
[operations/mediawiki-config@master] Beta commons: Define entity sources configuration

https://gerrit.wikimedia.org/r/569205

Change 569206 merged by jenkins-bot:
[operations/mediawiki-config@master] Beta cluster: use entity source Wikibase setting for all wikibase-enabled wikis

https://gerrit.wikimedia.org/r/569206

Change 569208 merged by jenkins-bot:
[operations/mediawiki-config@master] Beta cluster: remove custom wmgWikibaseClientRepositories settings

https://gerrit.wikimedia.org/r/569208

Test is now all wrapped up, so I can look at moving forward with this this or next week.
SO I'll put my face on it.

Addshore changed the task status from Open to Stalled.May 6 2020, 10:14 AM
Addshore removed Addshore as the assignee of this task.

While T248664: entitysources: Directly create entitySources config for WMF "test" wikis was rolled out T250181: test.wikidata.org - term store not updating was created.
After reviewing the current patches that are up, I think we need to solve the underlying issue in term storage there before moving forward with this...

WMDE-leszek changed the task status from Stalled to Open.Jun 3 2020, 6:35 AM

I believe it is no longer stalled since last week per changes in T252019 got merged, and also the config changes were adjusted to use "false" as a database name for the time being, see e.g. https://gerrit.wikimedia.org/r/c/operations/mediawiki-config/+/569258

Change 605608 had a related patch set uploaded (by WMDE-leszek; owner: WMDE-leszek):
[operations/mediawiki-config@master] Wikibase: added a config to simpler configuration of "entity sources"

https://gerrit.wikimedia.org/r/605608

Change 605609 had a related patch set uploaded (by WMDE-leszek; owner: WMDE-leszek):
[operations/mediawiki-config@master] Wikibase: Generate "entity source" config array based on source names

https://gerrit.wikimedia.org/r/605609

Change 605610 had a related patch set uploaded (by WMDE-leszek; owner: WMDE-leszek):
[operations/mediawiki-config@master] Wikibase: Removed no longer used wmgWikibaseEntitySources setting

https://gerrit.wikimedia.org/r/605610

Moving the last point to a new ticket

Finally moving this to verification, as the final parts of this change are just cleanup and the actual criteria have been met already.

Change 569261 abandoned by Addshore:
[operations/mediawiki-config@master] Wikidata/Wikibase: use entity source Wikibase setting for all wikibase-enabled wikis

Reason:
Abandoning for the reason I left in the comment

https://gerrit.wikimedia.org/r/569261

All subtickets now have all of their patches deployed and all wmf prod sites that need this config now have it!
Will leave it in verification for a day to see if anything pops up