Page MenuHomePhabricator
Create Task
Maniphest T236681

SqlEntityInfoBuilder reads and writes from the old term store regardless of the config
Closed, ResolvedPublic8 Estimated Story Points
Actions

Assigned To
Ladsgroup
Authored By
Ladsgroup
Oct 28 2019, 12:57 PM
Tags
Referenced Files
F31292883: image.png
Nov 26 2019, 2:25 PM
F31087621: image.png
Nov 15 2019, 10:58 AM
Subscribers
Addshore
Aklapper
• alaa_wmde
• Aleksey_WMDE
darthmon_wmde
gerritbot
Ladsgroup
View All 9 Subscribers
Tokens
"Baby Tequila" token, awarded by Addshore.

Description

This is a blocker for turning off writing to the old term store and should be fixed ASAP.

SqlEntityInfoBuilder::collectTermsForEntities() that empowers things like property_info table and such, directly reads from the old term store and doesn't respect the config. It queries wb_terms table 10K times per minute on average.

Acceptance Criteria:

  • Should have a cache
  • Should switch read / sql logic based on the existing configuration

Details

SubjectRepoBranchLines +/-
Wire EntityInfoBuildersmediawiki/extensions/Wikibasemaster+132 -7
Introduce ByIdDispatchingEntityInfoBuildermediawiki/extensions/Wikibasemaster+154 -0
Introduce DatabaseEntityInfoBuilder based on the new term storemediawiki/extensions/Wikibasemaster+1 K -0
Put a layer of APC cache on top of reading wb_terms in SqlEntityInfoBuildermediawiki/extensions/Wikibasewmf/1.35.0-wmf.5+160 -43
Put a layer of APC cache on top of reading wb_terms in SqlEntityInfoBuildermediawiki/extensions/Wikibasemaster+160 -43
Customize query in gerrit

Related Objects
Search...

StatusSubtypeAssignedTask
 ResolvedAddshoreT208425 [EPIC] Kill the wb_terms table
 ResolvedLadsgroupT229407 Spikes in DB traffic and rows/s reads when reading from new terms store
 ResolvedAddshoreT219303 [Checkpoint 5] Update Read Logic
 Resolved• alaa_wmdeT221710 [Story] Develop Lookup functionality
 InvalidNoneT221711 Create abstract lookup classes/methods in wikibase-term-store
 DeclinedNoneT221712 Implement lookup in doctrine-term-store
 InvalidNoneT221713 Implement lookup logic in mediawiki-term-store
 ResolvedNoneT223994 Add service to turn term IDs back into terms
 ResolvedLucas_Werkmeister_WMDET224504 Implement database-based PropertyTermStore
 InvalidNoneT224543 Update Wikibase terms lookup/search logic to use PropertyTermStore
 InvalidNoneT224544 Update Wikibase terms lookup/search logic to use ItemTermStore
 ResolvedLucas_Werkmeister_WMDET225000 Introduce way to bulk-lookup terms in normalized schema
 Resolved• alaa_wmdeT225002 Implement PrefetchingTermLookup for items using normalized storage
 Resolved• alaa_wmdeT225603 Wire up read logic for normalized storage in Wikibase
 Resolved• alaa_wmdeT225003 Implement PropertyLabelResolver using normalized storage
 Resolved• alaa_wmdeT225626 Remove fallback to master logic from resolver
 Resolved• alaa_wmdeT225717 Implement DatabaseTermIdsResolver::resolveTermsViaJoin
 Resolved• alaa_wmdeT225718 Add TypeIdsLookup and implement it in DatabaseTypeIdsStore
 Resolved• alaa_wmdeT225818 Add type and language filtering capabilities to DatabaseTermIdsResolver::resolveTermsViaJoin
 ResolvedLucas_Werkmeister_WMDET225001 Implement PrefetchingTermLookup for properties using normalized storage
 DuplicateNoneT236979 Fix remaining usecases still reading from old wb_terms schema
 ResolvedLadsgroupT236681 SqlEntityInfoBuilder reads and writes from the old term store regardless of the config

Event Timeline

Ladsgroup created this task.Oct 28 2019, 12:57 PM
Restricted Application added a subscriber: Aklapper. · View Herald TranscriptOct 28 2019, 12:57 PM
Maintenance_bot moved this task from incoming to in progress on the Wikidata board.Oct 28 2019, 1:15 PM
Addshore awarded a token.Oct 29 2019, 10:49 AM
Addshore edited projects, added Wikidata-Campsite; removed Wikidata-Campsite (Wikidata-Campsite-Iteration-∞ (On Hold)).Oct 29 2019, 1:51 PM
Addshore moved this task from Incoming to Ready to estimate on the Wikidata-Campsite board.
Addshore triaged this task as High priority.Oct 29 2019, 1:54 PM
Addshore updated the task description. (Show Details)
Addshore set the point value for this task to 8.Oct 29 2019, 1:58 PM
Addshore moved this task from Ready to estimate to Ready to pick up on the Wikidata-Campsite board.
Ladsgroup added a comment.Oct 30 2019, 11:57 AM

The patch for cache: https://gerrit.wikimedia.org/r/c/mediawiki/extensions/Wikibase/+/546659

gerritbot added a comment.Oct 30 2019, 12:25 PM

Change 546659 had a related patch set uploaded (by Ladsgroup; owner: Ladsgroup):
[mediawiki/extensions/Wikibase@master] Put a layer of object cache on top of reading wb_terms in SqlEntityInfoBuilder

https://gerrit.wikimedia.org/r/546659

gerritbot added a project: Patch-For-Review.Oct 30 2019, 12:25 PM
Addshore merged a task: T205193: Create EntityInfoBuilder implementation that uses cache instead of secondary SQL table.Oct 30 2019, 1:13 PM
Addshore merged a task: T198868: Do not get entity data from wb_terms when "prefetching" data for displaying on pages.
Addshore added subscribers: gerritbot, • Aleksey_WMDE.
Addshore added a parent task: T236979: Fix remaining usecases still reading from old wb_terms schema.Oct 31 2019, 7:38 AM
Addshore mentioned this in T208425: [EPIC] Kill the wb_terms table.Oct 31 2019, 7:44 AM
Ladsgroup mentioned this in T229407: Spikes in DB traffic and rows/s reads when reading from new terms store.Nov 5 2019, 1:31 PM
Ladsgroup claimed this task.Nov 11 2019, 4:23 PM
Ladsgroup edited projects, added Wikidata-Campsite (Wikidata-Campsite-Iteration-∞ (On Hold)); removed Wikidata-Campsite.

I need to rewrite half of the class to make it work with caching.

Restricted Application added a project: User-Ladsgroup. · View Herald TranscriptNov 11 2019, 4:23 PM
Ladsgroup moved this task from To Do (prioritised from top to bottom) to Doing on the Wikidata-Campsite (Wikidata-Campsite-Iteration-∞ (On Hold)) board.Nov 11 2019, 5:09 PM
Maintenance_bot moved this task from Incoming to In progress on the User-Ladsgroup board.Nov 11 2019, 5:15 PM
• alaa_wmde added a parent task: T229407: Spikes in DB traffic and rows/s reads when reading from new terms store.Nov 12 2019, 1:53 PM
Ladsgroup moved this task from Doing to Peer Review on the Wikidata-Campsite (Wikidata-Campsite-Iteration-∞ (On Hold)) board.Nov 12 2019, 7:27 PM
Ladsgroup moved this task from Peer Review to Doing on the Wikidata-Campsite (Wikidata-Campsite-Iteration-∞ (On Hold)) board.Nov 14 2019, 12:43 PM
gerritbot added a comment.Nov 14 2019, 12:48 PM

Change 546659 merged by jenkins-bot:
[mediawiki/extensions/Wikibase@master] Put a layer of APC cache on top of reading wb_terms in SqlEntityInfoBuilder

https://gerrit.wikimedia.org/r/546659

gerritbot added a comment.Nov 14 2019, 12:55 PM

Change 550828 had a related patch set uploaded (by Ladsgroup; owner: Ladsgroup):
[mediawiki/extensions/Wikibase@wmf/1.35.0-wmf.5] Put a layer of APC cache on top of reading wb_terms in SqlEntityInfoBuilder

https://gerrit.wikimedia.org/r/550828

ReleaseTaggerBot added a project: MW-1.35-notes (1.35.0-wmf.8; 2019-11-26).Nov 14 2019, 1:01 PM
gerritbot added a comment.Nov 14 2019, 1:12 PM

Change 550828 merged by jenkins-bot:
[mediawiki/extensions/Wikibase@wmf/1.35.0-wmf.5] Put a layer of APC cache on top of reading wb_terms in SqlEntityInfoBuilder

https://gerrit.wikimedia.org/r/550828

ReleaseTaggerBot edited projects, added MW-1.35-notes (1.35.0-wmf.5; 2019-11-05); removed MW-1.35-notes (1.35.0-wmf.8; 2019-11-26).Nov 14 2019, 2:00 PM
Maintenance_bot removed a project: Patch-For-Review.Nov 14 2019, 2:10 PM
Stashbot mentioned this in T231011: (euwiki) Mysterious, coordinated slowdowns every ~ 25 minutes on API servers.Nov 14 2019, 4:04 PM

Mentioned in SAL (#wikimedia-operations) [2019-11-14T16:04:20Z] <ladsgroup@deploy1001> Synchronized php-1.35.0-wmf.5/extensions/Wikibase: [[gerrit:550828|Put a layer of APC cache on top of reading wb_terms in SqlEntityInfoBuilder]] (T231011 T229407 T236681), Try II (duration: 00m 56s)

Ladsgroup added a comment.Nov 15 2019, 10:58 AM

The cache is there:

image.png (731×1 px, 83 KB)

Ladsgroup added a comment.Nov 26 2019, 2:25 PM

Fixing the cache issue increased the read but not that much:

image.png (683×1 px, 285 KB)

gerritbot added a comment.Dec 2 2019, 1:12 PM

Change 554057 had a related patch set uploaded (by Ladsgroup; owner: Ladsgroup):
[mediawiki/extensions/Wikibase@master] Introduce DatabaseEntityInfoBuilder based on the new term store

https://gerrit.wikimedia.org/r/554057

gerritbot added a project: Patch-For-Review.Dec 2 2019, 1:12 PM
Ladsgroup updated the task description. (Show Details)Dec 2 2019, 9:52 PM
Ladsgroup moved this task from Doing to Peer Review on the Wikidata-Campsite (Wikidata-Campsite-Iteration-∞ (On Hold)) board.
gerritbot added a comment.Dec 3 2019, 2:44 PM

Change 554057 merged by jenkins-bot:
[mediawiki/extensions/Wikibase@master] Introduce DatabaseEntityInfoBuilder based on the new term store

https://gerrit.wikimedia.org/r/554057

Ladsgroup moved this task from Peer Review to Doing on the Wikidata-Campsite (Wikidata-Campsite-Iteration-∞ (On Hold)) board.Dec 3 2019, 2:57 PM
ReleaseTaggerBot edited projects, added MW-1.35-notes (1.35.0-wmf.10; 2019-12-10); removed MW-1.35-notes (1.35.0-wmf.5; 2019-11-05).Dec 3 2019, 3:00 PM
Maintenance_bot removed a project: Patch-For-Review.Dec 3 2019, 3:10 PM
gerritbot added a comment.Dec 3 2019, 4:18 PM

Change 554322 had a related patch set uploaded (by Ladsgroup; owner: Ladsgroup):
[mediawiki/extensions/Wikibase@master] Introduce ByTypeAndIdDispatchingEntityInfoBuilder

https://gerrit.wikimedia.org/r/554322

gerritbot added a project: Patch-For-Review.Dec 3 2019, 4:18 PM
Ladsgroup moved this task from Doing to Peer Review on the Wikidata-Campsite (Wikidata-Campsite-Iteration-∞ (On Hold)) board.Dec 5 2019, 9:18 PM
gerritbot added a comment.Dec 6 2019, 4:05 PM

Change 555523 had a related patch set uploaded (by Ladsgroup; owner: Ladsgroup):
[mediawiki/extensions/Wikibase@master] [WIP] Wire EntityInfoBuilders

https://gerrit.wikimedia.org/r/555523

gerritbot added a comment.Dec 9 2019, 12:48 PM

Change 554322 merged by jenkins-bot:
[mediawiki/extensions/Wikibase@master] Introduce ByIdDispatchingEntityInfoBuilder

https://gerrit.wikimedia.org/r/554322

gerritbot added a comment.Dec 11 2019, 2:46 PM

Change 555523 merged by jenkins-bot:
[mediawiki/extensions/Wikibase@master] Wire EntityInfoBuilders

https://gerrit.wikimedia.org/r/555523

Ladsgroup moved this task from Peer Review to Test (Verification) on the Wikidata-Campsite (Wikidata-Campsite-Iteration-∞ (On Hold)) board.Dec 11 2019, 2:50 PM
ReleaseTaggerBot edited projects, added MW-1.35-notes (1.35.0-wmf.11; 2019-12-17); removed MW-1.35-notes (1.35.0-wmf.10; 2019-12-10).Dec 11 2019, 3:00 PM
Maintenance_bot removed a project: Patch-For-Review.Dec 11 2019, 3:11 PM
Addshore mentioned this in T241050: Investigate if FormatterCache is actually used everywhere it should be when rendering an entity.Dec 18 2019, 12:26 PM
Ladsgroup mentioned this in T241057: Cannot access the database: No working replica DB server: Unknown error (10.64.32.113).Dec 18 2019, 1:17 PM
Addshore closed this task as Resolved.Dec 23 2019, 1:03 PM
Content licensed under Creative Commons Attribution-ShareAlike (CC BY-SA) 4.0 unless otherwise noted; code licensed under GNU General Public License (GPL) 2.0 or later and other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL