Page MenuHomePhabricator

wbterms: Add a cache for PrefetchingPropertyTermLookup::prefetchTerms()
Closed, ResolvedPublic

Description

Graphs show us that PrefetchingPropertyTermLookup::prefetchTerms() is being called around 70K times every minute. This is clearly too much for something like properties. I suggest adding a layer of APCu cache with TTL of ten seconds to keep the hot data in the servers for long period of time without drowning it.

Event Timeline

Restricted Application added a subscriber: Aklapper. · View Herald TranscriptJan 15 2020, 3:35 PM

As noted in mattermost this might relate to T241050 and before we add any cache we should make sure we need to.
What exactly is causing all of these calls to prefetchTerms ?

Addshore triaged this task as Medium priority.Jan 15 2020, 4:11 PM
WMDE-leszek changed the task status from Open to Stalled.Jan 16 2020, 1:10 PM

To check next: why prefetching happens, and whether it is needed - if not, should stop happening.

WMDE-leszek changed the task status from Stalled to Open.Jan 16 2020, 1:10 PM
Addshore renamed this task from Add a cache for PrefetchingPropertyTermLookup::prefetchTerms() to wbterms: Add a cache for PrefetchingPropertyTermLookup::prefetchTerms().Jan 16 2020, 4:34 PM

It should use the formatter cache.

We dove a bit deeper into all of the things that are happening surrounding the prefetching, term buffers, lookups, fallbacks and caching and decided that to do this properly we should probably aim for something like this:

This doesnt really account for just wanting to retrieve raw terms (rarther than term fallbacks yet), but that could also be worked in, even with its own cache.
This topic area might be a good candidate for a Wikidata-Trailblazing-Exploration

Change 566779 had a related patch set uploaded (by Jakob; owner: Jakob):
[mediawiki/extensions/Wikibase@master] Add TermCache

https://gerrit.wikimedia.org/r/566779

Change 567020 had a related patch set uploaded (by Jakob; owner: Jakob):
[mediawiki/extensions/Wikibase@master] Add LatestRevisionResolvingTermCache

https://gerrit.wikimedia.org/r/567020

Change 567487 had a related patch set uploaded (by Jakob; owner: Jakob):
[mediawiki/extensions/Wikibase@master] Add CachingPrefetchingTermLookup

https://gerrit.wikimedia.org/r/567487

Change 566779 merged by jenkins-bot:
[mediawiki/extensions/Wikibase@master] Add TermCacheKeyBuilder

https://gerrit.wikimedia.org/r/566779

Change 568494 had a related patch set uploaded (by Jakob; owner: Jakob):
[mediawiki/extensions/Wikibase@master] LatestRevisionAndRedirectResolverTest: avoid repeated lookups

https://gerrit.wikimedia.org/r/568494

Change 569210 had a related patch set uploaded (by Jakob; owner: Jakob):
[mediawiki/extensions/Wikibase@master] Wire up caching around PrefetchingPropertyTermLookup

https://gerrit.wikimedia.org/r/569210

Change 567020 merged by jenkins-bot:
[mediawiki/extensions/Wikibase@master] Add RedirectResolvingLatestRevisionLookup

https://gerrit.wikimedia.org/r/567020

Change 567487 merged by jenkins-bot:
[mediawiki/extensions/Wikibase@master] Add CachingPrefetchingTermLookup

https://gerrit.wikimedia.org/r/567487

Change 568494 merged by jenkins-bot:
[mediawiki/extensions/Wikibase@master] LatestRevisionAndRedirectResolverTest: avoid repeated lookups

https://gerrit.wikimedia.org/r/568494

Change 569210 merged by jenkins-bot:
[mediawiki/extensions/Wikibase@master] Wire up caching around PrefetchingPropertyTermLookup

https://gerrit.wikimedia.org/r/569210

WMDE-leszek closed this task as Resolved.Feb 10 2020, 9:39 AM

I think we're good with this one. Should there be some work left, please re-open.

Change 571964 had a related patch set uploaded (by Jakob; owner: Jakob):
[mediawiki/extensions/Wikibase@master] Wrap PrefetchingPropertyTermLookup cache to emit metrics

https://gerrit.wikimedia.org/r/571964

Change 571964 merged by jenkins-bot:
[mediawiki/extensions/Wikibase@master] Wrap PrefetchingPropertyTermLookup cache to emit metrics

https://gerrit.wikimedia.org/r/571964