Description
Details
Related Objects
- Mentioned In
- T244340: Reduce read pressure on mc* servers by adding a machine-local Memcached instance (on-host memcached)
- Mentioned Here
- T218197: CacheAwarePropertyInfoStore & CachingPropertyInfoLookup should use WANObjectCache instead of BagOStuff
T97368: Fix inefficient CacheAwarePropertyInfoStore memcached access pattern
T209343: Add APC caching to CacheAwarePropertyInfoStore
T244340: Reduce read pressure on mc* servers by adding a machine-local Memcached instance (on-host memcached)
Event Timeline
I think I found out what's wrong. The APCu cache seems to be per-wiki but this doesn't need to be like that. Let me fix it.
I was wrong, we somehow removed the APCu cache bit altogether, I should find out what happened.
APC cache was removed in https://gerrit.wikimedia.org/r/#/c/mediawiki/extensions/Wikibase/+/498348/ T218197
I don't remember if this was intentional or not, commit msg, unfortunately, doesn't help me remember.
Change 602666 had a related patch set uploaded (by Ladsgroup; owner: Ladsgroup):
[mediawiki/extensions/Wikibase@master] Wrap WAN-cached PropertyInfoLookup with an APCu cache
Change 602666 merged by jenkins-bot:
[mediawiki/extensions/Wikibase@master] Wrap WAN-cached PropertyInfoLookup with an APCu cache
Change 603482 had a related patch set uploaded (by Ladsgroup; owner: Ladsgroup):
[mediawiki/extensions/Wikibase@wmf/1.35.0-wmf.35] Wrap WAN-cached PropertyInfoLookup with an APCu cache
Change 603482 merged by jenkins-bot:
[mediawiki/extensions/Wikibase@wmf/1.35.0-wmf.35] Wrap WAN-cached PropertyInfoLookup with an APCu cache
Mentioned in SAL (#wikimedia-operations) [2020-06-08T15:09:31Z] <ladsgroup@deploy1001> Synchronized php-1.35.0-wmf.35/extensions/Wikibase/lib/includes/Store/CachingPropertyInfoLookup.php: Wrap WAN-cached PropertyInfoLookup with an APCu cache, Part I out of III (T254536) (duration: 00m 59s)
Mentioned in SAL (#wikimedia-operations) [2020-06-08T15:10:54Z] <ladsgroup@deploy1001> Synchronized php-1.35.0-wmf.35/extensions/Wikibase/repo/includes/Store/Sql/SqlStore.php: Wrap WAN-cached PropertyInfoLookup with an APCu cache, Part II out of III (T254536) (duration: 00m 57s)
Mentioned in SAL (#wikimedia-operations) [2020-06-08T15:12:35Z] <ladsgroup@deploy1001> Synchronized php-1.35.0-wmf.35/extensions/Wikibase/client/includes/Store/Sql/DirectSqlStore.php: Wrap WAN-cached PropertyInfoLookup with an APCu cache, Part III out of III (T254536) (duration: 00m 57s)
This should be fixed, but I have no idea how to verify it – I couldn’t find any information on how to see if that cache key is still hot.
This changed the cache from having a TTL for 20 seconds to cacheDuration which is currently 24 hours.
This was deliberately set to be low before, although 24 hours seems to be doing fine?
This does mean that there is a higher change of classes fetching data to get old or outdated property info data.
We have some code that updates the cache when the property info store changes (CacheAwarePropertyInfoStore), but that only affects the WAN cache, not the APC cache. Maybe we should lower the TTL of the APC cache only, while keeping the WAN cache TTL higher?
So when it was changed, it was one hour only: https://gerrit.wikimedia.org/r/plugins/gitiles/mediawiki/extensions/Wikibase/+/refs/changes/66/602666/4/client/config/WikibaseClient.default.php#83
IMO, 24 hours is also fine unless users complain things are not getting updated
Looking at usages of that config variables now:
- Old usage from the entity cache that we previously removed (now just uses CACHE_NONE) https://gerrit.wikimedia.org/g/mediawiki/extensions/Wikibase/+/b2705a5cf9bc4daeae15466948f1dfce14305878/client/includes/Store/Sql/DirectSqlStore.php#383 https://gerrit.wikimedia.org/g/mediawiki/extensions/Wikibase/+/b2705a5cf9bc4daeae15466948f1dfce14305878/repo/includes/Store/Sql/SqlStore.php#388
- Usage in the property info caches https://gerrit.wikimedia.org/g/mediawiki/extensions/Wikibase/+/b2705a5cf9bc4daeae15466948f1dfce14305878/client/includes/Store/Sql/DirectSqlStore.php#440 https://gerrit.wikimedia.org/g/mediawiki/extensions/Wikibase/+/b2705a5cf9bc4daeae15466948f1dfce14305878/repo/includes/Store/Sql/SqlStore.php#452 https://gerrit.wikimedia.org/g/mediawiki/extensions/Wikibase/+/b2705a5cf9bc4daeae15466948f1dfce14305878/repo/includes/Store/Sql/SqlStore.php#494
- Usage in the termfallback cache https://gerrit.wikimedia.org/g/mediawiki/extensions/Wikibase/+/b2705a5cf9bc4daeae15466948f1dfce14305878/client/includes/WikibaseClient.php#1271 https://gerrit.wikimedia.org/g/mediawiki/extensions/Wikibase/+/b2705a5cf9bc4daeae15466948f1dfce14305878/repo/includes/WikibaseRepo.php#2321
- PropertyLabelResolver https://gerrit.wikimedia.org/g/mediawiki/extensions/Wikibase/+/b2705a5cf9bc4daeae15466948f1dfce14305878/client/includes/WikibaseClient.php#1330
- CachingEntityRevisionLookup https://gerrit.wikimedia.org/g/mediawiki/extensions/Wikibase/+/b2705a5cf9bc4daeae15466948f1dfce14305878/repo/includes/Store/Sql/SqlStore.php#417
We would probably benefit from:
- Continuing to remove this old entity cache that we pass CACHE_NONE into now
- Evaluating if we need PropertyLabelResolver in its current form, or if it should change somewhat now?
- Split the caches to have their own configurable ttls?
- Evaluate the final CachingEntityRevisionLookup and see if this even needs the cache any more