This really really shouldn't happen. I investigate.

I think I found out what's wrong. The APCu cache seems to be per-wiki but this doesn't need to be like that. Let me fix it.

I was wrong, we somehow removed the APCu cache bit altogether, I should find out what happened.

APC cache was removed in https://gerrit.wikimedia.org/r/#/c/mediawiki/extensions/Wikibase/+/498348/ T218197

I don't remember if this was intentional or not, commit msg, unfortunately, doesn't help me remember.

Change 602666 had a related patch set uploaded (by Ladsgroup; owner: Ladsgroup):
[mediawiki/extensions/Wikibase@master] Wrap WAN-cached PropertyInfoLookup with an APCu cache

https://gerrit.wikimedia.org/r/602666

This is basically a ticking bomb

Change 602666 merged by jenkins-bot:
[mediawiki/extensions/Wikibase@master] Wrap WAN-cached PropertyInfoLookup with an APCu cache

https://gerrit.wikimedia.org/r/602666

Change 603482 had a related patch set uploaded (by Ladsgroup; owner: Ladsgroup):
[mediawiki/extensions/Wikibase@wmf/1.35.0-wmf.35] Wrap WAN-cached PropertyInfoLookup with an APCu cache

https://gerrit.wikimedia.org/r/603482

Change 603482 merged by jenkins-bot:
[mediawiki/extensions/Wikibase@wmf/1.35.0-wmf.35] Wrap WAN-cached PropertyInfoLookup with an APCu cache

https://gerrit.wikimedia.org/r/603482

Mentioned in SAL (#wikimedia-operations) [2020-06-08T15:09:31Z] <ladsgroup@deploy1001> Synchronized php-1.35.0-wmf.35/extensions/Wikibase/lib/includes/Store/CachingPropertyInfoLookup.php: Wrap WAN-cached PropertyInfoLookup with an APCu cache, Part I out of III (T254536) (duration: 00m 59s)

Mentioned in SAL (#wikimedia-operations) [2020-06-08T15:10:54Z] <ladsgroup@deploy1001> Synchronized php-1.35.0-wmf.35/extensions/Wikibase/repo/includes/Store/Sql/SqlStore.php: Wrap WAN-cached PropertyInfoLookup with an APCu cache, Part II out of III (T254536) (duration: 00m 57s)

Mentioned in SAL (#wikimedia-operations) [2020-06-08T15:12:35Z] <ladsgroup@deploy1001> Synchronized php-1.35.0-wmf.35/extensions/Wikibase/client/includes/Store/Sql/DirectSqlStore.php: Wrap WAN-cached PropertyInfoLookup with an APCu cache, Part III out of III (T254536) (duration: 00m 57s)

This should be fixed, but I have no idea how to verify it – I couldn’t find any information on how to see if that cache key is still hot.

In T254536#6202431, @Lucas_Werkmeister_WMDE wrote:

This should be fixed, but I have no idea how to verify it – I couldn’t find any information on how to see if that cache key is still hot.

Comparing these two:

• https://performance.wikimedia.org/xhgui/run/view?id=5ede53c03f3dfa3c194a2b6e
• https://performance.wikimedia.org/xhgui/run/view?id=5eda2a183f3dfa026dc3b68c

Also these graphs:

• https://grafana.wikimedia.org/d/lqE4lcGWz/wanobjectcache-key-group?orgId=1&var-kClass=CacheAwarePropertyInfoStore&from=now-6h&to=now
• https://grafana.wikimedia.org/d/2Zx07tGZz/wanobjectcache?orgId=1&from=now-6h&to=now
• https://grafana.wikimedia.org/d/000000316/memcache?orgId=1&from=now-6h&to=now

Thanks, this looks quite conclusive:

This changed the cache from having a TTL for 20 seconds to cacheDuration which is currently 24 hours.
This was deliberately set to be low before, although 24 hours seems to be doing fine?

This does mean that there is a higher change of classes fetching data to get old or outdated property info data.

We have some code that updates the cache when the property info store changes (CacheAwarePropertyInfoStore), but that only affects the WAN cache, not the APC cache. Maybe we should lower the TTL of the APC cache only, while keeping the WAN cache TTL higher?

So when it was changed, it was one hour only: https://gerrit.wikimedia.org/r/plugins/gitiles/mediawiki/extensions/Wikibase/+/refs/changes/66/602666/4/client/config/WikibaseClient.default.php#83

IMO, 24 hours is also fine unless users complain things are not getting updated

Looking at usages of that config variables now:

• Old usage from the entity cache that we previously removed (now just uses CACHE_NONE) https://gerrit.wikimedia.org/g/mediawiki/extensions/Wikibase/+/b2705a5cf9bc4daeae15466948f1dfce14305878/client/includes/Store/Sql/DirectSqlStore.php#383 https://gerrit.wikimedia.org/g/mediawiki/extensions/Wikibase/+/b2705a5cf9bc4daeae15466948f1dfce14305878/repo/includes/Store/Sql/SqlStore.php#388
• Usage in the property info caches https://gerrit.wikimedia.org/g/mediawiki/extensions/Wikibase/+/b2705a5cf9bc4daeae15466948f1dfce14305878/client/includes/Store/Sql/DirectSqlStore.php#440 https://gerrit.wikimedia.org/g/mediawiki/extensions/Wikibase/+/b2705a5cf9bc4daeae15466948f1dfce14305878/repo/includes/Store/Sql/SqlStore.php#452 https://gerrit.wikimedia.org/g/mediawiki/extensions/Wikibase/+/b2705a5cf9bc4daeae15466948f1dfce14305878/repo/includes/Store/Sql/SqlStore.php#494
• Usage in the termfallback cache https://gerrit.wikimedia.org/g/mediawiki/extensions/Wikibase/+/b2705a5cf9bc4daeae15466948f1dfce14305878/client/includes/WikibaseClient.php#1271 https://gerrit.wikimedia.org/g/mediawiki/extensions/Wikibase/+/b2705a5cf9bc4daeae15466948f1dfce14305878/repo/includes/WikibaseRepo.php#2321
• PropertyLabelResolver https://gerrit.wikimedia.org/g/mediawiki/extensions/Wikibase/+/b2705a5cf9bc4daeae15466948f1dfce14305878/client/includes/WikibaseClient.php#1330
• CachingEntityRevisionLookup https://gerrit.wikimedia.org/g/mediawiki/extensions/Wikibase/+/b2705a5cf9bc4daeae15466948f1dfce14305878/repo/includes/Store/Sql/SqlStore.php#417

We would probably benefit from:

• Continuing to remove this old entity cache that we pass CACHE_NONE into now
• Evaluating if we need PropertyLabelResolver in its current form, or if it should change somewhat now?
• Split the caches to have their own configurable ttls?
• Evaluate the final CachingEntityRevisionLookup and see if this even needs the cache any more