Page MenuHomePhabricator

CacheAwarePropertyInfoStore performs 4000 Memc ops/s (APC not working?)
Closed, ResolvedPublic

Description

Background:

At T244340, @Joe wrote:

Some keys are super hot - take for instance WANCache:v:global:CacheAwarePropertyInfoStore:wikidatawiki:P244 which gets read about 4k times per second.

This doesn't make sense. This particular usage pattern from Wikibase has caused outages in the past and was given an APCu layer on top. […] Given we have less than 4000 servers, it sounds like this has stopped working?

Event Timeline

Krinkle created this task.Jun 4 2020, 11:38 PM
Restricted Application added a subscriber: Aklapper. · View Herald TranscriptJun 4 2020, 11:38 PM

This really really shouldn't happen. I investigate.

I think I found out what's wrong. The APCu cache seems to be per-wiki but this doesn't need to be like that. Let me fix it.

I was wrong, we somehow removed the APCu cache bit altogether, I should find out what happened.

Addshore added a comment.EditedJun 5 2020, 11:44 AM

APC cache was removed in https://gerrit.wikimedia.org/r/#/c/mediawiki/extensions/Wikibase/+/498348/ T218197

I don't remember if this was intentional or not, commit msg, unfortunately, doesn't help me remember.

Change 602666 had a related patch set uploaded (by Ladsgroup; owner: Ladsgroup):
[mediawiki/extensions/Wikibase@master] Wrap WAN-cached PropertyInfoLookup with an APCu cache

https://gerrit.wikimedia.org/r/602666

Maintenance_bot moved this task from incoming to in progress on the Wikidata board.Jun 5 2020, 5:15 PM

Change 602666 merged by jenkins-bot:
[mediawiki/extensions/Wikibase@master] Wrap WAN-cached PropertyInfoLookup with an APCu cache

https://gerrit.wikimedia.org/r/602666

Change 603482 had a related patch set uploaded (by Ladsgroup; owner: Ladsgroup):
[mediawiki/extensions/Wikibase@wmf/1.35.0-wmf.35] Wrap WAN-cached PropertyInfoLookup with an APCu cache

https://gerrit.wikimedia.org/r/603482

Change 603482 merged by jenkins-bot:
[mediawiki/extensions/Wikibase@wmf/1.35.0-wmf.35] Wrap WAN-cached PropertyInfoLookup with an APCu cache

https://gerrit.wikimedia.org/r/603482

CDanis added a subscriber: CDanis.Jun 8 2020, 3:05 PM

Mentioned in SAL (#wikimedia-operations) [2020-06-08T15:09:31Z] <ladsgroup@deploy1001> Synchronized php-1.35.0-wmf.35/extensions/Wikibase/lib/includes/Store/CachingPropertyInfoLookup.php: Wrap WAN-cached PropertyInfoLookup with an APCu cache, Part I out of III (T254536) (duration: 00m 59s)

Mentioned in SAL (#wikimedia-operations) [2020-06-08T15:10:54Z] <ladsgroup@deploy1001> Synchronized php-1.35.0-wmf.35/extensions/Wikibase/repo/includes/Store/Sql/SqlStore.php: Wrap WAN-cached PropertyInfoLookup with an APCu cache, Part II out of III (T254536) (duration: 00m 57s)

Mentioned in SAL (#wikimedia-operations) [2020-06-08T15:12:35Z] <ladsgroup@deploy1001> Synchronized php-1.35.0-wmf.35/extensions/Wikibase/client/includes/Store/Sql/DirectSqlStore.php: Wrap WAN-cached PropertyInfoLookup with an APCu cache, Part III out of III (T254536) (duration: 00m 57s)

This should be fixed, but I have no idea how to verify it – I couldn’t find any information on how to see if that cache key is still hot.

Thanks, this looks quite conclusive:


Good enough to close this task?

Ladsgroup closed this task as Resolved.Jun 8 2020, 4:55 PM
Ladsgroup claimed this task.

Good enough to close this task?

Ja

Restricted Application added a project: User-Ladsgroup. · View Herald TranscriptJun 8 2020, 4:55 PM
Maintenance_bot moved this task from In progress to Done on the User-Ladsgroup board.