Page MenuHomePhabricator
Create Task
Maniphest T243955

CachingPropertyInfoLookup doesn't cache lookups internally
Closed, ResolvedPublic
Actions

Description

(Found via auditing the performance)

In rendering a page in client that uses Wikidata with lua, there's 1823 calls to memcached from Wikibase\Lib\Store\CachingPropertyInfoLookup::getPropertyInfoFromWANCache. This means every time it needs a property info, it makes a call over network to mcrouter/memcached. The client page definitely doesn't use 1832 different properties of Wikidata. This can be a great performance boost by keeping the loaded properties as a class attribute and short circuit lookups.

Details

SubjectRepoBranchLines +/-
Cache PropertyInfoLookup internallymediawiki/extensions/Wikibasewmf/1.35.0-wmf.18+25 -18
Cache PropertyInfoLookup internallymediawiki/extensions/Wikibasewmf/1.35.0-wmf.16+25 -18
Cache PropertyInfoLookup internallymediawiki/extensions/Wikibasemaster+25 -18
Customize query in gerrit

Related Objects
Search...

Event Timeline

Ladsgroup created this task.Jan 30 2020, 1:07 PM
Ladsgroup mentioned this in T243915: Performance audit of Wikibase/Wikidata, Jan 2020.
Addshore triaged this task as High priority.Feb 2 2020, 8:52 PM
Addshore moved this task from Incoming to Needs Tech Work on the Wikidata-Campsite board.

At one point we did also cache this for ~10-20s on machines.
I both added and removed that last year.
It was removed in https://gerrit.wikimedia.org/r/#/c/mediawiki/extensions/Wikibase/+/498348/18/repo/includes/Store/Sql/SqlStore.php

However that was to get around fetching the large amount of data (as the whole table is fetched) rather than stopping multiple calls.

I see in that profile it says "WANObjectCache::fetchOrRegenerate 2059 calls" How many of them are for the property info?
We should definitely do something about this.

I guess our two options are wrap in another local cache as in the gerrit patch linked above.
Or use the process cache functionality built into the WAN cache.
Thoughts?

Ladsgroup added a comment.Feb 4 2020, 3:56 PM
In T243955#5843631, @Addshore wrote:

At one point we did also cache this for ~10-20s on machines.
I both added and removed that last year.
It was removed in https://gerrit.wikimedia.org/r/#/c/mediawiki/extensions/Wikibase/+/498348/18/repo/includes/Store/Sql/SqlStore.php

However that was to get around fetching the large amount of data (as the whole table is fetched) rather than stopping multiple calls.

I see in that profile it says "WANObjectCache::fetchOrRegenerate 2059 calls" How many of them are for the property info?

1823 of them: https://performance.wikimedia.org/xhgui/run/symbol?id=5e31d1ba3f3dfab22d4d158a&symbol=Wikibase%5CLib%5CStore%5CCachingPropertyInfoLookup%3A%3AgetPropertyInfoFromWANCache

gerritbot added a comment.Feb 4 2020, 5:41 PM

Change 570100 had a related patch set uploaded (by Ladsgroup; owner: Ladsgroup):
[mediawiki/extensions/Wikibase@master] Cache PropertyInfoLookup internally

https://gerrit.wikimedia.org/r/570100

gerritbot added a project: Patch-For-Review.Feb 4 2020, 5:41 PM
Addshore edited projects, added Wikidata-Campsite (Wikidata-Campsite-Iteration-∞ (On Hold)); removed Wikidata-Campsite.Feb 4 2020, 5:58 PM
Addshore moved this task from To Do (prioritised from top to bottom) to Doing on the Wikidata-Campsite (Wikidata-Campsite-Iteration-∞ (On Hold)) board.
Maintenance_bot moved this task from incoming to in progress on the Wikidata board.Feb 4 2020, 6:15 PM
Maintenance_bot moved this task from Incoming to In progress on the User-Ladsgroup board.
Addshore claimed this task.Feb 4 2020, 6:40 PM
Addshore moved this task from Doing to Peer Review on the Wikidata-Campsite (Wikidata-Campsite-Iteration-∞ (On Hold)) board.
Restricted Application added a project: User-Addshore. · View Herald TranscriptFeb 4 2020, 6:40 PM
gerritbot added a comment.Feb 5 2020, 11:13 AM

Change 570100 merged by jenkins-bot:
[mediawiki/extensions/Wikibase@master] Cache PropertyInfoLookup internally

https://gerrit.wikimedia.org/r/570100

ReleaseTaggerBot added a project: MW-1.35-notes (1.35.0-wmf.19; 2020-02-11).Feb 5 2020, 12:00 PM
Maintenance_bot removed a project: Patch-For-Review.Feb 5 2020, 12:10 PM
gerritbot added a comment.Feb 5 2020, 12:13 PM

Change 570300 had a related patch set uploaded (by Ladsgroup; owner: Addshore):
[mediawiki/extensions/Wikibase@wmf/1.35.0-wmf.16] Cache PropertyInfoLookup internally

https://gerrit.wikimedia.org/r/570300

gerritbot added a project: Patch-For-Review.Feb 5 2020, 12:14 PM

Change 570301 had a related patch set uploaded (by Ladsgroup; owner: Addshore):
[mediawiki/extensions/Wikibase@wmf/1.35.0-wmf.18] Cache PropertyInfoLookup internally

https://gerrit.wikimedia.org/r/570301

gerritbot added a comment.Feb 5 2020, 12:39 PM

Change 570300 merged by jenkins-bot:
[mediawiki/extensions/Wikibase@wmf/1.35.0-wmf.16] Cache PropertyInfoLookup internally

https://gerrit.wikimedia.org/r/570300

gerritbot added a comment.Feb 5 2020, 12:45 PM

Change 570301 merged by jenkins-bot:
[mediawiki/extensions/Wikibase@wmf/1.35.0-wmf.18] Cache PropertyInfoLookup internally

https://gerrit.wikimedia.org/r/570301

Ladsgroup added a comment.Feb 5 2020, 12:54 PM

Testing these patches in mwdebug1001, the number of memcached calls went from 1832 to 37: https://performance.wikimedia.org/xhgui/run/symbol?id=5e3aba323f3dfa5c659534da&symbol=Wikibase%5CLib%5CStore%5CCachingPropertyInfoLookup%3A%3AgetPropertyInfo

ReleaseTaggerBot edited projects, added MW-1.35-notes (1.35.0-wmf.18; 2020-02-04); removed MW-1.35-notes (1.35.0-wmf.19; 2020-02-11).Feb 5 2020, 1:00 PM
Maintenance_bot removed a project: Patch-For-Review.Feb 5 2020, 1:10 PM
Stashbot added a comment.Feb 5 2020, 1:15 PM

Mentioned in SAL (#wikimedia-operations) [2020-02-05T13:15:02Z] <ladsgroup@deploy1001> Synchronized php-1.35.0-wmf.16/extensions/Wikibase/lib/includes/Store/CachingPropertyInfoLookup.php: SWAT: [[gerrit:570301|Cache PropertyInfoLookup internally]] (T243955) (duration: 01m 07s)

Addshore added a comment.EditedFeb 5 2020, 1:21 PM

image.png (524×1 px, 66 KB)

image.png (434×1 px, 89 KB)

Addshore closed this task as Resolved.Feb 5 2020, 1:23 PM
Addshore reassigned this task from Addshore to Ladsgroup.
Addshore moved this task from Peer Review to Done on the Wikidata-Campsite (Wikidata-Campsite-Iteration-∞ (On Hold)) board.
Stashbot added a comment.Feb 5 2020, 1:24 PM

Mentioned in SAL (#wikimedia-operations) [2020-02-05T13:24:46Z] <ladsgroup@deploy1001> Synchronized php-1.35.0-wmf.18/extensions/Wikibase/lib/includes/Store/CachingPropertyInfoLookup.php: SWAT: [[gerrit:570301|Cache PropertyInfoLookup internally]] (T243955) (duration: 01m 07s)

Maintenance_bot moved this task from In progress to Done on the User-Ladsgroup board.Feb 5 2020, 2:15 PM
aaron mentioned this in T248890: MW Memcached get hit ratio trend over the past months.Mar 30 2020, 7:54 PM
Content licensed under Creative Commons Attribution-ShareAlike (CC BY-SA) 4.0 unless otherwise noted; code licensed under GNU General Public License (GPL) 2.0 or later and other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL