Many Wikipedia's Wikidata module iterate over all entity claims if a Statement is searched for by property label
Closed, ResolvedPublic
Actions

Assigned To

Authored By

	hoo
	Oct 12 2017, 9:03 PM

Description

Yesterday I tried to enable Statement usage tracking on cawiki (which means we're exactly tracking which Statement has been used, and not just that "all entity data" is used). When doing this I discovered that many many usages on cawiki are needlessly added due to a performance bug in their Mòdul:Wikidata (https://ca.wikipedia.org/w/index.php?title=M%C3%B2dul_Discussi%C3%B3:Wikidata&oldid=18938979#Critical_performance_improvement).

The problematic code

		-- otherwise, iterate over all properties, fetch their labels and compare this to the given property name
		for k, v in pairs(entity.claims) do
			if mw.wikibase.label(k) == property then return v end
		end
		return

can easily be replaced with

		property = mw.wikibase.resolvePropertyId(property)
		if not property then return end

		return entity.claims[property]

The problematic code is also on several other wikis: P6114

Details

	Subject	Repo	Branch	Lines +/-
	getAllStatements and access to property by name	mediawiki/extensions/Wikibase	master	+134 -13

Customize query in gerrit

Related Objects
Search...

View Standalone Graph

This task is connected to more than 200 other tasks. Only direct parents and subtasks are shown here. Use View Standalone Graph to show more of the graph.

Status	Assigned	Task
		· · ·
Resolved	Lydia_Pintscher	T49288 [Epic] Track Wikidata entity usage on client pages
Resolved	Ladsgroup	T178114 Many Wikipedia's Wikidata module iterate over all entity claims if a Statement is searched for by property label
		· · ·

Event Timeline

hoo created this task.Oct 12 2017, 9:03 PM

Restricted Application added a subscriber: Aklapper. · View Herald TranscriptOct 12 2017, 9:03 PM

Easy/workaround - When code access pairs of pseudo table (entity.claims as here and also entity.labels and entity.descriptions once T172914 is getting merged ) we should probably workaround it upstream either from UsageAggregator (T178079) or from Lua (whenever access pairs for entity.claims, count it as C.* instead of many rows).
- wbc_entity_usage will not get overloaded with too many rows. This is just workaround to avoid unintentional usage, as it still make the EU not efficient (from rc side)
Medium - Le Tour de Wikí 2017 go over all wikis and fix them (there is no central fix - T121470 T41610)

hoo mentioned this in T151717: Usage tracking: record which statement group is used.Oct 13 2017, 8:21 AM

Change 383990 had a related patch set uploaded (by Eranroz; owner: Eranroz):
[mediawiki/extensions/Wikibase@master] Access to property by name

https://gerrit.wikimedia.org/r/383990

gerritbot added a project: Patch-For-Review.Oct 13 2017, 8:47 AM

Brought up on enwiki: https://en.wikipedia.org/wiki/Module_talk:Wikidata#Critical_performance_improvement

Capankajsmilyo subscribed.Oct 13 2017, 10:17 AM

eranroz mentioned this in T166056: Add mw.wikibase.entity:getStatements for convenience.Oct 14 2017, 7:54 AM

Change 383990 merged by jenkins-bot:
[mediawiki/extensions/Wikibase@master] getAllStatements and access to property by name

https://gerrit.wikimedia.org/r/383990

This will go live this week, Would be good to measure the impact.

In T178114#3888964, @Ladsgroup wrote:

This will go live this week, Would be good to measure the impact.

The above patch (383990) doesn't have any impact. wikis have to adopt it, and this is just a convenient method to avoid improper property usage

Given that we built T185693: Implement a (more liberal) usage aspect deduplicater (days: 3) it can't blow up the database anymore.

Ladsgroup mentioned this in T188730: [C-DIS][SW] Enable statement usage tracking on Commons and Co.Mar 2 2018, 1:30 PM

I think this task can be considered as resolved (mainly thanks to T185693) from engineering POV.
If there are still wikis using such iterations - they should fix it, but it is now on their (community) hands.
I suggest that before moving this task to resolved, we should ask Community-Relations-Support to communicate to wikis to be aware to it.

Anyway, we should keep an open eyes on wikis with abnormal usage tracking - this may be more fruitful and surely more fun to do with ML/outliers detection based on the data (wbc_entity_usage) rather than greping the code :)

BTW I fixed it in the following diffs for some wikis:

MW template:
https://www.mediawiki.org/w/index.php?title=Module:Wikidata&diff=2728318&oldid=2412211

I wrote an script and fixed all of them in one go except the ones that were protected.

ladsgroup@terbium:~$ mwgrep --module "otherwise, iterate over all properties, fetch their labels and compare this to the given property name" --max-results 200
arwiki              Module:Wikidata
bgwiki              Module:Wikidata
iawiki              Module:Wikidata
idwiki              Module:Wikidata
incubatorwiki       Module:Wp/lrc/Wikidata
kuwiki              Module:Wikidata
mywiki              Module:Wikidata
ptwiki              Module:Wikidata
thwiki              Module:Wikidata
viwiki              Module:Wikidata
zhwiki              Module:Wikidata
zhwiki              Module:Wikidata2
zhwiki              Module:沙盒/Dabao qian/Wikidata
zhwiki              Module:沙盒/PhiLiP/Wikidata

I think getting this done is good because they would get only related notifications instead of getting it for every edit that happens on the statements.

Ladsgroup updated the task description. (Show Details)Mar 2 2018, 3:59 PM

Ladsgroup claimed this task.Mar 2 2018, 4:16 PM

Ladsgroup edited projects, added Wikidata-Sprint-2018-02-28; removed Patch-For-Review.

Restricted Application added a project: User-Ladsgroup. · View Herald TranscriptMar 2 2018, 4:16 PM

done arwiki
https://ar.wikipedia.org/w/index.php?title=Module:wikidata&type=revision&diff=27580612&oldid=20545352

Thank you very much @Mr.Ibrahem

Kenrick95 awarded a token.Mar 3 2018, 12:41 AM

Change done in idwiki: https://id.wikipedia.org/w/index.php?title=Modul:Wikidata&diff=13739172&oldid=10246681

Thanks @Ladsgroup for the notification in [Wikitech-ambassadors]

Change done in viwiki: https://vi.wikipedia.org/w/index.php?title=M%C3%B4_%C4%91un:Wikidata&diff=37494708&oldid=29164668

Thank you for notification :)

Thank you @minhhuy and @Kenrick95. I really appreciate it

Lucas_Werkmeister_WMDE added a project: Wikidata-Ministry-Of-Magic.Mar 6 2018, 10:44 AM

Lucas_Werkmeister_WMDE moved this task from Tasks to Blocked on the Wikidata-Ministry-Of-Magic board.

Only six wikis had this problem: ia, pt, my, bg, ku, th. I sent a message in talk page of them and ptwiki, iawiki and mywiki got fixed now \o/ Only three small wikis left now.

Many Wikipedia's Wikidata module iterate over all entity claims if a Statement is searched for by property labelClosed, ResolvedPublicActions

Description

Details

Related ObjectsSearch...

Event Timeline

Many Wikipedia's Wikidata module iterate over all entity claims if a Statement is searched for by property label
Closed, ResolvedPublic
Actions

Related Objects
Search...