Page MenuHomePhabricator
Create Task
Maniphest T104150

Investigation: Make sure client change handling scales
Closed, ResolvedPublic1 Estimated Story Points
Actions

Description

Given that for example Q159 (Russia) is being used more than 100000 times on ruwiki (mostly with aspect L.ru which makes use run Title::invalidateCache), we should make sure our client change handling can keep up with that.

Right now ChangeHandler iterates over all pages using a given Entity and then hits of the purge/… actions synchronously. I seriously doubt that will work for more than a few thousand pages, at best (given that jobs can only run for a limited time and memory is limited as well).

We need to investigate this and fix change handling (if necessary).

Related Objects

Event Timeline

hoo created this task.Jun 28 2015, 11:34 PM
hoo raised the priority of this task from to High.
hoo updated the task description. (Show Details)
hoo added projects: MediaWiki-extensions-WikibaseClient, Wikidata.
hoo added subscribers: hoo, aude, daniel and 2 others.
Restricted Application added a subscriber: Aklapper. · View Herald TranscriptJun 28 2015, 11:34 PM
JanZerebecki moved this task from incoming to consider for next sprint on the Wikidata board.Jun 29 2015, 2:52 PM
Tobi_WMDE_SW renamed this task from Make sure client change handling scales to Investigation: Make sure client change handling scales.Jun 30 2015, 1:29 PM
Tobi_WMDE_SW added a project: Wikidata-Sprint-2015-06-30.
Tobi_WMDE_SW set Security to None.
Tobi_WMDE_SW edited a custom field.Jun 30 2015, 2:07 PM
hoo closed this task as Resolved.EditedJul 9 2015, 11:59 PM
hoo claimed this task.

Resolving, as this is only about investigating… This is definitely a problem:

Example log entry:

Jul  9 08:28:46 mw1014:  #012Fatal error: request has exceeded memory limit in /srv/mediawiki/php-1.26wmf12/extensions/Wikidata/extensions/Wikibase/client/includes/Usage/Sql/EntityUsageTable.php on line 441

Happened several times today:

hoo@fluorine:/a/mw-log$ grep -c 'Fatal error: request has exceeded memory limit in /srv/mediawiki/php-1.26wmf13/extensions/Wikidata/extensions/Wikibase/client/includes/Usage/Sql/EntityUsageTable.php' hhvm.log 
37
hoo moved this task from Backlog to Done on the Wikidata-Sprint-2015-06-30 board.Jul 10 2015, 12:01 AM
hoo reopened this task as Open.Jul 10 2015, 12:09 AM

Maybe I shouldn't have closed this given we have no concrete ways forward.

@daniel: It would be nice if you could file tickets for improving this as needed (we probably want to implement batching on some level… AFAIR you already looked into this).

hoo closed this task as Resolved.Aug 2 2015, 10:15 PM
hoo mentioned this in T107722: [Task] Make client ChangeHandler scale by batching updates.

As Daniel didn't create any tasks, I went ahead and created T107722.

Content licensed under Creative Commons Attribution-ShareAlike (CC BY-SA) 4.0 unless otherwise noted; code licensed under GNU General Public License (GPL) 2.0 or later and other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL