Page MenuHomePhabricator
Create Task
Maniphest T243915

Performance audit of Wikibase/Wikidata, Jan 2020
Closed, ResolvedPublic
Actions

Description

After the outage of T243713: Time-out error; Babel/WikibaseRepo being somehow uncached, overloading the API, and causing general outage caused by T243725: HtmlPageLinkRendererBeginHookHandler creates a LanguageFallbackChain when it may not be needed, it would be a good to audit common bits of Wikibase/Wikidata to find some more performance bottlenecks avoiding issues in the future.

Some common parts are:

Related Objects
Search...

Event Timeline

Ladsgroup created this task.Jan 29 2020, 4:16 PM
Restricted Application added a project: User-Ladsgroup. · View Herald TranscriptJan 29 2020, 4:16 PM
Restricted Application added a subscriber: Aklapper. · View Herald Transcript
Ladsgroup updated the task description. (Show Details)Jan 29 2020, 4:17 PM
Krinkle awarded a token.Jan 29 2020, 4:18 PM
Krinkle added a project: Performance-Team (Radar).
Krinkle moved this task from Limbo to Watching on the Performance-Team (Radar) board.
Ladsgroup added a subtask: T243729: Check Wikibase services for expensive setup patterns (db & api calls etc) & fix / remove.Jan 29 2020, 4:19 PM
Ladsgroup added a subtask: T243912: The new PrefetchingTermLookup doesn't "prefetch".
Ladsgroup updated the task description. (Show Details)Jan 29 2020, 4:23 PM
Maintenance_bot moved this task from incoming to in progress on the Wikidata board.Jan 29 2020, 5:15 PM
Maintenance_bot moved this task from Incoming to In progress on the User-Ladsgroup board.
Ladsgroup updated the task description. (Show Details)Jan 29 2020, 6:29 PM
Ladsgroup updated the task description. (Show Details)Jan 29 2020, 6:39 PM
Ladsgroup added a comment.EditedJan 29 2020, 7:03 PM

Non-Wikibase bits that look weird:

  • After each purge or edit, EventBus does two MultiHttpClient::runMultiCurl (an example). This is the longest part of most of these requests: https://performance.wikimedia.org/xhgui/run/view?id=5e31d1b03f3dfaaf2d6bf636
  • The biggest memory hog in parsing a page with lots of lua is not the lua or the parser, it's Liuggio\StatsdClient\StatsdClient::doReduce. One example says 86MB (77% of total memory usage)
  • A lot of rendering pages in client is loading the sites table, here's an example. I guess T113034 and migrating the sites to a better cache would help.
  • Extensions Registry is really slow. Specially trying to call redis.
  • AtEase could be used and make things faster specially in DB Queries.
  • API responses spend interesting amount of time in Site::unserialize and UtfNormal\Validator::cleanUp
Ladsgroup updated the task description. (Show Details)Jan 29 2020, 8:11 PM
Ladsgroup moved this task from To Do (prioritised from top to bottom) to Peer Review on the Wikidata-Campsite (Wikidata-Campsite-Iteration-∞ (On Hold)) board.Jan 29 2020, 8:13 PM
Addshore added a comment.Jan 30 2020, 9:21 AM

I think everything in the description really needs to be split into a sub ticket so that conversation in here doesn't get very messy.

Agabi10 subscribed.Jan 30 2020, 9:59 AM
Ladsgroup mentioned this in T243950: RDF output of an entity loads all referenced entities, it shouldn't .Jan 30 2020, 11:18 AM
Ladsgroup updated the task description. (Show Details)
Ladsgroup updated the task description. (Show Details)Jan 30 2020, 11:45 AM
Ladsgroup updated the task description. (Show Details)Jan 30 2020, 12:13 PM
Ladsgroup updated the task description. (Show Details)Jan 30 2020, 12:54 PM
Ladsgroup updated the task description. (Show Details)Jan 30 2020, 1:08 PM
Ladsgroup updated the task description. (Show Details)Jan 30 2020, 1:17 PM
Ladsgroup added a comment.Jan 30 2020, 4:10 PM
In T243915#5839640, @Addshore wrote:

I think everything in the description really needs to be split into a sub ticket so that conversation in here doesn't get very messy.

Done

Jdforrester-WMF subscribed.Feb 4 2020, 6:03 PM
Addshore closed subtask T243955: CachingPropertyInfoLookup doesn't cache lookups internally as Resolved.Feb 5 2020, 1:23 PM
Addshore removed Ladsgroup as the assignee of this task.Feb 6 2020, 9:28 AM
Addshore edited projects, added Wikidata-Campsite; removed Wikidata-Campsite (Wikidata-Campsite-Iteration-∞ (On Hold)).
Addshore moved this task from Incoming to Needs Tech Work on the Wikidata-Campsite board.
Ladsgroup closed subtask T243951: Lua creates lots of ItemId and PropertyId objects as Declined.Feb 25 2020, 12:16 PM
Addshore closed subtask T243729: Check Wikibase services for expensive setup patterns (db & api calls etc) & fix / remove as Resolved.Mar 4 2020, 4:11 PM
Addshore closed subtask T243912: The new PrefetchingTermLookup doesn't "prefetch" as Resolved.May 5 2020, 11:47 AM
Michael subscribed.Jun 18 2020, 3:44 PM
Ladsgroup added a comment.Oct 6 2020, 5:29 PM

Should we call this done?

Ladsgroup closed subtask T243952: JSON representation of an item should not need property info as Invalid.Oct 6 2020, 5:32 PM
Ladsgroup mentioned this in T269675: Do an anlysis of wikidata requests in xhgui and arclamp.Dec 8 2020, 2:01 PM
Addshore added a project: [DEPRECATED] wdwb-tech.Apr 14 2021, 9:38 AM
Addshore closed this task as Resolved.Apr 14 2021, 3:36 PM
Addshore claimed this task.
In T243915#6522442, @Ladsgroup wrote:

Should we call this done?

Yes, lets.
Leaving the subtasks open though that were created as a result of this

Restricted Application added a project: User-Addshore. · View Herald TranscriptApr 14 2021, 3:36 PM
Addshore closed subtask T243954: Cache SnakFormat::getFallbackChain() inside the class as Resolved.Aug 24 2021, 12:08 PM
Content licensed under Creative Commons Attribution-ShareAlike (CC BY-SA) 4.0 unless otherwise noted; code licensed under GNU General Public License (GPL) 2.0 or later and other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL