Page MenuHomePhabricator

Optimize Wikidata's Composer class loader impact
Closed, ResolvedPublic

Description

While looking at T85182: Composer autoloader is slow and the upstream issue I decided to poke at the data from the flame graph that @Legoktm linked in the upstream bug report. I think there are opportunities to improve the resource footprint required by the Wikidata extension in addition to any optimizations that can be made in the Composer Autoloader class.

Related Objects

Event Timeline

bd808 created this task.Jan 3 2015, 9:29 PM
bd808 raised the priority of this task from to Needs Triage.
bd808 updated the task description. (Show Details)
bd808 added a project: Performance Issue.
bd808 added a subscriber: bd808.

The most expensive Composer related code path in the 2014-12-31 flame graph is:

composerRequirewikidata_1_25wmf12c looks to be the place to really start digging here. This is the inner call from the loop that loads everything in autoload_files.php. Wikidata is loading 18 dependent libraries (directly and indirectly) which specify a "files" autoloader component:

hoo added a subscriber: hoo.Jan 3 2015, 9:31 PM

Is this specifically about the *Wikidata* extension (and not the code from Wikibase etc.) or about the whole pile of code in Wikidata?

Several of these components seem to only use the "files" autoloader to make something show up in Special:Version. Could the new feature to expose Composer vendor libs be used or extended to do this instead?

Lydia_Pintscher triaged this task as Normal priority.Jan 3 2015, 9:47 PM
Lydia_Pintscher added a project: Wikidata.
Lydia_Pintscher set Security to None.
Lydia_Pintscher added subscribers: JeroenDeDauw, aude.

The include -> call_user_func -> closure pattern allows splitting up large files and keeping the global scope clean, but since both closures and call_user_func are bottlenecks for HHVM optimization it might be worth thinking about alternate implementations that use more readily optimized code.

I think the long tail of Composer\Autoload\includeFile -> Composer\Autoload\ClassLoader::loadClass -> Composer\Autoload\includeFile -> Composer\Autoload\ClassLoader::loadClass -> Composer\Autoload\includeFile -> Composer\Autoload\ClassLoader::loadClass may be the class_alias mappings.

I've heard @aude mention this as something that should be eliminated before.

In T85737#953452, @hoo wrote:

Is this specifically about the *Wikidata* extension (and not the code from Wikibase etc.) or about the whole pile of code in Wikidata?

I was looking at the code paths triggered by mediawiki/extensions/Wikidata so in that sense it is about the Wikidata extension, but I think the issues are in both the extension and it's dependencies.

Legoktm closed this task as Resolved.Dec 21 2017, 11:04 PM
Legoktm assigned this task to Addshore.
Legoktm added a subscriber: Legoktm.

Wikidata no longer uses the composer autoloader \o/

In addition, all of the libraries no longer register MediaWiki entrypoints since they are libraries. For the extensions, I think we got rid of most of the weird call_user_func stuff during the switch to extension.json.

Restricted Application added a project: User-Addshore. · View Herald TranscriptDec 21 2017, 11:04 PM