Page MenuHomePhabricator

Create a dependency graph of existing RL modules
Closed, ResolvedPublic

Description

Spend some time trying to visualize the existing resource loader modules' dependencies. This should allow us to easily identify modules that need to stay RL modules.

Event Timeline

Restricted Application added a subscriber: Aklapper. · View Herald TranscriptSep 12 2019, 11:37 AM
Ladsgroup added a subscriber: Ladsgroup.

This was fun. Wikibase (repo + lib + view + client + lexeme) registers 260 RL module.
This is just the dependency graph of Wikibase modules (not core or other things):


If we just take a look at the modules to untangle them, it's completely useless:

Using graph theory and networkx I found RL modules that I call "entry point". These modules are not being depend on anywhere. It means it's the point code inject the module. The graph analysis gives me these entry points:

wikibase.ui.entityViewInit
wikibase.ui.entitysearch
wikibase.special.newEntity
wikibase.special.mergeItems
wikibase.client.getMwApiForRepo
wikibase.client.init
wikibase.client.miscStyles
jquery.wikibase.linkitem
jquery.util.adaptlettercase
util.ContentLanguages
wikibase.serialization
jquery.valueview.experts.SuggestedStringValue
wikibase.common
wikibase.lexeme.lexemeview
wikibase.lexeme.special.NewLexeme.styles
wikibase.lexeme.special.NewLexeme
wikibase.lexeme.styles

(It also can mean the module is completely unused and can be dropped or there's soft dependencies like lazy loading. Needs more investigation)

Then I started categorizing RL modules whether they are directly or indirectly are part of an entry point or several. These modules showed up as shared between the RL modules:

jquery.wikibase.entityselector
mw.config.values.wbRefTabsEnabled
wikibase
wikibase.serialization.EntityDeserializer
wikibase.api.RepoApi
wikibase.api.getLocationAgnosticMwApi
wikibase.entityChangers.EntityChangersFactory
dataValues.values
jquery.valueview.experts.StringValue
mw.config.values.wbRepo
valueFormatters
valueParsers.ValueParserStore
wikibase.datamodel
jquery.wikibase.siteselector
jquery.wikibase.wbtooltip
wikibase.api.RepoApiError
wikibase.sites
mw.config.values.wbSiteDetails
wikibase.buildErrorOutput
util.inherit
dataValues.DataValue
jquery.animateWithEvent
jquery.AnimationEvent
jquery.PurposedCallbacks
jquery.focusAt
jquery.inputautoexpand
jquery.event.special.eachchange
jquery.ui.ooMenu
jquery.util.getscrollbarwidth
util.CombiningMessageProvider
util.HashMessageProvider
jquery.ui.suggester
util.highlightSubstring
jquery.ui.languagesuggester
jquery.ui.toggler
util.Extendable
util.Notifier
wikibase.datamodel.EntityId
wikibase.datamodel.Item
wikibase.datamodel.Property
wikibase.datamodel.PropertyNoValueSnak
wikibase.datamodel.PropertySomeValueSnak
wikibase.datamodel.PropertyValueSnak
wikibase.datamodel.__namespace
wikibase.datamodel.Claim
wikibase.datamodel.SnakList
wikibase.datamodel.Entity
wikibase.datamodel.FingerprintableEntity
wikibase.datamodel.Fingerprint
wikibase.datamodel.MultiTermMap
wikibase.datamodel.TermMap
wikibase.datamodel.Group
wikibase.datamodel.GroupableCollection
wikibase.datamodel.SiteLinkSet
wikibase.datamodel.StatementGroupSet
wikibase.datamodel.List
wikibase.datamodel.Map
wikibase.datamodel.MultiTerm
wikibase.datamodel.Snak
wikibase.datamodel.Reference
wikibase.datamodel.ReferenceList
wikibase.datamodel.SiteLink
wikibase.datamodel.Set
wikibase.datamodel.Statement
wikibase.datamodel.StatementGroup
wikibase.datamodel.StatementList
wikibase.datamodel.Term
globeCoordinate.js
dataValues
dataValues.TimeValue
valueParsers
wikibase.serialization.__namespace
wikibase.serialization.StrategyProvider
wikibase.serialization.ClaimDeserializer
wikibase.serialization.SnakListDeserializer
wikibase.serialization.Deserializer
wikibase.serialization.ItemDeserializer
wikibase.serialization.PropertyDeserializer
wikibase.serialization.FingerprintDeserializer
wikibase.serialization.MultiTermMapDeserializer
wikibase.serialization.TermMapDeserializer
wikibase.serialization.SiteLinkSetDeserializer
wikibase.serialization.StatementGroupSetDeserializer
wikibase.serialization.MultiTermDeserializer
wikibase.serialization.ReferenceListDeserializer
wikibase.serialization.ReferenceDeserializer
wikibase.serialization.SiteLinkDeserializer
wikibase.serialization.SnakDeserializer
wikibase.serialization.StatementGroupDeserializer
wikibase.serialization.StatementListDeserializer
wikibase.serialization.StatementDeserializer
wikibase.serialization.TermDeserializer
wikibase.serialization.ClaimSerializer
wikibase.serialization.SnakListSerializer
wikibase.serialization.TermMapSerializer
wikibase.serialization.Serializer
wikibase.serialization.ReferenceListSerializer
wikibase.serialization.ReferenceSerializer
wikibase.serialization.SnakSerializer
wikibase.serialization.StatementSerializer
wikibase.serialization.TermSerializer
jquery.valueview
jquery.valueview.valueview
jquery.valueview.Expert
jquery.valueview.ExpertStore
jquery.valueview.experts
jquery.valueview.ViewState
jquery.valueview.experts.EmptyValue
jquery.valueview.experts.UnsupportedValue
jquery.ui.EditableTemplatedWidget
wikibase.templates
jquery.wikibase.entityview
jquery.wikibase.listview
jquery.wikibase.referenceview
jquery.wikibase.statementview
jquery.wikibase.statementview.RankSelector.styles
wikibase.utilities
wikibase.view.__namespace
wikibase.utilities.ClaimGuidGenerator
wikibase.view.ControllerViewFactory
wikibase.view.ViewFactory
wikibase.view.ReadModeViewFactory
wikibase.lexeme

It doesn't mean they can't be merged, it means it'll be slightly more complex (not to mention the shared modules can be merged internally too, I should double check that) but 120 modules solely depend on one entry point and can be merged up to the entry point. Here's the list:

wikibase.ui.entitysearch (2 modules):

wikibase.ui.entityViewInit (56 modules):

wikibase.lexeme.lexemeview (46 modules)

wikibase.lexeme.special.NewLexeme (7 modules)

wikibase.serialization (14 modules)

Fixing these help us find out what can be done on shared modules.

Restricted Application added a project: User-Ladsgroup. · View Herald TranscriptSep 12 2019, 7:31 PM
Rosalie_WMDE renamed this task from Try to create a dependency graph of existing RL modules to Create a dependency graph of existing RL modules.Sep 13 2019, 11:31 AM

I forgot to put the code for this.

RL_graph.py
import json
import sys
import random
import math
from collections import defaultdict

import networkx as nx
import matplotlib.pyplot as plt
from networkx.drawing.nx_agraph import graphviz_layout
from networkx.drawing.nx_pydot import write_dot

G = nx.DiGraph()
with open('rl_modules.json', 'r') as f:
    wikibase_modules = json.loads(f.read())

with open(sys.argv[1], 'r') as f:
    modules = json.loads(f.read())
for i in range(len(modules)):
    module = modules[i]
    if module[0] not in wikibase_modules:
        continue
    if len(module) < 3:
        G.add_node(module[0])
        continue
    for module_dep in module[2]:
        if modules[module_dep][0] in wikibase_modules:
            G.add_edge(module[0], modules[module_dep][0])

print(G.number_of_nodes())
entry_points = []
for node in G.in_degree():
    if node[1] == 0:
        entry_points.append(node[0])
for entry_point in entry_points:
    print(entry_point)
directions = {}
for node in G.nodes():
    if node in entry_points:
        continue
    node_paths = {}
    for entry_point in entry_points:
        try:
            path = nx.shortest_path(G, entry_point, node)
        except:
            continue
        node_paths[entry_point] = path
    if len(node_paths) > 1:
        print('shared', node)
        continue
    directions[node] = node_paths

per_entry_point_nodes = defaultdict(set)
for node in directions:
    for entry_point in directions[node]:
        per_entry_point_nodes[entry_point] = per_entry_point_nodes[entry_point].union(set(directions[node][entry_point]))

write_dot(G,'G.dot')
sys.exit()
for entry_point in per_entry_point_nodes:
    subgraph = G.subgraph(per_entry_point_nodes[entry_point])
    size_factor =round(math.sqrt((len(subgraph.nodes())**2 ) / 25))
    if size_factor < 3:
        size_factor = 3
    plt.plot()
    fig= plt.figure(figsize=(size_factor*4,size_factor*3))
    pos=graphviz_layout(subgraph, prog='dot')
    nx.draw(subgraph, pos, with_labels=True)
    write_dot(subgraph,'nx_' + entry_point + '.dot')
    plt.savefig('nx_' + entry_point + '.png')

rl_modules.json is list of names of modules registered by Wikibase so we can filter and only care about those. and first argument should be the json file of RL modules (output of startup module in Wikidata). Something like this:

[
    [
        "site",
        "0jxs9iy",
        [
            1
        ]
    ],
.
.
.
]
Ladsgroup added a comment.EditedSep 16 2019, 1:10 PM

After dropping wikibase.serializtion. The structure has slightly changed:

wikibase.ui.entityViewInit 67:


wikibase.lexeme.lexemeview 48:

Ladsgroup closed this task as Resolved.Oct 4 2019, 10:59 AM
Maintenance_bot moved this task from In progress to Done on the User-Ladsgroup board.