Page MenuHomePhabricator

Drastically reduce the number of ResourceLoader modules that WikibaseClient registers
Closed, ResolvedPublic

Description

Motivation
As of 7 May 2018 in production, Wikibase's 248 RL modules result in 14,453 bytes (5,210 bytes after gzip) being shipped on every page view of every Wikipedia for every reader. That's 6.4TiB a day of extra load on readers.

Problem
The linkitem module in wikibase client actually uses lots of parts of WikibaseView which is designed for Repo so it makes us to load WikibaseView and all of its ~240 modules in all client wikis.

Suggested Solution / Technical Details
"linkitem module in WikibaseClient, depends on these modules in WikibaseView:

  • 'jquery.ui.ooMenu' (DatavaluesValueview)
  • 'jquery.ui.suggester' (DatavaluesValueview)
  • 'jquery.event.special.eachchange' (DatavaluesValueview)
  • 'util.inherit', (DatavaluesLib)
  • util.highlightSubstring (DatavaluesValueview)

' jquery.wikibase.siteselector' in lib depends on these modules in View:

  • jquery.event.special.eachchange
  • jquery.ui.ooMenu
  • jquery.ui.suggester
  • util.highlightSubstring

Possible solutions:

  • Rewrite linkitem so it doesn't use any of those modules in view, move jquery.wikibase.siteselector to view or
  • Take out lib subdirectoy of DatavaluesValueview and move that from view to lib. Move 'util.inheirt' from WikibaseDataValueLib to DatavaluesValueViewLib (the moved part)

Original bug report:

As of 7 May 2018 in production, Wikibase's 248 RL modules result in 14,453 bytes (5,210 bytes after gzip) being shipped on every page view of every Wikipedia for every reader. That's 6.4TiB a day of extra load on readers. There's even more if you count the jQuery UI stuff uses, but I'll leave that for now.

It looks like Wikibase is trying to use ResourceLoader as a class loader(!) which may be difficult to quickly re-architect, but any reduction would be worthwhile.

Progress

WikibaseClient:

Details

SubjectRepoBranchLines +/-
mediawiki/extensions/Wikibasemaster+193 -6 K
mediawiki/extensions/Wikibasemaster+51 -101
mediawiki/extensions/Wikibasemaster+5 -16
mediawiki/extensions/Wikibasemaster+9 K -247
mediawiki/extensions/Wikibasemaster+0 -1
mediawiki/extensions/Wikibasemaster+89 -89
mediawiki/extensions/Wikibasemaster+10 -21
mediawiki/extensions/Wikibasemaster+45 -117
mediawiki/extensions/Wikibasemaster+9 -28
mediawiki/extensions/Wikibasemaster+2 -8
mediawiki/extensions/Wikibasemaster+8 -52
mediawiki/extensions/Wikibasemaster+5 -26
mediawiki/extensions/Wikibasemaster+2 -20
mediawiki/extensions/Wikibasemaster+3 -12
mediawiki/extensions/Wikibasemaster+21 -25
mediawiki/extensions/Wikibasewmf/1.33.0-wmf.25+86 -92
mediawiki/extensions/Wikibasemaster+86 -92
mediawiki/extensions/Wikibasemaster+12 -68
mediawiki/extensions/Wikibasemaster+15 -96
mediawiki/extensions/Wikibasemaster+10 -60
mediawiki/extensions/Wikibasemaster+24 -55
mediawiki/extensions/Wikibasemaster+34 -69
mediawiki/extensions/Wikibasemaster+21 -58
mediawiki/extensions/Wikibasemaster+16 -46
mediawiki/extensions/Wikibasemaster+14 -63
mediawiki/extensions/Wikibasemaster+52 -135
mediawiki/extensions/Wikibasemaster+36 -105
mediawiki/extensions/Wikibasemaster+3 K -4 K
mediawiki/extensions/Wikibasemaster+11 -38
mediawiki/extensions/Wikibasemaster+7 -45
mediawiki/extensions/Wikibasemaster+3 -20
mediawiki/extensions/Wikibasemaster+42 -60
mediawiki/extensions/Wikibasemaster+48 -68
Show related patches Customize query in gerrit

Event Timeline

There are a very large number of changes, so older changes are hidden. Show Older Changes

Change 501981 merged by jenkins-bot:
[mediawiki/extensions/Wikibase@master] Merge wikibase.termbox.messages into wikibase.termbox (13)

https://gerrit.wikimedia.org/r/501981

Change 502635 had a related patch set uploaded (by Krinkle; owner: Krinkle):
[mediawiki/extensions/Wikibase@wmf/1.33.0-wmf.25] Merge wikibase.termbox.messages into wikibase.termbox (13)

https://gerrit.wikimedia.org/r/502635

Change 502635 abandoned by Krinkle:
Merge wikibase.termbox.messages into wikibase.termbox (13)

https://gerrit.wikimedia.org/r/502635

Krinkle updated the task description. (Show Details)
Krinkle updated the task description. (Show Details)
Krinkle updated the task description. (Show Details)

Change 501980 merged by jenkins-bot:
[mediawiki/extensions/Wikibase@master] Consolidate more view/resources modules (12)

https://gerrit.wikimedia.org/r/501980

Change 508411 had a related patch set uploaded (by Krinkle; owner: Krinkle):
[mediawiki/extensions/Wikibase@master] Consolidate more view/resources modules (13)

https://gerrit.wikimedia.org/r/508411

Change 508411 merged by jenkins-bot:
[mediawiki/extensions/Wikibase@master] Consolidate more view/resources modules (13)

https://gerrit.wikimedia.org/r/508411

Change 509166 had a related patch set uploaded (by Krinkle; owner: Krinkle):
[mediawiki/extensions/Wikibase@master] Consolidate more view/ modules resources (14)

https://gerrit.wikimedia.org/r/509166

Change 509166 merged by jenkins-bot:
[mediawiki/extensions/Wikibase@master] Consolidate more view/ modules resources (14)

https://gerrit.wikimedia.org/r/509166

Change 512407 had a related patch set uploaded (by Krinkle; owner: Krinkle):
[mediawiki/extensions/Wikibase@master] Consolidate more view/ modules resources (15)

https://gerrit.wikimedia.org/r/512407

Change 512407 merged by jenkins-bot:
[mediawiki/extensions/Wikibase@master] Consolidate more view/ modules resources (15)

https://gerrit.wikimedia.org/r/512407

Change 516240 had a related patch set uploaded (by Krinkle; owner: Krinkle):
[mediawiki/extensions/Wikibase@master] Consolidate more view/ modules resources (16)

https://gerrit.wikimedia.org/r/516240

Change 516240 merged by jenkins-bot:
[mediawiki/extensions/Wikibase@master] Consolidate more view/ modules resources (16)

https://gerrit.wikimedia.org/r/516240

Change 518753 had a related patch set uploaded (by Krinkle; owner: Krinkle):
[mediawiki/extensions/Wikibase@master] Consolidate more view/ modules resources (17)

https://gerrit.wikimedia.org/r/518753

Change 518753 merged by jenkins-bot:
[mediawiki/extensions/Wikibase@master] Consolidate more view/ modules resources (17)

https://gerrit.wikimedia.org/r/518753

Change 520926 had a related patch set uploaded (by Krinkle; owner: Krinkle):
[mediawiki/extensions/Wikibase@master] Consolidate more view/ modules resources (18)

https://gerrit.wikimedia.org/r/520926

Change 520926 merged by jenkins-bot:
[mediawiki/extensions/Wikibase@master] Consolidate more view/ modules resources (18)

https://gerrit.wikimedia.org/r/520926

Change 522607 had a related patch set uploaded (by Ladsgroup; owner: Ladsgroup):
[mediawiki/extensions/Wikibase@master] Reduce number of client RL modules

https://gerrit.wikimedia.org/r/522607

Change 522612 had a related patch set uploaded (by Ladsgroup; owner: Ladsgroup):
[mediawiki/extensions/Wikibase@master] Reduce number of client RL modules

https://gerrit.wikimedia.org/r/522612

Change 522613 had a related patch set uploaded (by Ladsgroup; owner: Ladsgroup):
[mediawiki/extensions/Wikibase@master] Use Package files to inject config

https://gerrit.wikimedia.org/r/522613

Change 522607 abandoned by Ladsgroup:
Reduce number of client RL modules

Reason:
Split it to Iec0fe4be7a2781d9b91c72dc3f51528fee2817ce and If711f9251a054efe4c0ffb3718b41419a7c0e3f0. One of them can be complex (deprecating the config variable.)

https://gerrit.wikimedia.org/r/522607

Change 522612 merged by jenkins-bot:
[mediawiki/extensions/Wikibase@master] Reduce number of client RL modules

https://gerrit.wikimedia.org/r/522612

Change 523002 had a related patch set uploaded (by Ladsgroup; owner: Ladsgroup):
[mediawiki/extensions/Wikibase@master] Stop loading WikibaseView in client

https://gerrit.wikimedia.org/r/523002

Actually the underlying problem here is that we load WikibaseView, the frontend designed to show entities on Wikidata on clients, this should not happen. Client and lib depend only on eight modules of view, I move them to lib and we will be done with this.

Change 523011 had a related patch set uploaded (by Ladsgroup; owner: Amir Sarabadani):
[mediawiki/extensions/Wikibase@master] Move view/lib/wikibase-api to lib/resouces/wikibase-api

https://gerrit.wikimedia.org/r/523011

  • wikibase.api.RepoApiError' (wikibase-api)
  • 'jquery.ui.ooMenu' (DatavaluesValueview)
  • 'jquery.ui.suggester' (DatavaluesValueview)
  • 'jquery.event.special.eachchange' (DatavaluesValueview)
  • 'wikibase.api.RepoApi', (wikibase-api)
  • 'wikibase.api.getLocationAgnosticMwApi' (wikibase-api)
  • 'util.inherit', (DatavaluesLib)
  • util.highlightSubstring (DatavaluesValueview)

Except the wikibase-api, the rest actually are not needed and can be replaced by some other functionality or by T226976: Convert linkitem module away from jQuery UI

Change 523011 merged by jenkins-bot:
[mediawiki/extensions/Wikibase@master] Move view/lib/wikibase-api to lib/resouces/wikibase-api

https://gerrit.wikimedia.org/r/523011

@alaa_wmde @Ladsgroup: for the issue of WikibaseClient loading way too much stuff, and how to solve, could you please work together on defining the problem space, and the sensible first iteration of what we would like to achieve? It seems like a dedicated "epic"/"story"/however you want to label it, in any case a separate phabricator ticket, would make the most sense to not miss the focus of this broader ticket.

@alaa_wmde @Ladsgroup: for the issue of WikibaseClient loading way too much stuff, and how to solve, could you please work together on defining the problem space, and the sensible first iteration of what we would like to achieve? It seems like a dedicated "epic"/"story"/however you want to label it, in any case a separate phabricator ticket, would make the most sense to not miss the focus of this broader ticket.

I agree that the ticket has a vague AC (Should we reduce it to 100? 10?) but to me it seems the intention of the authors was to clean the WikibaseView which should not be loaded in client in the first place and @Krinkle was happy that if we just remove that from client and call this done. I can make another ticket for this and focus our efforts there.

Change 525107 had a related patch set uploaded (by Ladsgroup; owner: Ladsgroup):
[mediawiki/extensions/Wikibase@master] Stop loading WikibaseView in WikibaseClient

https://gerrit.wikimedia.org/r/525107

Change 523002 abandoned by Ladsgroup:
Stop loading WikibaseView in client

Reason:
We do it with I0e190b0c723e

https://gerrit.wikimedia.org/r/523002

@Ladsgroup: thanks. It seems to me that fixing the problem quickly first, and then spending some more time without a pressure how to resolve the issue "the right way" would be a very sensible approach.

@Ladsgroup: thanks. It seems to me that fixing the problem quickly first, and then spending some more time without a pressure how to resolve the issue "the right way" would be a very sensible approach.

This is basically what we are intending to do: T203696#5357444 (moving code out of submodule)

Change 525107 merged by jenkins-bot:
[mediawiki/extensions/Wikibase@master] Stop loading WikibaseView in WikibaseClient

https://gerrit.wikimedia.org/r/525107

Change 526149 had a related patch set uploaded (by Ladsgroup; owner: Ladsgroup):
[mediawiki/extensions/Wikibase@master] Make the namespace.js of wikibase-api not a module but a file in all others

https://gerrit.wikimedia.org/r/526149

Change 526149 merged by jenkins-bot:
[mediawiki/extensions/Wikibase@master] Make the namespace.js of wikibase-api not a module but a file in all others

https://gerrit.wikimedia.org/r/526149

From Grafana: WebPageReplay dashboard

capture.png (956×2 px, 133 KB)

The last two grey vertical lines indicate the (final) deployment of the 1.34-wmf.16 branch (it was attempted earlier that day as well and then reverted for unrelated reasons).

From Grafana: WebPageTest dashboard we can see the bandwidth difference as well:

capture.png (668×720 px, 40 KB)

2 KB wouldn't normally have this much impact in JS payloads, but I believe it's able to have this impact because it's taken from the early startup module (less bandwidth taken from other requests, and less CPU time overall for the processing of all that data, which can instead be spent on rendering the page).

Change 522613 merged by jenkins-bot:
[mediawiki/extensions/Wikibase@master] Use Package files to inject config

https://gerrit.wikimedia.org/r/522613

Change 535667 had a related patch set uploaded (by Ladsgroup; owner: Ladsgroup):
[mediawiki/extensions/Wikibase@master] Move WikibaseView-only modules from lib/resource/lib to WikibaseView

https://gerrit.wikimedia.org/r/535667

Change 535667 merged by jenkins-bot:
[mediawiki/extensions/Wikibase@master] Move WikibaseView-only modules from lib/resource/lib to WikibaseView

https://gerrit.wikimedia.org/r/535667

Maintenance_bot moved this task from Incoming to In progress on the User-Ladsgroup board.
Maintenance_bot moved this task from In progress to Done on the User-Ladsgroup board.