Page MenuHomePhabricator

Start using webpack for dependency management and bundling
Open, Needs TriagePublic

Description

Motivation
Our "dependency management" (more accurately, requiring js files/modules within other ones) of javascript files/modules are being currently handled through the Resource Loader.
Resource Loader dependency management feature was not design to also host this kind of use-case.

Using javascripts's require/import solution is more appropriate for that use-case. Using a packing solution on top of that is also appropriate to reduce the amount of http traffic overhead if we load these files asynchronously (which is still not supported by all browsers to this date https://caniuse.com/#feat=es6-module-dynamic-import) as well as

Problem

  • Our frontend codebase in Wikibase Repo is pretty modular which is good but we are using RL for dependency management and class loading resulting to have 300 RL modules with a very complex dependency relationship that causes TTFB to be high and pretty big network overhead (e.g. T203696: Drastically reduce the number of ResourceLoader modules that WikibaseClient registers: "It looks like Wikibase is trying to use ResourceLoader as a class loader(!) which may be difficult to quickly re-architect, but any reduction would be worthwhile.").
  • Testing modules and developing them is rather hard due to the deep integration with RL.
  • RL's minification method is also a little bit outdated and inefficient (See T49437: Consider a pipeline for enhanced minification (e.g. support UglifyJS)). Webpack solves or improves all of these issues.
  • Currently we can't use ES6.

Suggested Solution
Use Webpack to bundle our javascript modules in single files. Webpack is chosen for the following reasons:

Impact

  • On dev productivity: can be big, allowing us to use modern javascript through Babel
  • On performance: Webpack is not the only mean to improve performance.
    • Every module removed from lib or client will have ~30GB/day reduction on network. We planning to drop probably around 10-20 of them
    • Every module removed from view or repo will have ~55 MB/day (wikidata) + 90 MB/day (commons) reduction on network. We are planning to drop probably around 100-200 of them.
    • TTFB is hard to measure. Can be measured afterwards.
    • The network reduction due to using webpack instead of RL manification is hard to measure. Can be measured afterwards.

Estimated Trailblazing length
couple of weeks (2-3 weeks for 100-150% FTE)

Event Timeline

Change 524515 had a related patch set uploaded (by Ladsgroup; owner: Ladsgroup):
[mediawiki/extensions/WikibaseLexeme@master] Start using webpack

https://gerrit.wikimedia.org/r/524515

alaa_wmde moved this task from Backlog to Todo: Focus on the User-Alaa board.
alaa_wmde updated the task description. (Show Details)Jul 24 2019, 9:43 PM

above patch was meant to be as a proof of concept

As of 7 May 2018 in production, Wikibase's 248 RL modules result in 14,453 bytes (5,210 bytes after gzip) being shipped on every page view of every Wikipedia for every reader. That's 6.4TiB a day of extra load on readers.

Just a clarification, this is for Client that is being tackled in another way, client will also enjoy some of the benefits of webpack but the main benefits will be towards performance of Wikidata's frontend by squeezing more than 300 modules we register in repo to maybe at most 100.

@Ladsgroup can you rephrase the Problem better, perhaps adding more details? I have a feeling the current one isn't really reflecting the actual problem behind our motivation here

alaa_wmde updated the task description. (Show Details)Tue, Jul 30, 2:14 PM
Ladsgroup updated the task description. (Show Details)Wed, Jul 31, 9:45 AM
Ladsgroup updated the task description. (Show Details)Wed, Jul 31, 9:53 AM
Ladsgroup updated the task description. (Show Details)

@Ladsgroup does the MobileFrontend skin use minification in webpack, or leave it down the RL?
My main though here is it would be nice to not have minified things with debug=true?

Every module removed from lib or client will have ~30GB/day reduction on network. We planning to drop probably around 10-20 of them
Every module removed from view or repo will have ~55 MB/day (wikidata) + 90 MB/day (commons) reduction on network. We are planning to drop probably around 100-200 of them.

What impact would are the clients predicted to see bandwidth / response time wise?
Or are these the bits that are hard to predict?

What is the deployment / development workflow going to look like here?
building the JS and commiting it to wikibase.git for review? I guess that is what is done with T195475 ?

I dont have any major concerns with any of this really.

@Ladsgroup does the MobileFrontend skin use minification in webpack, or leave it down the RL?

Webpack does it. But also it does a better job in minfication like tree-shaking, and/or other configurable magic like renaming variables, etc. RL minfication is pretty inefficient (but fast so it can respond in timely manner). T49437: Consider a pipeline for enhanced minification (e.g. support UglifyJS) talks more about this, it uses uglifyJS but the same principle can be applied everywhere. In short, it can reduce size of our bundles up to 10%.

My main though here is it would be nice to not have minified things with debug=true?

Yes, that's the biggest caveat of webpack IMO but most debuggers have prettifies you can use, so it's not that bad.

What impact would are the clients predicted to see bandwidth / response time wise?
Or are these the bits that are hard to predict?

We already saw them in real-life, dropped 137 module and T203696#5387672 happened, It's too noisy to give an exact value but you can assume dropping 137 module, increased speed of pages in desktop with latency of 100ms by 4%, that adds up in case of clients. For repos, not that much because our bottleneck is not module names but actually the whole modules are pretty big and a better minifier might give a hand. (There are other ways to improve things like lazy loading but that's definitely outside of scope of this work). Dropping name of modules might give a small boost to commons though.

What is the deployment / development workflow going to look like here?
building the JS and commiting it to wikibase.git for review? I guess that is what is done with T195475 ?

Yes, basically before committing a patch, you run npm build, and it builds for you and you commit both codes (here's an example).During development you run "npm start" and it listens to your files and if they change, it rebuilds the build file(s).

This is the RFC that I mentioned might be of interest when looking into this: T133462.

Questions from the bonfire that we might want to answer/consider as part of the trailblazing

  • How build process of webpack packs will integrate with our build pipeline? Checking in compiled packs will cause merging separate patches harder
  • Is this for Wikibase only for now, or WikibaseLexeme too?
  • What is the Acceptance Criteria of this trailblazing?
    • what metrics we want to measure before/after?
    • after what time we should re-visit where we are, whether we achieved our goals, what we have achieved, and plan what to do next?
  • Will we need a security review needed for Webpack or other babel presets/polyfills?
  • How unit tests of those modules packed with Webpack will be run?