Page MenuHomePhabricator

Start using webpack for dependency management and bundling
Open, Needs TriagePublic

Description

Motivation
Our "dependency management" (more accurately, requiring js files/modules within other ones) of javascript files/modules are being currently handled through the Resource Loader (RL).
Resource Loader dependency management feature was not designed to also host this kind of use-case.

Using javascripts's require/import solution is more appropriate for that use-case.
Using a packing solution on top of that is also appropriate to reduce the amount of http traffic overhead if these files were loaded asynchronously (which is still not supported by all browsers to this date https://caniuse.com/#feat=es6-module-dynamic-import) as well as

Problem

Other problems, partly related

  • Frontend code in Wikibase Repo is pretty modular which is good but we are using RL for dependency management and class loading resulting to have 300 RL modules with a very complex dependency relationship that causes TTFB (time to first byte) to be high and pretty big network overhead (e.g. T203696: Drastically reduce the number of ResourceLoader modules that WikibaseClient registers: "It looks like Wikibase is trying to use ResourceLoader as a class loader(!) which may be difficult to quickly re-architect, but any reduction would be worthwhile.").

Suggested Solution
Try usings Webpack to bundle our javascript modules in single files.
Webpack is chosen for the following reasons:

Expected Impact

  • On dev productivity: can be big, allowing us to use modern javascript through Babel
  • On performance (secondary goal here): Webpack is not the only mean to improve performance.
    • Every module removed from lib or client will have ~30GB/day reduction on network. We planning to drop probably around 10-20 of them
    • Every module removed from view or repo will have ~55 MB/day (wikidata) + 90 MB/day (commons) reduction on network. We are planning to drop probably around 100-200 of them.
    • TTFB is hard to measure. Can be measured afterwards.
    • The network reduction due to using webpack instead of RL manification is hard to measure. Can be measured afterwards.

Estimated Trailblazing length
couple of weeks (2-3 weeks for 100-150% FTE)

Event Timeline

Change 524515 had a related patch set uploaded (by Ladsgroup; owner: Ladsgroup):
[mediawiki/extensions/WikibaseLexeme@master] Start using webpack

https://gerrit.wikimedia.org/r/524515

alaa_wmde moved this task from Backlog to Todo: Focus on the User-Alaa board.
alaa_wmde updated the task description. (Show Details)Jul 24 2019, 9:43 PM

above patch was meant to be as a proof of concept

As of 7 May 2018 in production, Wikibase's 248 RL modules result in 14,453 bytes (5,210 bytes after gzip) being shipped on every page view of every Wikipedia for every reader. That's 6.4TiB a day of extra load on readers.

Just a clarification, this is for Client that is being tackled in another way, client will also enjoy some of the benefits of webpack but the main benefits will be towards performance of Wikidata's frontend by squeezing more than 300 modules we register in repo to maybe at most 100.

@Ladsgroup can you rephrase the Problem better, perhaps adding more details? I have a feeling the current one isn't really reflecting the actual problem behind our motivation here

alaa_wmde updated the task description. (Show Details)Jul 30 2019, 2:14 PM
Ladsgroup updated the task description. (Show Details)Jul 31 2019, 9:45 AM
Ladsgroup updated the task description. (Show Details)Jul 31 2019, 9:53 AM
Ladsgroup updated the task description. (Show Details)

@Ladsgroup does the MobileFrontend skin use minification in webpack, or leave it down the RL?
My main though here is it would be nice to not have minified things with debug=true?

Every module removed from lib or client will have ~30GB/day reduction on network. We planning to drop probably around 10-20 of them
Every module removed from view or repo will have ~55 MB/day (wikidata) + 90 MB/day (commons) reduction on network. We are planning to drop probably around 100-200 of them.

What impact would are the clients predicted to see bandwidth / response time wise?
Or are these the bits that are hard to predict?

What is the deployment / development workflow going to look like here?
building the JS and commiting it to wikibase.git for review? I guess that is what is done with T195475 ?

I dont have any major concerns with any of this really.

@Ladsgroup does the MobileFrontend skin use minification in webpack, or leave it down the RL?

Webpack does it. But also it does a better job in minfication like tree-shaking, and/or other configurable magic like renaming variables, etc. RL minfication is pretty inefficient (but fast so it can respond in timely manner). T49437: Consider a pipeline for enhanced minification (e.g. support UglifyJS) talks more about this, it uses uglifyJS but the same principle can be applied everywhere. In short, it can reduce size of our bundles up to 10%.

My main though here is it would be nice to not have minified things with debug=true?

Yes, that's the biggest caveat of webpack IMO but most debuggers have prettifies you can use, so it's not that bad.

What impact would are the clients predicted to see bandwidth / response time wise?
Or are these the bits that are hard to predict?

We already saw them in real-life, dropped 137 module and T203696#5387672 happened, It's too noisy to give an exact value but you can assume dropping 137 module, increased speed of pages in desktop with latency of 100ms by 4%, that adds up in case of clients. For repos, not that much because our bottleneck is not module names but actually the whole modules are pretty big and a better minifier might give a hand. (There are other ways to improve things like lazy loading but that's definitely outside of scope of this work). Dropping name of modules might give a small boost to commons though.

What is the deployment / development workflow going to look like here?
building the JS and commiting it to wikibase.git for review? I guess that is what is done with T195475 ?

Yes, basically before committing a patch, you run npm build, and it builds for you and you commit both codes (here's an example).During development you run "npm start" and it listens to your files and if they change, it rebuilds the build file(s).

This is the RFC that I mentioned might be of interest when looking into this: T133462.

Questions from the bonfire that we might want to answer/consider as part of the trailblazing

  • How build process of webpack packs will integrate with our build pipeline? Checking in compiled packs will cause merging separate patches harder
  • Is this for Wikibase only for now, or WikibaseLexeme too?
  • What is the Acceptance Criteria of this trailblazing?
    • what metrics we want to measure before/after?
    • after what time we should re-visit where we are, whether we achieved our goals, what we have achieved, and plan what to do next?
  • Will we need a security review needed for Webpack or other babel presets/polyfills?
  • How unit tests of those modules packed with Webpack will be run?
  • How build process of webpack packs will integrate with our build pipeline? Checking in compiled packs will cause merging separate patches harder

Yeah, that's a drawback

  • Is this for Wikibase only for now, or WikibaseLexeme too?

I would recommend WikibaseLexeme first. It's more self-contained but doing both should not be super hard.

  • What is the Acceptance Criteria of this trailblazing?

Start using webpack in production and migrate considerable number of scripts to use it sounds good to me.

  • what metrics we want to measure before/after?

Given that uglifyJS has better compression rate. We should start measuring size of bundles we send to users + the reduction in size of startup module in client and repo

    • after what time we should re-visit where we are, whether we achieved our goals, what we have achieved, and plan what to do next?
  • Will we need a security review needed for Webpack or other babel presets/polyfills?

All of steps of this trailblaze is aiming to do is already done on other products, like popups, mobile frontend. I don't see any difference.

  • How unit tests of those modules packed with Webpack will be run?

https://www.mediawiki.org/wiki/User:Jdlrobson/Developing_with_Webpack_and_ResourceLoader + You can take a look at the code in mobile frontend.

  • How build process of webpack packs will integrate with our build pipeline? Checking in compiled packs will cause merging separate patches harder

Yeah, that's a drawback

I think this is the main thing that we want to think about.

  1. How long will the build step take for developers? (I hope not too long)
  2. Can we have CI to make sure that people remember to rebuild when changing JS that needs to be built?

Point 1 I guess will be one of the things that we figure out once we get there.
Point 2 we can probably investigate a little before we start.

  • How build process of webpack packs will integrate with our build pipeline? Checking in compiled packs will cause merging separate patches harder

Yeah, that's a drawback

I think this is the main thing that we want to think about.

  1. How long will the build step take for developers? (I hope not too long)

It's not. I did it for MobileFrontend and my POC

  1. Can we have CI to make sure that people remember to rebuild when changing JS that needs to be built?

@Jakob_WMDE mentioned there's an interesting way to do it that's already being done in termbox and the bridge. The CI rebuilds the dist files and checks if git status fails. This is important to also prevent attacks that a person puts an innocent change in gerrit but a backdoor in the dist file.

nray added a subscriber: nray.Aug 27 2019, 8:04 PM
NOTE: good concern raised during Frontend Guild Session 1: we have to make sure we do not invest too much time re-writing source or test code of javascript modules, as part of this trailblazing, when they are being replaced by other hikes. Example is termbox hike has replaced old termbox with old termbox, or there would be no point of rewriting old termbox for the sake of making it work when we transition to webpack, as it may be dropped by the time we start/finish the trailblazing.
alaa_wmde moved this task from Todo: Focus to Done on the User-Alaa board.Sep 5 2019, 1:55 PM
WMDE-leszek updated the task description. (Show Details)Sep 9 2019, 8:17 AM
WMDE-leszek updated the task description. (Show Details)Sep 9 2019, 8:50 AM
Jakob_WMDE updated the task description. (Show Details)Sep 9 2019, 10:12 AM
WMDE-leszek updated the task description. (Show Details)Sep 10 2019, 6:19 AM

Using a packing solution on top of that is also appropriate to reduce the amount of http traffic overhead if we load these files asynchronously (which is still not supported by all browsers to this date https://caniuse.com/#feat=es6-module-dynamic-import) as well as

this sentence in the description seems somehow cut off (as well as WHAT?). @Ladsgroup: any chance you'd remember what additional things were meant to be listed here?

WMDE-leszek updated the task description. (Show Details)Sep 10 2019, 6:21 AM
WMDE-leszek updated the task description. (Show Details)Sep 10 2019, 7:05 AM