The lack of tooling or support for tooling has been causing problems in complicated code bases like the codebase for our mobile site, so we carved out a proposal to create a bridge from our existing codebase to a more modern one using Webpack. I'll talk about what we did and why.

The majority of Wikipedia's front-end assets are served by a system called ResourceLoader, that has been part of the MediaWiki software since 2010. Of particular interest to us is how it packs JavaScript assets, by concatenating and compressing them. This capability predates Webpack (2012) and similar tools so this should not be considered a case of Not invented here.

This tooling has allowed developers to build front-end code without tooling. Back in 2010, this was actually the norm. If you look at similar projects which have been around for as long as ours in the open source ecosystem, you'll find Makefiles concatenating JavaScript files - artifacts of this era.

As the JavaScript ecosystem has flourished, it's becoming impossible to build JavaScript without some reliance on tools. Needless to say most front-end developers are now accustomed to working with tooling. Being unable to use tooling to build our JavaScript has arguably handicapped us a little, especially as we turn our attention to more complicated ambitious projects such as the page previews feature.

Unlike many JavaScript module systems, the ResourceLoader system we use focuses on the collection and delivery of various loosely coupled assets which it discovers from a manifest rather than a file. Currently, when writing JavaScript we need to define a module inside a special file called extension.json which is then interpreted in PHP and sent to the user which looks like so:

"mobile.toc": {
        // define which environments to run this module
"targets": [
                "mobile",
                "desktop"
        ],
        // manifest of dependencies (think require/import)
"dependencies": [
                "mobile.startup",
                "mobile.toc.images"
        ],
// script files in order they need to be concatenated
        "scripts": [
                "resources/mobile.toc/TableOfContents.js"
        ],
        // styles that should be loaded via JavaScript
"styles": [
                "resources/mobile.toc/toc.less"
        ],
        // manifest of templates to load to support JavaScript
"templates": {
                "toc.hogan": "resources/mobile.toc/toc.hogan",
                "heading.hogan":  "resources/mobile.toc/tocHeading.hogan"
        },
        // message keys needed to load to support internationalization
"messages": [
                "toc"
        ]
},

Without a Node.js module system, our developers have had to work with a client-side library similar to Require.js and manual editing of what's essentially a manifest to discover JavaScript whilst keeping a mental picture of how it all fits together and the code is split. We had to manage the JavaScript dependency trees ourselves. As my work colleague, Stephen put so nicely:

we essentially had to fill out paperwork to create a file" and now "adding a new file is as easy as right click new file".

Using Webpack to handle our JavaScript rather than our own in-house ResourceLoader has achieved several things for us:

• Webpack manages complicated dependency trees for us which avoids loading errors (e.g. code loading in the wrong order)
• gives us more control over public and private interfaces to community gadgets
• allows us to expose interfaces for testing
• encourages separation of logic into reusable modules (files) without any mental strain
• allows delegation of problems such as code splitting to tooling rather than the human mind

Making code easier to work with

Previously, adding or even renaming any source file to our repository required not only creating the file but registering it by listing it in an array in a JSON file (see developer notes at the end of this article for more). We had 86 JavaScript files which we reduced to 19 files built via Webpack from 101 source files. The increase in source files reflects the teams ability to embrace Webpack and separate code responsibilities more effortlessly.

Now that we make use of Webpack, we only have to require it via a require statement. This might seem basic stuff, but it has made a big difference.

Similarly, we've not had to worry about polluting the global namespace. Previously all our files were wrapped in an IIFE but now we've been able to remove that wrapper and also the indenting associated with it.

A familiar stack

While we didn't measure it, and it could just because we hire awesome people, we've all noticed that our new hire got up and running with our code much quicker than previous hires.

We're using tooling that is well supported and has utilized new tooling to help us write better code. We're making use of bundlesize to track the size of our JavaScript assets in the code itself (and preventing unexpected increases with our continuous integration stack). We're also making use of nyc to track code coverage (see below).

Faster unit tests

Previously, our unit tests couldn't be run cheaply on the command line in Node.js. Any npm script wanting to run them would have to boot up a browser e.g. Phantomjs and have a working MediaWiki instance. This was slow, manual and error-prone as tests from other unrelated projects could break our own tests. It was pseudo-integration testing than unit testing. Now, with a few minor changes (mock libraries that emulate "MediaWiki"), we can run these tests from the command line in headless mode using the qunit node library. We use Webpack to build a file that can be run in the old method, to avoid confusing developers in other teams who are used to running tests this way. During our refactor, 44 QUnit tests were slowly migrated to run in Node.js.

This is obviously much faster as it doesn't require a MediaWiki install and doesn't require booting up a browser. As a result, there is more motivation for engineers to write tests in the first place, in fact we've already added 15 new test files.

Code coverage

Previously, our bespoke tooling made testing coverage very tricky and manual. As a result we didn't measure. Now that we are using Webpack and headless Node.js tests, we are able to work this out. After porting a file to the new module loading system, we made sure to document the code coverage of the file. It has shown us that roughly 50% of the JavaScript code we run is covered by tests. More alarmingly, 45 of our 81 files had 0% coverage. In particular it became clear that we were avoiding writing tests where it meant exposing private interfaces on a global JavaScript variable.

As we refactor with our remaining project time we aim closer to 100% test coverage, now that our tooling makes it easier to write tests and we are now able to enforce coverage in the repository via the nyc library meaning that code coverage can only get better.

More future-proof code

While we've been forced to look at the code, we've been noticing ways to improve it and prepare it for a modern future. We've been replacing calls to jQuery with calls to a wrapper for jQuery, meaning it's becoming clearer about what we use jQuery for, and when and how we might not need it. By keeping the definition of our project around problems rather than solutions, it's been easy to justify and prioritize this work.

Similarly, we've been migrating to use ES5 functions where possible instead of jQuery.

While we're not removing jQuery from our stack just yet, we've found inspiration in other efforts to do this such as Github to at least make this a real possibility.

Versioning

Our mobile front-end relies heavily on the Hogan template library. Previously, any vendor JavaScript had to copy and pasted into the repository itself. When we started we didn't actually know what version of Hogan we were using and had to diff it with several production versions! Now that we use Webpack, we can pull Hogan directly from the npm repository, so we know exactly what we are shipping and can upgrade easily if necessary. We are exploring leaning more on our tooling with ideas to use transpiling and include template source code in JavaScript files.

Webpack didn't solve everything

While Webpack has helped us organize our JavaScript files better, it doesn't seem to solve all our problems (yet). For example. since Wikipedia supports over 200 languages, we haven't found a way Webpack can ship message strings in the scaleable way that ResourceLoader does. I'm excited about the prospect of identifying these problems and filling in those blanks where necessary, but right now we have a nice balance of the best of ResourceLoader and Webpack in our codebase.