Page MenuHomePhabricator

Provide standard-compatible way to load multi-file packages (not plain concatenation)
Closed, ResolvedPublic

Description

Problem statement

When ResourceLoader was first released in 2011, the web was mostly still in a state where front-end JavaScript uses the global object as way to "connect the dots" between pieces of code. Whether it be module registration, plugin registration, public/static classes, private methods. They are mostly executed in either the global scope, or in a private closure.

As such, any code split up over multiple files could be safely concatenated. Private scopes could not cross the file-on-disk boundary because there was no good way to connect two in-Git-maintained files on the client-side without also making them completely public/global.

Since then, CommonJS and other module patterns have become popular, especially through the Node eco system.

As of 2016, we support require() and module.exports for entire modules, but ResourceLoader still concatenates all the files within a module. This has resulted in a confusing situation where, if you do decide to split up the files within a module, they effectively share the same exports object. This is non-standard and basically means use of module.exports is only useful if the ResourceLoader module consists of a single file (e.g. a simple module, a hard to maintain module, or a module maintained elsewhere that is included as external library).

Objectives

  • Get rid of non-standard behaviour of having a "shared exports object" between files. This was unintentionally introduced, is undocumented/unsupported, and will go away as part of this RFC without deprecation.
  • Allow one to implement a module with multiple files.
  • Be compatible with Node.js, RequireJS, and WebPack.

Proposal

This RFC proposes to allow modules in ResourceLoader that are capable of exporting multiple files to the client-side that can individually be accessed within the module scope in a standard-compatible way, e.g. through require('./foo.js'). This will allow modules to be composed of multiple files, and also makes those modules interoperable between other environments such as RequireJS, WebPack, and Node.js.

This makes it easy for modules written for MediaWiki to be published on npm, whilst also making it easy for MediaWiki developers to integrate external libraries (like we do already with Composer for PHP).

Note that the question of automatically downloading and managing such packages is orthogonal (T107561). Regardless of whether upstream code needs to be manually checked into Git, we still currently only support single-file modules. This RFC deals with that. After T107561 is resolved, the model introduced by this RFC (T133462) would allow those external modules to be registered and loaded by ResourceLoader without any modifications. Currently we are unable to check in modules in their package form, and instead have to find classic style "single-file" combined distributions of upstream libraries. While libraries targeted at the browser landscape do often still provide such distributions, newer/generic libraries published on npm do not.

Usage examples

Old model with separate files
Module registration
"module.name": {
  "styles": [ filePaths, .. ],
  "scripts": [ "foo.js", "bar.js" ]
}
foo.js
var other = require( 'other.module' );
mw.foo = other.make( 'foo' );
bar.js
mw.bar = mw.foo.make( 'bar' );
load.php
function (mw, $, require) {
  var other = require( 'other.module' );
  mw.foo = other.make( 'foo' );
  mw.bar = mw.foo.make( 'bar' );
}
Old model with a single module export
Module registration
"module.name": {
  "styles": [ filePaths, .. ],
  "scripts": [ "module.js" ]
}
module.js
var other = require( 'other.module' );
var foo = other.make( 'foo' );
module.exports = foo.make( 'bar' );
load.php
function (mw, $, require, module) {
  var other = require( 'other.module' );
  var foo = other.make( 'foo' );
  module.exports = foo.make( 'bar' );
}
Proposed model supporting multi-file package
Module registration
"module.name": {
  "styles": [ filePaths, .. ],
  "package": {
    "files": [ "foo.js" ],
    "main": "index.js"
  }
}
foo.js
var other = require( 'other.module' );
module.exports = other.make( 'foo' );
index.js
var foo = require( './foo.js' );
module.exports = foo.make( 'bar' );
load.php
{
  main: 'index.js',
  files: {
    'foo.js': function (mw, $, module, require) {
      var other = require( 'other.module' );
      module.exports = other.make( 'foo' );
    },
    'index.js': function (mw, $, module, require) {
      var foo = require( './foo' );
      module.exports = foo.make( 'bar' );
    }
  }
}

See also:

Details

Related Gerrit Patches:

Related Objects

Event Timeline

There are a very large number of changes, so older changes are hidden. Show Older Changes

With node 4 supporting ES6 modules, arrow functions etc we are seeing a growing number of node libraries using those & dropping 0.10 support. While CommonJS modules are here to stay, the roadmap for module loading should probably set include a solution for handling ES6 modules, even if that plan is to compile those down to ES5 CommonJS modules.

Krinkle assigned this task to Catrope.Sep 29 2016, 2:46 AM
Krinkle raised the priority of this task from Normal to High.
Krinkle lowered the priority of this task from High to Normal.
Krinkle moved this task from (unused) to Under discussion on the TechCom-RFC board.
Krinkle added a project: TechCom-Has-shepherd.
Krinkle moved this task from Backlog to Catrope on the TechCom-Has-shepherd board.

On IRC @Krinkle told me his ideas for this task, which I'll summarize here.

A module would be defined something like this in extensions.json:

 modules: {
    ....
    foo: {
        package: true,
        files: [
            'lib/foo/one.js',
            'lib/foo/two.js',
            'lib/foo/bar/three.js',
            'lib/foo/bar/four.js'
        ],
        main: 'lib/foo/main.js'
}

(Alternatively, we might allow discovery with something like dir: 'lib/foo' replacing the list of files.)

Server-side logic then generates virtual file names for each file, and load.php?modules=foo would output an object mapping virtual file names to closures:

mw.loader.implement( 'foo', {
    'one.js': function( module, require ) {
        // content of one.js
        // require is bound so that require( './two.js' ), require( './bar/three.js' ) and require( './bar/four.js' ) work
    },
    'two.js': function( module, require ) {
        // content of two.js
        // require is bound so that require( './one.js' ), require( './bar/three.js' ) and require( './bar/four.js' ) work
    },
    'bar/three.js': function( module, require ) {
        // content of bar/three.js
        // require is bound so that require( '../one.js' ), require( '../two.js' ) and require( './four.js' ) work
    },
    'bar/four.js': function ( module, require ) {
        // content of bar/four.js
        // require is bound so that require( '../one.js' ), require( '../two.js' ) and require( './three.js ' ) work
    'main.js': function ( module, require ) {
        // content of main.js
        // require is bound so that require( './one.js' ), require( './two.js' ), require( './bar/three.js' ) and require( './bar/four.js' ) work
        // When external code calls require( 'foo' ), this is the code that will be run, and whose module.exports will be returned
    }
}, /* CSS stuff etc */ );

When executing the module, only the main closure is executed, and it's passed a require() function that knows about the virtual files. The virtual files themselves aren't executed immediately, they are only executed if the main closure require()s them. When they are executed, they receive a require() function that knows about their virtual file path and the other virtual file paths, and is able to resolve ./ and ../.

None of this works in debug mode, and async calls to require() are already broken in debug mode. We would have to redesign or remove debug mode for this to work. One idea is to use sourceURL, but that has potential issues with localStorage evals and doesn't address CSS-related use cases for debug mode.

Volker_E added a subscriber: Volker_E.

@Jdlrobson What's Reading-Web's interest in this? I heard a rumor that it might be related to the offline / ServiceWorker / web app work, is that right or is there another reason?

We want this. We have been having trouble with dependency management via ResourceLoader, to the point that now in the Popups repository we are using webpack to manage these dependencies and generate a single JS file for inclusion with ResourceLoader. Reflecting aside from one solveable problem of merge conflicts due to shipping compiled assets in the repository we prefer this approach and are probably going to use it elsewhere. So anything that plays into that / removes the need for webpack would be highly useful.

We are also planning to share more JS with the iOS and Android apps, so are thinking a lot about packaging up libraries on npm so we can share them amongst ourselves. Right now our webpack approach means we can do this quite easily.

We want this. We have been having trouble with dependency management via ResourceLoader, to the point that now in the Popups repository we are using webpack to manage these dependencies and generate a single JS file for inclusion with ResourceLoader. Reflecting aside from one solveable problem of merge conflicts due to shipping compiled assets in the repository we prefer this approach and are probably going to use it elsewhere. So anything that plays into that / removes the need for webpack would be highly useful.
We are also planning to share more JS with the iOS and Android apps, so are thinking a lot about packaging up libraries on npm so we can share them amongst ourselves. Right now our webpack approach means we can do this quite easily.

Are you using Webpack primarily because of the module.exports / require() support, or are there other reasons why RL's dependency management doesn't work for you?

We documented our decision here: https://github.com/wikimedia/mediawiki-extensions-Popups/blob/master/doc/adr/0004-use-webpack.md

[..]

  • More reliable debug via source map support

[..]

@Jdlrobson Could you elaborate a little about how source maps are being used in the Popups extension?

Right now, when in development and debug=true, source maps are served for the popups bundle file, giving access to the original common.js modules in the source tree. So if you pop open the devtools you can see inspect and interact with the original files even if they where bundled into one. Example:

As you can see there, under Popups/resources/dist, is the bundled file, and under src in orange are the original files shown from the source maps.

Sadly in production right now they don't work since the servers haven't whitelisted .map files (https://ca.wikipedia.org/w/extensions/Popups/resources/dist/index.js.map). So the browser can't fetch them. We're planning to get that fixed at some point.

Krinkle updated the task description. (Show Details)Aug 17 2017, 3:37 AM

Sadly in production right now they don't work since the servers haven't whitelisted .map files (https://ca.wikipedia.org/w/extensions/Popups/resources/dist/index.js.map). So the browser can't fetch them.

While the development community at large has popularised the idea of using .map for the source map, this is merely an (unofficial) social convention. The file extension is not required for Source Maps to work. It is also not on any standards track for Mime, and is unlikely to be given the many existing programs using the same extension.

The format used by Source Maps is JSON and is always linked up by a url pointer in the source code. As such, it can have any arbitrary file extension (or no extension at all). I'd recommend trying the .json extension which is supported, standardised and whitelisted by MediaWiki in production.

Tgr added a subscriber: Tgr.Aug 19 2017, 6:00 PM
Restricted Application added a project: Performance-Team. · View Herald TranscriptNov 10 2017, 8:27 PM
Krinkle raised the priority of this task from Normal to High.Dec 21 2017, 11:24 PM
Krinkle moved this task from Backlog to Accepted Enhancement on the MediaWiki-ResourceLoader board.
Krinkle updated the task description. (Show Details)Jun 25 2018, 5:31 PM
phuedx added a subscriber: phuedx.Jun 27 2018, 6:31 PM
kchapman added a subscriber: kchapman.

TechCom is putting this on Last Call pending approval. Last Call will end on 2018-07-05

This RFC has been approved by TechCom

kchapman edited projects, added TechCom-RFC (TechCom-Approved); removed TechCom-RFC.
nray added a subscriber: nray.Jul 12 2018, 11:21 PM
Catrope updated the task description. (Show Details)Sep 19 2018, 2:39 AM
TheDJ added a subscriber: TheDJ.Sep 24 2018, 12:13 PM

Change 471370 had a related patch set uploaded (by Catrope; owner: Catrope):
[mediawiki/core@master] ResourceLoader: Add support for packageFiles

https://gerrit.wikimedia.org/r/471370

The proposed model of using require and module.exports concerns me given import/export is gaining traction. The goal posts are continuously moving. What is our migration path, to using import/exports in future?

Babel, with plugins, is also making it possible to import non-JS files. Do we have any plans for supporting that e.g. CSS imports? We are already importing json files in MobileFrontend and considering template/json importing.

I feel a roadmap would be helpful here as while I think it will be useful to have file importing for smaller extensions, it's always going to be non-standard and it's going to require investment to maintain and I suspect investing/keeping up with everything is not sustainable.

I had a nice chat with Roan earlier in the week.

While I think this is an improvement to ResourceLoader, I raised concerns about the roadmap and the never-ending list of wishes that we'll want to add on top of it - for instance, requiring packages from npm; allowing es6 transpiling; compatability.... the list continues! It doesn't seem sustainable for us to port all Node.js tooling into PHP. Knowing what we plan to add support for and why and what we don't plan to add support for would be useful.

I talked about the work reading web is doing and the RFCs we are planning to write - around headless unit tests and the mediawiki stub we are using for headless testing and the benefits that brings such as faster CI; tests from the command line; non-blocking failures from other extensions; and code coverage reports; and how it reduces Special:javaScript/qunit to serve as integration tests. I talked about how we're making using of webpack to handle the dependency trees and that we are already a few steps ahead of this task so even with this capability ResourceLoader will not be in a state that we can use just yet.

However, it seems that the work he's doing here, at the very least could be a good interim stage for applying some of the things we are doing in MobileFrontend to other extensions that are not using webpack. For instance, with ResourceLoader file loading of this kind, we could convert every ResourceLoader module into a file that exports modules for testing purposes and have ResourceLoader modules that are trivial to port to npm/and run headless with our node-qunit library. To me at least, that provides a strong motivation here for adding this support to ResourceLoader in order to make all of MediaWiki have command line qunit tests since we have a lot of legacy code.

Longer term however, I think that we need to come up with a roadmap for where we want to be with ResourceLoader and force ourselves to imagine some other solutions e.g. imagine a script that is run at deployment that generates a webpack config from all the extension.jsons of loaded extensions and generates JavaScript bundles for every ResourceLoader module. that requires no runtime processing. It would be useful to take some time to imagine different (maybe radically different!) futures of ResourceLoader to allow us to weigh pros and cons and work out where we need to be.

So in summary:

  1. Let's roadmap what the future of ResourceLoader is
  2. Let's ensure the output of this is compatible with importing/exporting from headless tests (without ResourceLoader).
TheDJ added a comment.Dec 6 2018, 12:28 PM

we could convert every ResourceLoader module into a file that exports modules for testing purposes and have ResourceLoader modules that are trivial to port to npm/and run headless with our node-qunit library

That would indeed be very nice !

@Jdlrobson We should also document the gap between webpack and resource loader (and keep it up to date). That helps give everyone a lot better insight into what we need in order to be able to ditch one versus the other.

Also, while i'm at it... if someone could document how I'm supposed to do lazy loading in webpack for MediaWiki. There is currently 0 documentation about best practices for webpack in MediaWiki. As a developer, without something like https://www.mediawiki.org/wiki/ResourceLoader/Developing_with_ResourceLoader, webpack doesn't exist for me.

Also, while i'm at it... if someone could document how I'm supposed to do lazy loading in webpack for MediaWiki.

@TheDJ we'll in the process of working this out as we speak (see T210210)

s a developer, without something like https://www.mediawiki.org/wiki/ResourceLoader/Developing_with_ResourceLoader, webpack doesn't exist for me.

Per your request I've started https://www.mediawiki.org/wiki/ResourceLoader/Developing_with_Webpack to start documented what we've been doing. Please open talk topics to help guide me with what needs to be covered. I've done the bare minimum for now :)

@Krinkle following on from https://phabricator.wikimedia.org/T133462#4841164 here's a clear example of how I'd like to see us use this:

https://gerrit.wikimedia.org/r/#/c/mediawiki/core/+/487168 WIP/POC: Headless Qunit tests (oh my!)

Change 471370 merged by jenkins-bot:
[mediawiki/core@master] ResourceLoader: Add support for packageFiles

https://gerrit.wikimedia.org/r/471370

Krinkle closed this task as Resolved.Feb 7 2019, 3:20 AM
Krinkle moved this task from In progress to Implemented on the TechCom-RFC (TechCom-Approved) board.
He7d3r added a subscriber: He7d3r.Jun 9 2019, 5:25 PM