Page MenuHomePhabricator

ResourceLoader: Implement support for Source Maps
Open, NormalPublic

Description

Source maps are a technique for mapping combined and minified JavaScript back to the original files. This could be very useful for debugging ResourceLoader in production mode. This can be necessary since behavior is different from debug mode (in ways other than minification).

This is supported in the Chrome, Firefox 50 and IE 11 debuggers and Closure compiler, and some other stacks have code to generate the maps.

The spec is at https://docs.google.com/document/d/1U1RGAehQwRypUTovF1KRlpiOFze0b-_2gc6fAH0KY0k/edit?hl=en_US&pli=1# and a good overview is at http://www.html5rocks.com/en/tutorials/developertools/sourcemaps/ .

The minified file points to the source map with a line like:

//# sourceMappingURL=/path/to/file.js.map

or a header like:

X-SourceMap: /path/to/file.js.map

If we use a dynamic URL, that should allow doing it in production. It would build the source maps on demand (just like the minification) for people that have them enabled (and are debugging), without slowing the site for anyone else.

https://developer.mozilla.org/en-US/docs/Web/HTTP/Headers/SourceMap

Details

Reference
bz45514

Event Timeline

bzimport raised the priority of this task from to Normal.Nov 22 2014, 1:32 AM
bzimport set Reference to bz45514.

To clarify, I mean something like:

//@ sourceMappingURL=/sourcemap.php?modules=...

for the dynamic URL, with the response built and cached on demand.

Because of version numbers and caching, the link should probably include a hash instead of the module name (or both).

ResourceLoader already has this infrastructure in place. When building packages, we only rebuild if the hash of the module contents are different.

That hash is also output for debugging purposes with the cache key at the end of the package.

We can use that same hash to tie a packaged response to the sourcemap, which we'll need to generate while packaging, not on-demand from the sourcemap request.

Could we also use the hash for identifying versions in regular requests, rather than the version parameter? That could potentially solve the problem of out of date HTML getting new JS+CSS.

(In reply to Matthew Flaschen from comment #3)

Could we also use the hash for identifying versions in regular requests,
rather than the version parameter? That could potentially solve the problem
of out of date HTML getting new JS+CSS.

No.

Firstly, the version number we have now would be more than enough for that purpose. The reason cached HTML is getting newer resources is because we explicitly *don't* embed version numbers or hashes of any kind in the HTML.

That is not a bug but a feature (or bug fix) by design in the core of how ResourceLoader did what old $wgStyleVersion did wrong.

Krinkle added a comment.EditedMar 8 2014, 12:07 AM

As for implementation, building on comment #2, the bottom of load.php javascript request would contain something like:

/* cache key: enwiki:resourceloader:3957c1d7aa */
//# sourceMappingURL=load.php?action=sourcemap&id=3957c1d7aa

Aside from incorporating a source map generator in PHP in the first place, doing this would be a pretty easy first step for minified -> unminifed mapping.

However it's slightly more elaborate to also maintain where the modules came from to their respective individual non-concatenated files because when we're in the minifier, it's for the entire request as a whole. We'll have to maintain that state somehow and accumulate that context as we go on.

Krinkle lowered the priority of this task from Normal to Low.Jan 5 2015, 3:00 PM
Krinkle updated the task description. (Show Details)
Krinkle set Security to None.
Tgr added a subscriber: Tgr.Feb 26 2015, 7:46 PM
Krinkle removed Krinkle as the assignee of this task.Aug 6 2015, 6:10 AM
jeblad removed a subscriber: jeblad.Aug 25 2017, 10:11 PM
Seb35 added a subscriber: Seb35.Sep 16 2017, 4:55 PM

Some PHP libraries:

Given MediaWiki uses a specific minifier JavaScriptMinifier, it should be added there some tracking of the column/row of the source and generated files, but it shouldn’t be too difficult given the code is quite clear.

The next step is to compute the source map and mainly the base64-VLQ; for this, either a library can be used, either it can be created by hand, but the base64-VLQ is itself a bit complicated, and the positions of the segments are relative to the previous hence some caution must be taken.

The integration into the ResourceLoader shouldn’t be too complicated: each module is minified separately in ResourceLoader::makeModuleResponse(), and an additional flag could be added to compute the Source Map and to collect the origin file names.

A point to be verified is that many modules (=files) are grouped together, each in a mw.loader.implement section, I guess it should be used an "index map" (see spec page 5); I hope this is supported by browsers, it seems it is not widely used.

Beyond Source Map for JavaScript, it could be considered for CSS. For this it should be implement in CSSMin in the same way as JavaScriptMinifier. It can be considered also for LESS, and in this case two maps should be computed one after the other (LESS transpiler then CSSMin). For CSS there is also the difficulty that it can be embedded as a JavaScript string in mw.loader.implement (I have no idea if Source Map can/should be used in this case).

For the interface I thought about something like /load.php?debug=sourcemap&lang=fr&modules=ext.uls.common,init,interface&skin=vector&version=05ghpbi to keep the same stateless infrastructure. The downside is that the comment //# in the main file will be longer than in Krinkle’s proposition.

Seb35 updated the task description. (Show Details)Sep 16 2017, 4:57 PM
Krinkle updated the task description. (Show Details)Sep 16 2017, 10:03 PM
Restricted Application added a project: Performance-Team. · View Herald TranscriptFeb 24 2018, 12:07 AM
Krinkle raised the priority of this task from Low to Normal.Aug 18 2018, 6:20 AM
Krinkle moved this task from Backlog to Accepted: Enhancement on the MediaWiki-ResourceLoader board.