Page MenuHomePhabricator

Introduce ResourceLoader debug mode v2
Closed, ResolvedPublic

Description

Motivation

As of 2015, debug mode serves makes load.php requests serially for each resource (style, scripts) of each module.

It works this way because when serving a raw JS file from disk, browsers execute the file immediately upon download, which means the only way to ensure code executes in the correct order, is to not request the next file until the previous one has completed.

This is is inefficient and slow, and makes debug mode unsuitable for large features such as VisualEditor (see blocked/parent tasks).

The other issue with this is that the execution scope is changed from local module scope, to global scope. This is the only reason that we have the convention to wrap each JS file in a closure (for files in legacy non-packageFiles modules). Because that way, it has module scope even in debug mode. For production mode the closure is redundant, given the load.php/implement() wrapper already has local scope. This module scope has been part of ResourceLoader since the very beginning in 2010, and is similar to what Node.js, RequireJS, Rollup etc. do nowadays.

Requirements
  • JavaScript code will execute with local scope in debug mode (same as production).
  • Modules that depend on each other will load in parallel in debug mode (same as production).
  • Modules with N JS files will require significantly fewer than N web requests to load in debug mode (close to or equal to production).
  • JavaScript code and CSS code will continue to be served without minification in debug mode, preserving all code comments and whitespace.

Proposal

debug=2 (and later by default for debug=true)

Create a fresh with a new debug mode that is much closer to production.

It will essentially just do two things at first:

  • Disable minification. This means all code comments and whitespace is preserved. It also means the development server won't need to perform as much minification, which helps in slow VM-based environments like MediaWiki-Vagrant.
  • Disable multi-module batching. This means each module will load in its own module=… request.

In practical terms, this means debug mode will behave for all modules similar to how it behaves for packageFiles-modules today already. In other words, the only thing we lose is raw file paths. So instead of an error being attributed to ve.ui.FooBar#onClick() on line 150 of ve/ui/Foo/FooBar.js, it would be attributed to ve.ui.FooBar#onClick() of modules=ext.visualEditor.

Depending on your workflow, this might not make much difference. E.g. in most text editors, a file can be quickly opened by typing cmd-P/ctrl-P, enter "ve ui FooBar" and the relevant file is there, etc.

Also, if you've been using production mode as-is for debugging (like me), then this will have been how you open files anyhow. The difference will be that you'd now get code formatting back without needing to use the in-browser "Prettify" button in DevTools, and get code comments as well within the browser.

Long term, we will get line numbers back with source maps (T47514). But in the interest of moving this forward one step at a time, I've changed this proposal from the original version to not include source maps initially.

debug=1

If some people prefer the old debug mode we have today (with its flaws) over the above proposal, I'd be open to maintaining that a little longer (at least until source maps are ready). Which you could opt-in, and possibly rename to debug=old after flipping the defaut. Let me know in the comments below if this is something you think you'd make use of.

Related Objects

Event Timeline

There are a very large number of changes, so older changes are hidden. Show Older Changes

Disable minification

It would be nice to have this functionality for projects using Webpack too. One solution is to use debugScripts (example in Popups).

Disable minification

It would be nice to have this functionality for projects using Webpack too. One solution is to use debugScripts (example in Popups).

If the original non-minified source is given to RL, then both the old and new debug mode will not minify it. That'll work out of the box.

What is it that you'd like RL to make work with Webpack?

What is it that you'd like RL to make work with Webpack?

Ship production Webpack-bundled assets by default (this is the current behavior). Ship different, development-only, Webpack-bundled assets when debug=1. These assets, for example, may have special debugging capabilities that could be determined at compile time and stripped from production sources. (Maybe not the best example, but here's a runtime test in Popups.)

If we use packagefiles feature in RL, none of those source files will be listed in devtools(and map to a source file in filesytem) even in debug mode. The current Vue integration without build step relies on packagefiles and not having a proper client side debugging support makes it hard for developers. Is this a known problem and is there any solution?

What is it that you'd like RL to make work with Webpack?

Ship production Webpack-bundled assets by default (this is the current behavior). Ship different, development-only, Webpack-bundled assets when debug=1. These assets, for example, may have special debugging capabilities that could be determined at compile time and stripped from production sources. (Maybe not the best example, but here's a runtime test in Popups.)

Ah okay. I guess I should clarify that debugScripts (to add debug-only files) and packageFiles's closure support (to replace a prod file) are not going anywhere. Do those cover your needs currently?

For packageFiles nothing is planned to change as part of this task. For legacy modules, when using debug=1, the unminified files (or debugScripts files) would now be concatenated and served from load.php instead of directly from /w/extensions/… like today (as requested by developers). And debug=old would behave the same as today even for legacy modules – if there is demand for that option.

I should clarify that debugScripts (to add debug-only files) and packageFiles's closure support (to replace a prod file) are not going anywhere. Do those cover your needs currently?

I think so! However, it would be very nice to avoid the need for custom entry points. It's possible to clumsily workaround but I think it would be a lot nicer if exclusively scripts OR debugScripts assets were shipped instead of both. In general, I want a completely different version of the same code (either a debug or production build) not a production build with debug sprinkled on top. The latter is the current behavior. I would rather intentionally require production sources from debug than intentionally exclude them.

For packageFiles nothing is planned to change as part of this task.

I think packageFiles are less applicable when using an external bundler. A debug-only target seems like it could be useful to add flexibility. I don't have any need for it at the moment but wanted to mention it.

Edit: similar PHP workaround for debugScripts:

	'vue' => [
		'packageFiles' => [
			'resources/src/vue/index.js',
			'resources/src/vue/i18n.js',
			[
				'name' => 'resources/lib/vue/vue.js',
				'callback' => function ( ResourceLoaderContext $context, Config $config ) {
					// Use the development version if development mode is enabled, or if we're in debug mode
					$file = $config->get( 'VueDevelopmentMode' ) || $context->getDebug() ?
						'resources/lib/vue/vue.common.dev.js' :
						'resources/lib/vue/vue.common.prod.js';
					return new ResourceLoaderFilePath( $file );
				}
			],

		],
		'targets' => [ 'desktop', 'mobile' ],
	],

^It'd be awesome to be able to keep configuration data as JSON files strictly in skins and extensions.

Change 597912 had a related patch set uploaded (by Krinkle; owner: Krinkle):
[mediawiki/core@master] resourceloader: Add internal handling for debug=2

https://gerrit.wikimedia.org/r/597912

Change 597912 merged by jenkins-bot:
[mediawiki/core@master] resourceloader: Add internal handling for debug=2

https://gerrit.wikimedia.org/r/597912

Change 715295 had a related patch set uploaded (by Krinkle; author: Krinkle):

[mediawiki/core@master] resourceloader: Skip version hash calculation in debug mode

https://gerrit.wikimedia.org/r/715295

Change 715296 had a related patch set uploaded (by Krinkle; author: Krinkle):

[mediawiki/core@master] resourceloader: Always split requests in debug mode

https://gerrit.wikimedia.org/r/715296

Change 721287 had a related patch set uploaded (by Krinkle; author: Kosta Harlan):

[mediawiki/core@master] resourceloader: Add setting for controlling minification

https://gerrit.wikimedia.org/r/721287

Krinkle renamed this task from ResourceLoader: Redesign how debug mode operates to Introduce ResourceLoader debug mode v2.Sep 15 2021, 5:34 PM

Change 715295 merged by jenkins-bot:

[mediawiki/core@master] resourceloader: Skip version hash calculation in debug mode

https://gerrit.wikimedia.org/r/715295

Change 721287 had a related patch set uploaded (by Kosta Harlan; author: Kosta Harlan):

[mediawiki/core@master] ResourceLoader: Add setting for controlling minification

https://gerrit.wikimedia.org/r/721287

Change 723620 had a related patch set uploaded (by Krinkle; author: Krinkle):

[mediawiki/core@master] resourceloader: Preserve new 'debug' param in getScriptURLsForDebug()

https://gerrit.wikimedia.org/r/723620

Change 723620 merged by jenkins-bot:

[mediawiki/core@master] resourceloader: Preserve new 'debug' param in getScriptURLsForDebug()

https://gerrit.wikimedia.org/r/723620

Change 744072 had a related patch set uploaded (by Krinkle; author: Krinkle):

[mediawiki/core@master] resourceloader: Allow debug=2 on JavaScriptTest, and misc whitespace

https://gerrit.wikimedia.org/r/744072

Change 744073 had a related patch set uploaded (by Krinkle; author: Krinkle):

[mediawiki/core@master] resourceloader: Implement debug=2 request splitting

https://gerrit.wikimedia.org/r/744073

Change 721287 abandoned by Kosta Harlan:

[mediawiki/core@master] ResourceLoader: Add setting for controlling minification

Reason:

https://gerrit.wikimedia.org/r/744073

https://gerrit.wikimedia.org/r/721287

Change 744072 merged by jenkins-bot:

[mediawiki/core@master] resourceloader: Allow debug=2 on JavaScriptTest, and misc whitespace

https://gerrit.wikimedia.org/r/744072

Change 715296 abandoned by Krinkle:

[mediawiki/core@master] resourceloader: Always split requests in debug mode

Reason:

Obsoleted by https://gerrit.wikimedia.org/r/744073

https://gerrit.wikimedia.org/r/715296

Change 744073 merged by jenkins-bot:

[mediawiki/core@master] resourceloader: Implement debug=2 request splitting

https://gerrit.wikimedia.org/r/744073

Anecdotally, I'm profiling debug=2 per T85805 on my local wiki (using mediawiki-docker-dev), and noticing something funny. Despite taking fewer requests and allowing better use of concurrency, debug=2 is slower than debug=1 on a simple dev wiki.

Main_Page?debug=1Main_Page?debug=2
Page load time: 1 second.Page load time: 4 seconds.
a.png (1×2 px, 387 KB)
b.png (1×2 px, 535 KB)
Waterfall: Long and inefficient trickle.Waterfall: All start concurrently. With requested artificially held back by the browser, per HTTP 1.1 conventions in browsers to use 6 connections per hostname and wait for responses to reuse them.

In debug=1, 44 JS files load in order one at a time, each waiting for the previous to be requested, then downloaded, then parsed, then executed, and then the next file. No concurrency or overlap in any of the stages, wasting both CPU idling and network idling other workers remain idle. (This is by design as under debug=1 we relied on the global scope and thus can't delay execution or otherwise control the order without making debugging itself dififcult and unpleasant).

In debug=2, 30 JS files load in order, mostly in parallel. No CPU time no longer wasted as much as we're offering to do multiple of each and letting the browser figure it out optimally on its own. Yet, it takes over 4 seconds in total. Debug v1 served 43 of the 44 files in 3ms time each, read straight off disk by Apache. But, debug v2 pulls them through load.php where MW Setup.php is taking a good 200ms for each response.

Is ResourceLoader slow? Not exactly. An excimer profile (P18786) indicates that a majority of time (about 120ms of 200ms) is spent in AutoLoader::autoload, mostly via Hooks::runner, which as we know from T228895 is first triggered by WebRequest::getIP.

Tentatively blocking adoption of debug=2 until T274041 is resolved (before we adopt it in CI for QUnit, and as default for debug=true).

Thanks for working on this!

Long term, we will get line numbers back with source maps (T47514). But in the interest of moving this forward one step at a time, I've changed this proposal from the original version to not include source maps initially.

Would it be hard to include that? The inability to reliably place breakpoints and other disruption caused by Chrome's de-minification feature (e.g. useless stack traces, useless source tab titles, having to click the pretty-print button all the time) are one of the main pain points when debugging ResourceLoader module code in my opinion. Source maps for minified code is a hard problem. Source maps for merely concatenated code seems fairly easy.

Change 777427 had a related patch set uploaded (by Krinkle; author: Krinkle):

[mediawiki/extensions/Gadgets@master] GadgetRepo: Fix missing purging on delete and simplify hook handling

https://gerrit.wikimedia.org/r/777427

I'm working on a faster HookRunner for load.php (and possibly elsewhere opt-in) which is tracked at T274041: Reduce performance impact of HookRunner.php loading 500+ interfaces.

Meanwhile, I'm noticing that on a fresh install, half the time is spent in an uncachable MediaWikiGadgetsDefinitionRepo::loadGadgets

100.00% 357.877      1 - main()
 54.51% 318.291     22 - MediaWiki\MediaWikiServices::getService
..
 53.76% 313.897      1 - wfLoadMain
..
 39.28% 229.324      1 - MediaWiki\MediaWikiServices::getResourceLoader
 38.68% 225.861     17 - Psr\Log\AbstractLogger::debug
 38.64% 225.581     15 - MediaWiki\Logger\LegacyLogger::log
..
 35.33% 206.286      5 - MediaWiki\HookContainer\HookContainer::run
 35.22% 205.645      1 - MediaWiki\ResourceLoader\HookRunner::onResourceLoaderRegisterModules
..
 35.12% 205.072      1 - MediaWiki\Extension\Gadgets\Hooks::registerModules
 35.09% 204.904      1 - MediaWiki\Extension\Gadgets\MediaWikiGadgetsDefinitionRepo::getGadgetIds
 35.09% 204.890      1 - MediaWiki\Extension\Gadgets\MediaWikiGadgetsDefinitionRepo::loadGadgets
 34.50% 201.463      1 - WANObjectCache::getWithSetCallback
 34.50% 201.446      1 - WANObjectCache::fetchOrRegenerate
 20.52% 179.795      1 - MediaWiki\Extension\Gadgets\MediaWikiGadgetsDefinitionRepo::fetchStructuredList

Change 777478 had a related patch set uploaded (by Krinkle; author: Krinkle):

[mediawiki/core@master] WebRequest: Optimise getIP

https://gerrit.wikimedia.org/r/777478

Another highlight may be WebRequest::getIP.

Production for comparison, thought this is mostly deferred and inevitable setup cost for HookRunner

Change 777482 had a related patch set uploaded (by Krinkle; author: Krinkle):

[mediawiki/core@master] debug: Fix $wgDebugRawPage to work with PSR-3 debug logging

https://gerrit.wikimedia.org/r/777482

Change 777427 merged by jenkins-bot:

[mediawiki/extensions/Gadgets@master] GadgetRepo: Fix missing purging on delete and simplify hook handling

https://gerrit.wikimedia.org/r/777427

Change 777478 merged by jenkins-bot:

[mediawiki/core@master] WebRequest: Micro-optimise getIP

https://gerrit.wikimedia.org/r/777478

Change 777482 merged by jenkins-bot:

[mediawiki/core@master] debug: Fix $wgDebugRawPage to work with PSR-3 debug logging

https://gerrit.wikimedia.org/r/777482

Change 890909 had a related patch set uploaded (by Krinkle; author: Krinkle):

[mediawiki/core@master] ResourceLoader: Fix mw.inspect tests to work on debug=2

https://gerrit.wikimedia.org/r/890909

Change 890910 had a related patch set uploaded (by Krinkle; author: Krinkle):

[mediawiki/core@master] qunit: Enable debug=2 by default for SpecialJavaScriptTest

https://gerrit.wikimedia.org/r/890910

Change 890909 merged by jenkins-bot:

[mediawiki/core@master] ResourceLoader: Fix mw.inspect tests to work on debug=2

https://gerrit.wikimedia.org/r/890909

Change 890910 merged by jenkins-bot:

[mediawiki/core@master] qunit: Enable debug=2 by default for SpecialJavaScriptTest

https://gerrit.wikimedia.org/r/890910