Page MenuHomePhabricator

Allow Wikimedia-Minify to skip minification of already-minified files
Closed, DeclinedPublicFeature

Description

This is just an idea, if Wikimedia-Minify already this does or it's not worth anyone's time because there's not enough to gain I apologize in advance.

I have no personal need for this, it's just an idea.

Feature summary (what you would like to be able to do and where):
Allow Wikimedia-Minify to skip minification of already-minified files.

I imagine this could be realized by either having Wikimedia-Minify autodetect this (perhaps by looking for unusually long lines?) or some indicator in the minified file or its filename.

Use case(s) (list the steps that you performed to discover that problem, and describe the actual underlying problem which you want to solve. Do not describe only a solution):
Some JS files are already minified. For example Vue and Pako. (T346075) This is also true for some gadgets, for example https://en.wikipedia.org/wiki/MediaWiki:Gadget-XFDcloser-core.js.

I assume Wikimedia-Minify will still spend some CPU-cycles trying to compress this further. I'd expect it to fail to compress these files any further. So some performance benefit may be possible by skipping Wikimedia-Minify here, or only removing comments at the top+bottom.

Benefits (why should this be implemented?):
Saving a few CPU cycles on the Wikimedia servers. I have no idea how "expensive" Wikimedia-Minify is, so it may or may not be worth the hassle. It's just an idea.

I actually skimmed over https://gerrit.wikimedia.org/r/plugins/gitiles/mediawiki/libs/Minify/+/refs/heads/master/src/JavaScriptMinifier.php and nothing stood out that could skip already-minified files, but I only skimmed over it so I could have missed it.

Event Timeline

Restricted Application added a subscriber: Aklapper. · View Herald Transcript
Krinkle subscribed.

Thanks for thinking about this and proposing the idea!

We actually have mechanism like this in MediaWiki. However, we consider it the responsibility of the caller, so this mechanism is actually in MediaWiki-ResourceLoader rather than the Wikimedia-Minify library. To learn how thi works, look for ResourceLoader::FILTER_NOMIN and references to it in Codesearch. Basically, if a certain token is found at the start or end of a script, we skip the minifier.

The reason we added this, though, might surprise you. We added it to:

  • prevent pollution of the minifier cache in case of variable or personalised content. An example can be found in the form of ResourceLoader\UserOptionsModule, which generates a new set of CSRF tokens on every page view based on an HMAC-embedded timestamp. This is nowadays triggered via 'cache' => false passed to ResourceLoader::filter(), which in turn is set automatically based on shouldEmbedModule() and getGroup()=private.
  • prevent corruption of unknown syntax. To experiment with new ECMAScript versions or non-standard flavours, extension can temporarily opt-out of the minifier.

Benefits (why should this be implemented?):
Saving a few CPU cycles on the Wikimedia servers. I have no idea how "expensive" Wikimedia-Minify is, so it may or may not be worth the hassle. It's just an idea.

Minification generally doesn't pose a performance bottleneck. The minifier we have was specifically designed and written from the ground up to allow low-latency responses from load.php while applying it to 100% of load.php traffic.

When all caching fails, based on meauring on a production server running PHP 7.4 (mw1461), a bare invocation of JavaScriptMinifier takes around:

  • ~2ms for most files (upto 10K input, e.g. jquery.client.js)
  • ~10ms for pako_deflate.es5.min.js (26KB input)
  • ~50ms for large files like jquery.js (280KB input).

Measured as follows:

  private function runJs( string $file = null ): void {
    $data = $file === null ? stream_get_contents( $this->in ) : file_get_contents( $file );
+   $t = microtime(true);
-   $this->output( JavaScriptMinifier::minify( $data ) );
+   $out = JavaScriptMinifier::minify( $data );
+   echo (microtime(true) - $t) * 1000;
+   $this->output( "\n" . (string)strlen( $out ) );
  }
krinkle@mw1461:/srv/mediawiki/php-1.41.0-wmf.26$ php -d 'opcache.enable_cli=1' vendor/wikimedia/minify/bin/minify js resources/lib/jquery.client/jquery.client.js
2.4581588745117
3311

There is significant caching at every level, including: per-module level on the server-side using php-apcu to cache previous minification results in RAM on each server (this allows partial re-use between different batches that include and minify the same JS file), at the HTTP level (CDN/Varnish), browser level (HTTP max-age), and within JavaScrript (defragmented mw.loader.store in localStorage).

Stats for how often load.php is served by Varnish vs calling MediaWiki, and stats on how often (when Varnish is a miss) when call to ResourceLoader::filter is served from RAM (php-apcu) vs calling Wikimedia-Minify can be found at https://grafana.wikimedia.org/d/000000066/resourceloader as "Varnish frontend cache hit rate" and "minify-js cache hit ratio".

As of writing there is 99.9% cache hit ratio at Varnish level, and that applies to logged-in users as well. Of the 37,000 requests per second, about 200 requests make it to the MediaWiki servers. Those 200 requests per second, invoke JavaScriptMinifier collectively ("minify-js miss.rate") about 18 times per second (or 0.09 times per request). I believe the majority of those 200 Varnish-miss requests don't invoke the minifier at all, with a few of them invoking it 1 or 2 times.

You can use WikimediaDebug to invoke "Excimer UI" and get a glimpse of how long the minifier takes. When invoking it, take into account that it forces a cache-miss aat the Varnish level, and typically routes you to a mwdebug server with no other people requesting on it in recent enough history to have non-empty apcu RAM, thus generally missing the in-memory minifier APCU cache as well.