Page MenuHomePhabricator

[EPIC] Make jQuery an optional dependency in Minerva
Open, NormalPublic

Description

To be clear, I am not saying remove jQuery from our stack - this is complicated just as removing jQuery UI is proving to be (T49145). We can continue to support jQuery in our stack but we must make changes now to prepare for a future where we may not want it.

Background

As various teams, myself included, begin using ES5 it's becoming clear that jQuery is not as useful in parts of our stack. In mobile in particular I'm keen for us to defer the loading of jQuery till after firstInteractive to optimise our critical path (see also T127328).

In mobile, there are few of the browser quirks that desktop relies on jQuery to resolve; and native JavaScript methods are more widely supported. Browsers also have regular update paths, meaning browsers are getting more powerful much quicker than desktop. In mobile the use of the jQuery global has been limited to various places by eslint. When we inevitably support ES6, (which will bring Object.assign) our usage will be even more minimal, and soon it will be hard to justify including jQuery. Other large projects with less technical debt and younger than us than us are achieving just this (recently Github: https://twitter.com/mislav/status/1022058279000842240)

As a skin developer, I would also like to build a skin that doesn't depend on jQuery. Freedom to innovate is important and this doesn't seem to be possible in our current ecosystem.

It's clear (at least to me!) that jQuery is not going to be as important in the future. Strategically we should be untangling our mandatory MediaWiki code from jQuery and make this a dependency that RL modules must declare. This is also important for developments in progressive web apps and external mediawiki libraries and services, for which we will want to make use of common MediaWiki libraries for the API without the forced requirement of jQuery). Already we find ourselves building wdio-mediawiki using other libraries (mwbot) instead of code we ship to our production users.

The fallback skin (which has no interactivity) loads 79.4kb of JavaScript (35.8kb is the startup module and 43.9kb jQuery+mediawiki)

At minimum we should provide an abstraction between jQuery and MediaWiki to allow 3rd parties to use MediaWiki without
We should seek alternatives to jQuery in our core code where ES5/6, non-jQuery plugin libraries exist.

Much of our code needs modernising. For instance, we use $.map in various places even though Array.prototype.map is available in all the browsers we support.

jQuery's plugin architecture and use of chaining makes it hard to gauge usage of the library, but actually our reliance on it for our critical code is minimal and it's essential that we

Benefits

  • Our libraries become more useful outside the MediaWiki ecosystem e.g. in NPM
  • We allow experiences that do not depend on jQuery
  • We make our code more malleable and easy to change and react to changes in frontend tech.

Implementation proposal

  1. Add an interface library that provides library agnostic functionality, where jQuery/browser standards overlap.

This should cover (but not be limited to): adding events, DOMElement selector, Deferred, extend

MobileFrontend already does this to limit and control its jQuery usage:
https://github.com/wikimedia/mediawiki-extensions-MobileFrontend/blob/master/resources/mobile.startup/util.js#L11

Here is a crude limited version of what this will look like:

mw.helpers = {
  querySelectorAll: function ( selector ) {
    return $( selector );
  },
  Deferred: function () {
    return $.Deferred();
  },
  extend: function () {
    return $.extend.apply( $, arguments );
  }
}

In future as browser support improves it might look like:

mw.helpers = {
  querySelectorAll: function ( selector ) {
    return ourCustomMixin( document.querySelectorAll( selector ) );
  },
  Deferred:function () {
    return new Promise( ....
  },
  extend: function () {
    return Object.assign.apply( null, arguments );
  }
}
  1. Update existing critical JS to use new helper library or native JavaScript where possible:
  2. The mediawiki.base module uses $.Deferred() and $.Callbacks
  3. ext.eventLogging (querySelectorAll, $.append, $.html, $.createElement, $.click, $.hide, $.extend, $.type, $.noop, $.Deferred)
  4. Replace jquery.accessKeyLabel jquery.client jquery.cookie and jquery.throttledebounce with non-jQuery dependent solutions
  5. Replace jquery.msg with an on-jQuery dependent solution
  6. mediawiki.Title ($.extend, $.inArray, $.type, querySelectorAll equivalent)
  7. mediawiki.Uri should use Array.forEach instead of $.each and mw.helpers.extend rather than $.extend. Remove usage of $.isPlainObject as decided in T192623
  8. mediawiki.api ($.extend, $.Deferred, $.ajax) including the more complicated mediawiki.api.upload ($.each, $.prop, $.on, $.one, $.css, $.createElement, querySelectorAll, $.addClass, $.remove)
  9. mediawiki.experiments ($.isEmptyObject)
  10. mediawiki.jqueryMsg ($.createElement, $.attr, $.map, $.each, $.append, $.text $.click)
  11. mediawiki.language ($.extend)
  12. mediawiki.template ($.parseHTML)
  13. mediawiki.user ($.extend, $.Deferred)
  14. mediawiki.util ($.noop, querySelectorAll, $.parseHTML, document.ready, $.wrap, $.attr, $.append, $.hasClass, $.removeClass, $.click, $.params)
  15. mediawiki.viewport (querySelectorAll, width, height)
  16. mediawiki.page.startup (querySelectorAll)
  1. Proof of concept
  2. It should be possible to remove the jQuery dependency from the mediawiki.base module and replace it with another library that implements an equivalent interface e.g. using native JS
  3. It should be possible for a skin to extend ResourceLoaderStartUpModule with a different startup module package that does not include jQuery.

Measuring success

We can measure success by looking at the amount of bytes the fallback skin ships.

Event Timeline

Jdlrobson created this task.Aug 1 2018, 2:52 AM
Restricted Application added a project: Performance-Team. · View Herald TranscriptAug 1 2018, 2:52 AM
Restricted Application added a subscriber: Aklapper. · View Herald Transcript
Jdlrobson updated the task description. (Show Details)Aug 1 2018, 2:52 AM
Jdlrobson added subscribers: Nuria, Milimetric, Niedzielski and 3 others.

note: while there is much overlap with T192623 here, I'm not sure if T192623 goes far enough as it doesn't include things like jQuery.msg and EventLogging and that's where my main focus is - please correct me if I'm wrong.

As I understand it, MobileFrontend's startup module is registered as a regular module and is already loaded, in its entirety, asynchronous and not blocking paint/rendering or user interaction. It is typically bundled with the third (serially dependant) JS resource.

The first two request are 1) ResourceLoader's startup module, and 2) the base modules (jquery+mediawiki.js). These requests, as of 2015, are also asynchronous, and non-blocking from perspective of rendering/painting.

Once is T192623 completed, the reality will be that the second request no longer exists. Instead, mediawiki.js is now part of the ResourceLoader startup module (no longer dependant on jQuery), and jQuery will be bundled (in parallel) with any other modules in the second round (previously, third round) of JS requests.

The execution order is kept the same for compatibility, but in terms of time delay, regular JS-modules (such as MobileFrontend) will now execute a full server-roundtrip earlier than currently the case. Thus, allowing the first interaction to take place sooner as well.

d, mediawiki.js is now part of the ResourceLoader startup module (no longer dependant on jQuery), and jQuery will be bundled (in parallel) with any other modules in the second round (previously, third round) of JS requests.

Sure, so I guess this is a continuation of this work - paving the way to remove jQuery altogether.

Krinkle added a comment.EditedAug 1 2018, 3:48 AM

This task, as stated ("make startup and mw object not depend on jQuery) has been completed and deployed to production.

I do not believe that further removing use of jQuery will have any significant performance win in execution time, code size, or bandwidth - because it remains mandatory for many years to come to load it on all pages. And per T192623, we won't be paying the cost of its roundtrip anymore. Allowing code to execute sooner as if jQuery wasn't there.

That's not to say we shouldn't remove uses of jQuery where native options are available today. Various people, myself included, have been (slowly) doing that for years in core, and other places, and help on that would be very welcome! (esp. @Fomafix 93aafce97, 8fb63ba, 1edba80, 9d67e99, ffb376bd, ..)

However, I do not think we should introduced a custom-made framework of our own to abstract jQuery methods. Methods like Array#map and document.querySelector can and are being used directly today (our startup module aborts early if querySelector is absent). They're the language of the web and make code easy to maintain, contribute to, and generally feel familiar to developers.

For more complex Web APIs (such as IndexedDB, or XHR), or utility methods not (yet) on the standards track (such as isEmptyObject) we could certainly introduce an abstraction though (or leverage existing ones in open-source!).

After updating the RFC per the above, I'm not sure there'd be a lot left in terms of mobile/performance, but let's find out.

At the same time, I'm also thinking of other ideas and tasks we have laying around that I know will impact mobile/performance. Perhaps you'd perhaps like to get involved with some of these:

  • T183720 (be interactive without waiting for modules to download)
  • T140664 (faster rendering for users with a session, and effectively support non-PHP skins without tech debt; also effectively unblocks the idea of mobile/micro-contributions without the huge perf hit of bypassing Varnish)
  • T187207 (EventLogging without any code to lazy-load)
  • T176262 (download modules without executing them, avoids hitting the 50ms frame budget in cases where execution wasn't yet needed)
  • T120984 / T121730 (reduce size of render-blocking stylesheets, continue to avoid FOUC on touch/focus via preload instead of embed)
  • And various small tasks linked from T127328 that, together, should noticably reduce bandwidth, script execution, and time to interactive.

These may not all be ideally scoped for reading-web, but I'd be happy to sit and think together to find something that is :)

Legoktm added a subscriber: Legoktm.Aug 1 2018, 4:09 AM

Much of our code needs modernising. For instance, we use $.map in various places even though Array.prototype.map is available in all the browsers we support.

Yes! Let's clean that up. Is it possible to have eslint (or whichever tool) ban jQuery methods where we can use the native ones? e.g. $.isArray(), and a few others...

There are still tasks about killing IE 8 hacks as well that really just need developers to work on them.

Krinkle added a subscriber: Fomafix.Aug 1 2018, 4:11 AM
Jdlrobson renamed this task from RFC: Modernise our JavaScript with the goal that the startup module and mediawiki object should not depend on jQuery to RFC: Modernise our JavaScript with the goal that mediawiki global object should not depend on jQuery.Aug 1 2018, 4:50 AM
Jdlrobson moved this task from Inbox to Next up on the User-Jdlrobson board.Aug 1 2018, 6:57 AM
Ricordisamoa updated the task description. (Show Details)Aug 1 2018, 7:50 AM
Ricordisamoa added a subscriber: Ricordisamoa.

Yes! Let's clean that up.

If specific elements to be replaced could be identified in subtasks I see some potential Google-Code-in-2018 tasks, please? :)

There are still tasks about killing IE 8 hacks as well that really just need developers to work on them.

For reference: T123218: Remove IE8 hacks/workarounds/etc. from extensions.

TheDJ added a subscriber: TheDJ.EditedAug 1 2018, 11:13 AM

If we rly want to fix this in core/extension code, I'd suggest we develop our own eslint plugin to first mark these as warnings in our CI and later mark as errors. Having CI checks has proven critical in the past to move code forward in these areas. As a matter of fact, someone already did:
https://www.npmjs.com/package/eslint-plugin-jquery

We'd still need checks for the mw.helpers stuff though.

because it remains mandatory for many years to come to load it on all pages.

I think this is a bit pessimistic. As I've outlined above, with some minor changes and focus, it's quite realistic to think that Minerva could stop using jQuery for it's critical JavaScript by June 2019. The Minerva skin is JavaScript heavy, but mostly uses jQuery for convenience. Using an ES6 shim would eliminate most of its usages, provided MediaWiki's libraries e.g. mw.Api could be untangled from jQuery.

Also, you forget 3rd party code.
When building https://github.com/jdlrobson/weekipedia I had to clone many of MediaWiki's core files and rewrite them to not use jQuery e.g. mediawiki.messages. This was extremely straightforward and a little frustrating given that most of the usages were unnecessary. Likewise, we use mwbot for our Selenium tests rather than our own MediaWiki.Api code. These libraries should be the same!

That's not to say we shouldn't remove uses of jQuery where native options are available today. Various people, myself included, have been (slowly) doing that for years in core, and other places, and help on that would be very welcome!

It seems a more focused effort would be useful so I've setup T200877

However, I do not think we should introduced a custom-made framework of our own to abstract jQuery methods. Methods like Array#map and document.querySelector can and are being used directly today (our startup module aborts early if querySelector is absent

This wasn't quite what I had in mind. Take $.Deferred for example - it looks like there are over 800 usages of this in our environment. In future we may want to use native JavaScript Promise's. Switching over is going to be time consuming. We can however prepare for this change by providing a wrapper function and start promoting that now.
It might look like this:

mw.Promise = function () {
			var d = $.Deferred(),
				warning = 'Use Promise compatible methods `then` and `catch` instead.';

			log.deprecate( d, 'fail', d.fail, warning );
			log.deprecate( d, 'always', d.always, warning );
			log.deprecate( d, 'done', d.done, warning );
			return d;
}

In future we could make this new Promise under the hood. My point is there are steps we can take now, to simplify our lives in future.

At the same time, I'm also thinking of other ideas and tasks we have laying around that I know will impact mobile/performance. Perhaps you'd perhaps like to get involved with some of these:

These are great and I'm happy they are happening!

because it remains mandatory for many years to come to load it on all pages.

I think this is a bit pessimistic. As I've outlined above, with some minor changes and focus, it's quite realistic to think that Minerva could stop using jQuery for it's critical JavaScript by June 2019.

Sorry for the confusion; I wasn't referring to refactoring Minerva. Minerva (and its dependencies) can definitely be made to work without jQuery, and I'm glad this is being prioritised!

I was referring to the outcome of not loading jQuery on a production page view. The skin is only a small part of that. There's front-end code in core, extensions, gadgets, and site-scripts; all participating on a view. Past migrations took years (wikibits, jQuery 1.9, jQuery 3.0), with the last one taking "only" 12-14 months because it got prioritised for its perf impact (shorter startup time, lower DOM overhead, less browser-compat code).

Without hard data, I'm not convinced it it would (significantly) improve page load times. In a simple/naive web app using jQuery, its removal would likely improve load times a lot. WMF on the other hand, has been living with it for years. And for years, we've done everything we can to avoid its overhead from every corner of the page load process, against its mere presence, and against our use of it. Examples of that are: Loading all code asynchronously, eliminating FOUC, and strongly preferring server-side HTML + plain CSS in favour of JS code (any JS code, regardless of jQuery).

Refactoring code without jQuery might not make a significant load-time improvement. In my opinion, there's even a very real risk of regressing load times if we don't do it well.

I do, however, still agree with the direction. All of our core and extension components definitely can (and should!) aim towards decoupling from this dependency in favour of:

  • native methods,
  • shims for native methods,
  • or narrow libraries for specific features.

My main motivation for that direction is code cleanliness and run-time performance (as opposed to load-time performance). Regarding code cleanliness, front-end code using jQuery has a strong correlation with code smell and tech debt. Not always, but often enough that it's a good way to start refactoring things with good principles in mind. Regarding run-time performance, jQuery methods have a tendency to be overly simplistic in their signature, hiding (too) many significant parts of how the browser works. The result is that developers cannot expresses their expectations, and jQuery has to infer everything, which comes at a high run-time cost. Running code that manually does what you think jQuery is doing (or what it effectively ends up doing) is often much faster than asking jQuery to do it.

Refactoring code without jQuery might not make a significant load-time improvement.

Agreed. I'm interested in it purely from a maintenance pov.

I was referring to the outcome of not loading jQuery on a production page view

If you are talking about a desktop page view sure, but given mobile has very limited number of site wide gadgets, I think this is more doable. It of course requires buy-in to do this on those core modules e.g. mw.loader, mw.config etc

Maybe a more clear goal would be to make it possible for a new skin to be jQuery-less on startup. As someone building skins that would be a highly desirable place to be.

E.g. jquery is a dependency pulled in like everything else in our codebase not part of the startup script.

It is also possible to migrate to ES6 promises while maintaining a jQuery.Deferred-like API. This would allow use to remove the dependency more easily, then switch over to a native Promise API in a piecemeal fashion (if we want to).

See P7788 for an example of $.Deferred() using native promises.

He7d3r added a subscriber: He7d3r.Nov 24 2018, 10:47 AM

I was referring to the outcome of not loading jQuery on a production page view

It of course requires buy-in to do this on those core modules e.g. mw.loader, mw.config etc

As I mentioned at T200868#4467782, this is already done. In FY2018–19Q1, the Performance Team completed its goal tracked at T192623, which included:

  • The ResourceLoader Client and other internals were refactored to not use or depend on jQuery in any way (mw.loader. mw.Map etc.).
  • The load queue and startup module were overhauled to no longer block loading of "page modules" on the presence of jQuery. (Page modules are those modules queued via $out->addModules by skins and extensions.)
Before Sept 2018
Roundtrip 0. HTML
  Roundtrip 1. Startup module
    Roundtrip 2. jquery.js + mediawiki.js ("base modules")
      Roundtrip 3. Page modules (JS from MobileFrontend, Minerva, CentralNotice, EventLogging, etc.)
Today
Roundtrip 0. HTML
  Roundtrip 1. Startup module
    Roundtrip 2. Page modules (JS from ...)

The outcome was that modules execute sooner, and thus time-to-interactive and mwLoadEnd are reached earlier.

Refactoring code without jQuery might not make a significant load-time improvement.

Agreed. I'm interested in it purely from a maintenance pov.

While it took considerable effort of the last 5 years, for better or worse, we've been able to remove virtually all negative performance impact from jQuery's presence on our platform. The only remaining aspects being 1) 30kb of non-blocking/low-priority network bandwidth, 2) the effect on code quality and maintenance overhead as a result of existing jQuery use for use cases we now deem inappropriate.

It has always been possible to develop a skin that doesn't depend on jQuery. The platform ("MediaWiki") will still load it because a majority of our interactive user interfaces in core and in extensions will (and afaik, should; under current best practices) make use of jQuery in some form or another; as well as for user scripts and gadgets. But that's a platform concern, not a skin concern. You can dis-use jQuery and pay close to no penalty for its incidental presence on the page.

I think there's a lot of interesting ideas presented in this task's description and later comments, but I don't see anything cross-cutting or controversial that would require an RFC. Perhaps these would be better tracked as separate tasks, or maybe we can bring together a subset of the related ideas and re-purpose this task for that, and make it into a shared quarter goal for Reading&Perf?

Jdlrobson renamed this task from RFC: Modernise our JavaScript with the goal that mediawiki global object should not depend on jQuery to Make jQuery an optional dependency in Minerva.Dec 12 2018, 12:11 AM

Perhaps these would be better tracked as separate tasks, or maybe we can bring together a subset of the related ideas and re-purpose this task for that, and make it into a shared quarter goal for Reading&Perf?

Nice! Yes, let's talk about what this might look like and dissect this task into smaller pieces. Maybe all hands might be a good time to kick off such a conversation?

Jdlrobson renamed this task from Make jQuery an optional dependency in Minerva to [EPIC] Make jQuery an optional dependency in Minerva.Dec 12 2018, 12:14 AM
Jdlrobson moved this task from To Triage to Epics/Goals on the Readers-Web-Backlog board.
Jdlrobson triaged this task as Normal priority.Jan 11 2019, 12:54 AM

I'd be keen to explore a poc prototype but I notice it's not possible to extend the ResourceLoaderStartupModule and simply change the default modules as I'd hoped...

Could this line (or its function) be made protected so the class can be subclassed?
https://github.com/wikimedia/mediawiki/blob/master/includes/resourceloader/ResourceLoaderStartUpModule.php#L377

How about a new $wg that specified an alternate set of baseModules? Like $wgAlternateBaseModules and if it exists, use that instead of https://github.com/wikimedia/mediawiki/blob/master/includes/resourceloader/ResourceLoaderStartUpModule.php#L376

Jdlrobson moved this task from Next up to Inbox on the User-Jdlrobson board.Feb 28 2019, 8:59 PM
Izno added a subscriber: Izno.Mar 16 2019, 6:45 PM

Looking at the question of base modules specifically, those are implied dependencies for the system overall. It's not something I think makes sense to vary by MW version, site configuration, or skin; because the code written for the platform isn't fragmented in that way, either. They are already the minimum they can be, and adding more would needlessly slow down the user experience.

I do think it's something we could consider changing, but in order to be able to weigh that in terms of cost and benefit, it would help to have a high-level problem/objective. Then myself and others would be able to better offer feedback, suggestions, and think about the intended outcome and how we can achieve that.