Page MenuHomePhabricator

Update JavaScript syntax checker for gadgets and user-scripts for ES6
Open, LowPublic

Assigned To
None
Authored By
Schnark
Nov 24 2014, 9:01 AM
Tokens
"Like" token, awarded by SlickKilmister."Like" token, awarded by AS."Like" token, awarded by DannyS712."Like" token, awarded by Silent."Like" token, awarded by Masumrezarock100."Mountain of Wealth" token, awarded by Daimona."Like" token, awarded by TerraCodes."Like" token, awarded by Shreyasminocha."Like" token, awarded by Vedmaka."Doubloon" token, awarded by Yair_rand."Like" token, awarded by MichaelSchoenitzer."Yellow Medal" token, awarded by Ricordisamoa.

Description

Steps to reproduce:

  1. Open your Special:MyPage/common.js for editing, add the code
var val, a = ['a', 'b'];
for (val of a) {
	console.log(val);
}
  1. Preview your change. The console should show "a" and "b" (given your browser support that much of ES6, but most browsers should, by now).
  2. Save the page.

Expected result: Same as step 2.
Actual result: The code from common.js is replaced with code that just throws an error:
Error: JavaScript parse error: Parse error: Unexpected token; token ; expected in file 'Benutzer:Schnark/common.js' on line 5

While for the site wide MediaWiki:Common.js etc. this is a good feature (to make sure nobody accidentally puts in some ES6 features only available in his browser but not in other users' browser), for personal JS a user should not be prevented to use whatever features his browser is able to provide.

Workaround

See T75714#4674421.

Related Objects

Event Timeline

There are a very large number of changes, so older changes are hidden. Show Older Changes

I have also removed the sub task for requiring ES6 for MediaWiki Grade A code because that can be enforced statically via ESLint already. It does not prevent us from being able to parse and validate ES6 syntax in user scripts or other opt-in areas.

Once the minifier is able to handle ES6, sooner or later some common.js (or default gadget) will start to use ES6 syntax and thus break in older browsers.

While ESLint can force ES5 compatibility for core JS, it can't do so for site JS.

@Schnark Indeed. For wiki modules such as the 'site' module (and perhaps default-on gadgets) we may want to keep using the old parser for validation, or perhaps the new parser will have an option to decide what ecmaVersion to allow and we could pin those to ES5.

For example, ESLint uses an ES7-compatible parser, but has modes to only allow ES3, ES5 or ES6 features.

Under MW 1.29.1, if I insert a template literal (surrounded by backticks) into a script in the MediaWiki namespace, the ResourceLoader gives a parse
error.

You can disable this ES3 validation step with the $wgResourceLoaderValidateJS configuration variable. Which should make it so that you can use ES6 syntax on your wiki.

Please note that this feature does exist for a reason, and disabling it may have certain side effects. Most notably, it means that

  1. People that visit your wiki using older browsers (those without full ES6 support) will have no interactive features at all (not even basic stuff like search suggestions, or collapsible tables). And if they are using Internet Explorer or Opera, the browser may also tell the user that the website is broken "Script error".
  2. If an administrator on your wiki makes a mistake and saves an edit to Common.js with a syntax error, all interactive feature will be broken for all users, including users with modern browsers.

When the ValidateJS feature is left enabled (as is the default), these issues don't happen because it will have found the syntax on these wiki pages before the code is combined. It then substituted the broken script with a valid script that simply logs a message to the console - thus preserving the integrity of the JavaScript bundle overall, and with that other features.

I think the only significant browser that does not support ES6 today (apart from some very fringe ES6 changes which are less widely supported but no one ever uses, like a slightly different set of unicode characters allowed in identifiers) is IE 11, which is currently ~3% of our traffic. Which is large enough not to break, but small enough not to optimize for. Maybe we could allow modern syntax and transpile it to a legacy version for old browsers (using Babel or something like that)? There probably won't be any PHP library we can use, but for Wikimedia sites, it could be done via a service / local executable, and for sites on shared hosting who can't make use of such functionality, the loss of IE 11 users seems acceptable.

(The other option would be transpiling statically, which is I think what the WMF Web team has adopted. For gadgets, that's probably not workable though.)

(The other option would be transpiling statically, which is I think what the WMF Web team has adopted. For gadgets, that's probably not workable though.)

Well, that’s what I currently do for the AC/DC gadget. I might be the only one, though :)

Although setting the $wgResourceLoaderValidateJS to false is too dangerous, why not add a feature to support validating es6 or higher version JS for the site which dont care about IE visitor?

@AnnAngela This task is not blocked on deciding how to implement it. It is blocked on the cost of implementing the feature in question. That will require an engineer to spent significant amounts of time to work on that. That engineer is likely to be me at some point. I have other priorities at the moment. This will not become my priority until after it becomes useful for Wikipedia to enable such feature.

Krinkle renamed this task from ResourceLoader JavaScript parser should allow ES6 syntax features to Allow ES6 syntax features in gadgets and other site-wide scripts.Jun 16 2020, 3:39 PM
Jdlrobson added a subscriber: Jdlrobson.

FYI gadget developers are working round this by loading the JS raw [1] and there seem to be a large amount of errors coming from gadgets where users are trying to load gadgets that are written in ES6 and their browser is incapable of viewing so this is a concern from the production error logging we are rolling out.

Two particular examples:
https://zh.wikipedia.org/wiki/MediaWiki:Gadget-confirm-logout.js
and
https://ru.wikipedia.org/w/index.php?title=%D0%A3%D1%87%D0%B0%D1%81%D1%82%D0%BD%D0%B8%D0%BA:%D0%94%D0%B8%D0%BC%D0%B074/yoficator.js

In the case of the former, I had to patch it with feature detection as it was one of our top errors as it was being loaded as a site wide gadget.

Allowing gadget developers to run ES6 is helpful, but we should have better protection for users who opt into them without knowing their browser doesn't support them.

Possibly adding support inside mw.loader.load for checking ES6 support might help here. Could mw.loader.load calls be restricted to ES6 only browsers or catch and report such errors to end users via mw.notify?

[1] e.g. https://global-search.toolforge.org/?q=yoficator.js&regex=1&namespaces=&title=

AS added a subscriber: AS.

This task is not a production error. More generally, code from gadgets are outside the scope of Wikimedia-production-error.

If RL could minify ES6, it would still crash if served to ES5 browsers. This task is about supporting minification for user scripts that want to use ES6.

I refer to T262493 for the general issue of not being distracted in Logstash by community choices in user scripts.

Allowing gadget developers to run ES6 is helpful, but we should have better protection for users who opt into them without knowing their browser doesn't support them.

We don't right now. The code in question ceased to be a gadget to moment it became a raw-loaded plain text file loaded over HTTP outside MW or RL. When enabling a gadget, it is expected to work if the option is presented. We have skin filters for gadgets that want to limit their support. One could file a feature request to the Gadgets extension if you think we shoul also fragment that by browser support.

More productive use of our time, I think, however would be spent on the future of the web and not its past, such as using ES6 natively and raising our browser support accordingly. The bifurcation we see today is short-lived and unlikely to happen again any time soon. It's recognised as a mistake the way ES6 jumped ahead of reality and turned developers against their own rational principals. With more iterative and polyfillable changes going forward, this won't be neccecary. (One could also argue, that any useful features in ES6 were already polyfillable and that the syntax is insignificant.)

See also the direction of global gadgets more generally, which if placed in Git, would allow for more normal development tooling, including for example to transpile from other syntaxes. As I understand it, most well-maintained gadgets naturally are already maintained outside MW as developing a code base of any meaningful complexity is not feasible in the form of wiki pages and no tooling, CI, or staging ability. Hence why this and related tasks have been fairly low priority.

I think people also like to underestimate how much browser support there actually is for ES6. ALL major browsers (this includes mobile) have been shipping with full ESM support for over 2 years now, and old browsers aren't common and are becoming rarer by the day. As long as old browsers are considered so that the webpage doesn't break and has basic functionality, is it really necessary to spend so much effort in trying to give them ALL the same features? At this point, is it really any different from someone who disabled JavaScript in their browser? Everyone has the ability to get an up-to-date browser, they are really making their own bed and should carry the consequences themselves.

More productive use of our time, I think, however would be spent on the future of the web and not its past, such as using ES6 natively and raising our browser support accordingly.

This is not only something that would benefit the project itself, but It would open up the doors for hords of enthusiastic and skilled JS/ES devs who might be interested in contributing to this cause, but are effectively unable to do so because of the archaic conditions. I would love to do work related to scripts and gadgets, but the effort required is insane for me. I recently made a simple script to hide a few boxes that were bothering me; what would usually take me around 5 minutes to do ended up taking over 1 hour....

I started working with JS/ES a bit over 2 years ago, and have been using TypeScript exclusively for about 1 year. I have zero experience with working in a world without Intellisense and easy to use automation. And looking at the popularity of JS/ES and TypeScript in OSS, I can't be the only one who was discouraged by this to contribute to the project.

It should also be noted how ES6+ syntax tends to have slightly better performance (most notably in the V8 engine), and how it can massively reduce effort needed for code maintenance and debugging with the TypeScript Language Service and all the tools that rely on it.

Krinkle renamed this task from Allow ES6 syntax features in gadgets and other site-wide scripts to Update JavaScript syntax checker for gadgets and user-scripts for ES6.Dec 13 2020, 8:34 PM

This task is about the development of the JS syntax checker used by user-supplied gadgets so that use of ES6 syntax is possible there.

For MediaWiki software development in general, there is task T178356, where it will be decided whether to actively require ES6 (discard ES5) browsers.

Note that use of modern web platform features, including ES6 features like WeakMap and Promise, and other APIs like Navigation Timing and Web Audio, are already possible in gadgets today and has no relation to the syntax checker.

This task appears to be incorrectly named. At the moment the JS syntax checker blocks use of ES6 in (i) gadgets, (ii) sitewide JS pages and (iii) user common.js page. It is very much possible for user scripts to use ES6 and import them using user-level common.js page (or from gadgets or sitewide JS pages, for that matter). It's just that ES6 shouldn't directly appear in them. https://en.wikipedia.org/wiki/MediaWiki:Gadget-CommentsInLocalTime.js is an example of a gadget that uses ES6 by loading it from userspace.