Page MenuHomePhabricator

Allow ES8 (ES2017) code in gadgets
Open, Needs TriagePublic

Description

Support use of ES8 code in gadgets. This involves confirming that Grade A browsers support ES8 syntax, switching the Peast validator to ES8 mode, and changes in the JS minifier.

Event Timeline

Restricted Application added a subscriber: Aklapper. · View Herald Transcript

Discussion continued from T277675:

This doesn't work today because MediaWiki core requires ES6 and the gadget syntax validator allows up to ES7.

It can of course be saved (pending T76204), and loaded outside ResourceLoader's checks via action=raw or via the temporary requiresES6 option in gadgets. However, misuse of requiresES6 to store ES8 code is not supported and will stop working after we remove that flag (announced in Tech News at T75714#9442330). The flag was added because we did not yet have a strategy for updating syntax validation. The new solution allows us to trivially raise it in a one-line change any time that MediaWiki core's browser requirements change. There are no plans currently to raise browser requirements to ES8/ES2017 or ES2020 etc. Not likely until browser engines, browser vendors/apps, audience usage thereof catch up.

Please, please keep the requiresES6 flag at least until ES8 can be used in scripts. On frwiki, we have scripts that use async/await, and this language feature makes the codes incredibly simpler. We even had codes with the ?. operator, but I rewrote them to avoid using this operator (for now). But the async/await is a must have.

Alternatively, another temporary solution could be to add an esversion=integer option, which thanks to its more precise control, would replace requiresES6 and its pitfalls. I know the plan is to make all scripts (core, gadget, site, user) use the same ES version, just suggesting a better temporary solution.

Well... we did tell people not to do that.. its not called requiresES6 for nothing.

There are no plans currently to raise browser requirements to ES8/ES2017 or ES2020 etc. Not likely until browser engines, browser vendors/apps, audience usage thereof catch up.

I fought with myself a bit about introducing async/await, but I wouldn't be able to keep the process sane and readable without async/await here:
https://github.com/Eccenux/wiki-DYKCzyWiesz/blob/main/src/DykProcess.js

So I'm definitely for keeping requiresES6. I did have to use that option to make a new Did-You-Know tool work. So I can also confirm that async/await didn't work without requiresES6.

Fun fact. I just learned that chrome status page for async/await is so old that most links on that page are dead already (including the MS demo page and TC39's page). Also Github shows 10 million matched files for await which probably many of them are not for await/async, still that seems like a lot. So I think it is fair to say the feature is very much established. Also interesting fact both jQuery.ajax and mw.Api can be used with await. And the code is SO MUCH better with await.

So please do consider upgrading to next ES.

@Nux Standardisation is not in question. Unlike new Web APIs (such as fetch), which can be polyfilled, or used via a conditional "if" statement, new syntax, if allowed even once, immediately breaks all code in many browsers that we still support. You can complain to upstream TC39 for having introducing features in ES2017 in a way that is backwards-incompatible. When MediaWiki raises its minimum requirement to ES8/ES2017, we will on the same day, raise the ESLint and site script validator to also allow this syntax.

The validator already supports all ES versions upto and including ES2024. The raising of our configuration is a product decision about audience and browser support. It is not a technical decision.

So please do consider upgrading to next ES.

There is nothing for us to upgrade. We intentionally restrict the allowed syntax based on what the browsers used by Wikipedia readers, support. See T178356#8740573 for a peak behind the curtain, in which I lay out the reality of browser support.

The marketing of "evergreen browsers" is mostly a lie spread by privileged developers in the west, given ever steeper and shorter lifecycles for OS/browser support, and also a growing landscape of alternative browsers on Android that embed widely outdated copied of browser engines (e.g. in 2023, the latest version of UCWeb browser on Android, seemingly embedded a version of Chrome from several years older; likewise, iOS and Android devices are often unable to upgrade to a newer browser due to discontinued vendor support).

In any event, the topic of browser support raising is a separate one. Once usage is at an acceptable level, I expect WMF will raise Grade A requirements again, as we always have.

@Nux Standardisation is not in question. Unlike new Web APIs (such as fetch), which can be polyfilled, or used via a conditional "if" statement, new syntax, if allowed even once, immediately breaks all code in many browsers that we still support. You can complain to upstream TC39 for having introducing features in ES2017 in a way that is backwards-incompatible. When MediaWiki raises its minimum requirement to ES8/ES2017, we will on the same day, raise the ESLint and site script validator to also allow this syntax.

I was on the TC39 lists. Changes discussed were far more breaking than they are, even for ES6 (well for Harmony back then). So I'm not complaining. Also, it was then 10 years ago. Back then, I didn't even like building JS with JS and also had a PHP compiler and was working on a JQuery Mobile framework... That was a long, long time ago, and I've moved on. Wikipedia should move on too :)

Meanwhile, please keep this task on-topic. This task is a technical task, about the minifier, not the validator.

I am talking about the minifier. Or at least I think I am. The minifier was breaking on this version of the DYK gadget:
https://pl.wikipedia.org/w/index.php?title=MediaWiki%3AGadget-CzyWiesz.js&diff=72699054&oldid=72675626

There was some exception in JS console. It started to work after adding requiresES6 in this change:
https://pl.wikipedia.org/w/index.php?title=MediaWiki%3AGadgets-definition&diff=72699123&oldid=72407946

I'm using eslint with "es2017": true, so I'm fairly sure the code is compatible with ES2017, so the minifier is not.

The marketing of "evergreen browsers" is mostly a lie spread by privileged developers in the west, given ever steeper and shorter lifecycles for OS/browser support...

I'm quite aware. I work for a company doing software for Polish libraries. Some of them still use Windows XP. As it so happens, though:

In 2017, the last version of the Firefox browser available for Windows XP was released (Firefox 52 ESR)[3][27].

And that is the version that supports async :-). So you know... https://developer.mozilla.org/en-US/docs/Web/JavaScript/Reference/Statements/async_function#browser_compatibility

2023, the latest version of UCWeb browser on Android, seemingly embedded a version of Chrome from several years older; likewise, iOS and Android devices are often unable to upgrade to a newer browser due to discontinued vendor support

Yes, browsers are discontinued, and being evergreen is a lie, you are right. But they are not that old. Even KitKat had updates beyond what Windows XP had.
https://groups.google.com/a/chromium.org/g/chromium-dev/c/ypAS49lvN1M/m/LC0OFek5EAAJ?pli=1

So you know. It's not super, but it is not as bad as you might think :) And the point is to make the gadgets work. Not to raise compatibility for everything. Just for gadgets that people choose to use (or not). So as said - please keep the option requiresES6. Name it requiresESnext if you must, but please keep it.

... if allowed even once, immediately breaks all code in many browsers that we still support.

Which browsers are those? If my understanding is correct (per T178356#8782595), all Grade A browsers (that pass the current feature check) fully implement ES8. Even if MediaWiki core and extensions are still written for ES6, I don't think it's reasonable to disallow gadgets from using features that are supported in all browser clients.

Async/await has been around for quite a while now. It goes a long way in improving the gadget ecosystem, and even makes the language more accessible to people accustomed to writing in non-async systems like in Java. We should not expect people learning JavaScript today to learn the legacy technique just for MediaWiki scripting. After all, they are volunteers with limited time. Online courses on JavaScript today hardly mention anything other than async-await for asynchronous code.

If we want to be safe and ensure ES8 gadgets don't break the core experience in some as-yet unknown browser that we consider supported but doesn't support ES8, we can just put all gadgets in a dedicated RL group.

I feel that the only thing that blocks the switch to ES8 is that async is not yet fully supported in the minifier (per T277675).

I just want to add a few things to those already quoted by @SD0001 (thanks).

I've tested building my DYK tool with Babel.

  • The build is significantly slower. Think multiple times slower. This will differ very much on specific build tools. It takes 1,7 seconds to exec my build via gulp of which the actual build is 485 ms. And with babel+uglify I get 5-7 seconds (yes, that's from 0.5 s to 6 s).
  • The build is significantly larger. I had to uglify it make this reasonable even... It is still about 48% larger then the original. Quite a lot for adding no features.

Also worth noting that JS is not enabled on Windows XP on Wikipedia. I use it as a reference point because even Firefox on Windows XP has async support (see also: T178356#9458355).

And to make this versions less abstract here is a list of changes for each JavaScript (EcmaScript) version:

Version Released Main Changes
6 – ES2015June 2015Syntax changes: Classes (class), modules (import), const and let declarations, arrow functions, default function parameters, function rest parameters (...args), object literals ({variable1,variable2}), for/of loops, template literals (text = Description: ${variable}.;), destructuring assignments, generators (yield), new Unicode sequences, function call checks as classes (new.target). Others include collections (''Set'', ''Map'' etc.), typed arrays, proxies (meta objects).
7 – ES2016June 2016Exponentiation operator (**), Array.prototype.includes.
8 – ES2017June 2017async functions and await expressions, String.prototype.padStart/padEnd, Object.values/entries.
9 – ES2018June 2018Asynchronous iterators, object property rest syntax, new regular expression features (including named groups), finally function for promises.
10 – ES2019June 2019Array flattening methods (flat), changes to array sorting method and Object.fromEntries method.
11 – ES2020June 2020Optional chaining operator (a?.b) and nullish coalescing operator (??), BigInt primitive.
12 – ES2021June 2021String.prototype.replaceAll(), Promise.any(), weak references (including WeakRef), logical assignments, long number separators (thousands).
13 – ES2022June 2022Class fields, static class properties, private class methods, Object.hasOwn, cause exception property, /d option for RegExp.
14 – ES2023June 2023[].findLast, [].findLastIndex, support for hashbang.

[…] we can just put all gadgets in a dedicated RL group.

It's not that simple I'm afraid. To diverge browser support in Gadgets from the rest of MediaWiki in this way you'd need customisation for mw.loader, for startup.js, script validation override, opt-out from edit save hooks for T76204, as well as messaging on Special:Preferences, and presumably some level of education/awareness to deal with the fact that some modules can no longer be safely loaded as dependency by other modules. For example, extensions sometimes await user, site or a specified plugin/module for VisualEditor.

This would incur high tech debt and take a significant step back for accessibility. It favours a reality in which most developers will (unknowingly) pretend something works that in fact does not. The jump from Basic to Modern is fairly easy to test and explain. A secondary gap within this layer is not likely to receive the same level understanding.

[…] I've tested building my DYK tool with Babel. The build is significantly slower. […]

This is an excellent argument against such build step! We generally don't use one for MediaWiki. Do you still need it? For many devs, they mainly did this in ~2013 as a way to get "Package files" and ES6 syntax. These are supported natively now. Build tools now mostly a left-over trend from the 2010s era of frontend development, involving needless complexity for largely unproven and misunderstood benefits.

Having said that, I believe the slow down is avoidable, and likely originates from non-standard features (like Webpack or TypeScript), or needless transpilation (e.g. Babel is often configured to with a legacy ES5 target).

If the Git repo for your gadget uses standard JavaScript, and esbuild with --target=es2016, you can enjoy async-await today with fast and reasonably sized build outputs.

Async/await has been around for quite a while now. […]

  • Version: 8 – ES2017
  • Released: June 2017
  • Syntax changes: async functions and await expressions, String.prototype.padStart/padEnd, Object.values/entries.

I believe you're quoting the date when the TC39 committee publishes their annual revision to the formal ECMAScript specification.

This is different from when browser engine vendors implement the first of these features.
This is different from when browser engine vendors implement the last of these features.
This is different from when browser engine vendors ship releases.
This is different from when browser app vendors update their forked or embedded engines.
This is different from when customers upgrade their devices or apps.

I've shown in previous tasks how we measure, analyze, and verify browser support. As well as examples of sometimes surprising delays in adoption in some cases. See also:

[…] We should not expect people learning JavaScript today to learn the legacy technique just for MediaWiki scripting.

I would not describe ES6 Promise as a "legacy" technique. It is very much a part of JavaScript today. The language is, to a first approximation, only getting bigger, not smaller. In any event, this point is exactly why we developed the Principles section in our guidelines:

  1. Users. […]
  2. Developers. Strive for the best developer productivity as long as it doesn't compromise the user experience […]
  3. Servers. […]

By structuring it this way, we mostly avoid arguments about productivity. These can't "win" over the user experience. In practice, however, I find it also saves developer cost. This is because it means we reduce the need for smaller communities to duplicate efforts by developing their own tools merely because someone upstream couldn't be bothered to spend a few seconds to satisfy an extra lint check.

This is important because one could make a variant of your point about every ES version, including version ES2023 of last year. Knowing what exists is not enough to decide where to draw the line. Another way to look at it, is that we shouldn't encourage adoption of technologies that alienate part of our "Modern" audience, and further the equity gap within a given population group.

Of course, when a technology is ready, we should adopt it! ES2023 isn't ready yet. But is ES2017? It might be!

We aim to be fairly aggressive in raising JavaScript requirements for modern browsers, which reduces costs of development and maintenance, and also reduces payload size ("page weight"). This aim is only achievable when components start out with a solid and functional Basic experience, with server-rendered access to information, and traditional request-response cycles for contributing to the wiki.

Now that we have well-maintained validators and minifiers that are easy to update, the blocker here is not technical, but one of proof, usage data, social awareness, and management decision. Once a decision is made, relevant folks can make the needed tweaks. I recognise that patches by Hannah and myself on T277675, may have confused you. The only reason you saw us working on that was because it made for a good on-boarding task to explain the code base to a new member of the team. Normally, we'd do that during a week or two after a decision is made, not before.

... if allowed even once, immediately breaks all code in many browsers that we still support.

Which browsers are those? […]

I suggest renaming this task and focussing your effort instead on raising the requirement for MediaWiki in general. You could then also kickstart it by identifying the relevant browsers/devices in question. Specifically, the "Practical implications" and "Browser support matrix" tables, noting that for Chrome/Firefox.

Further reading:

[…] I've tested building my DYK tool with Babel. The build is significantly slower. […]

This is an excellent argument against such build step! We generally don't use one for MediaWiki. Do you still need it? For many devs, they mainly did this in ~2013 as a way to get "Package files" and ES6 syntax. These are supported natively now. Build tools now mostly a left-over trend from the 2010s era of frontend development, involving needless complexity for largely unproven and misunderstood benefits.

I agree with the first sentence. I don't want to take that step. I want MediaWiki to support ES 2017 for gadgets :-)

Actually, I would love it if the minifier supported ES2020 (e.g., optional chaining) and let me decide which browser I need to support in a specific gadget, depending on the gadget audience.

Version: 8 – ES2017
Released: June 2017
Syntax changes: async functions and await expressions, String.prototype.padStart/padEnd, Object.values/entries.

I believe you're quoting the date when the TC39 committee publishes their annual revision to the formal ECMAScript specification.
...

Yes, but Firefox 52 was also released in 2017 (2017-03-07 to be exact). MDN added release dates to their compat tables. You can click on each version and see when was its release. For Chrome that was version 55 Released 2016-12-01.
https://developer.mozilla.org/en-US/docs/Web/JavaScript/Reference/Statements/async_function#browser_compatibility

Also worth noting that the TC39 process now requires that there is an implementation already before proceeding to the final stage of releasing an ES standard.

Polish Wikipedia has a good article on that ;)
https://pl.wikipedia.org/wiki/TC39

I suggest renaming this task and focussing your effort instead on raising the requirement for MediaWiki in general. You could then also kickstart it by identifying the relevant browsers/devices in question. Specifically, the "Practical implications" and "Browser support matrix" tables, noting that for Chrome/Firefox.

I strongly disagree here. Things that are not equal should not be treated equally. All gadgets are, by definition, enhancements. Core functions are already available from well... the MediaWiki core team(s). Community developers provide add-ons based on core functions. Gadgets are only loaded when they are enabled. If they don't work, people can disable them and still be able to perform all tasks without them.

As I said, gadgets are enhancements. Core is core, core needs to work. Gadgets work if they work.

As I said, gadgets are enhancements. Core is core, core needs to work. Gadgets work if they work.

As someone who spent years maintaining user scripts and gadget on en.wp and providing support for users on WP:VP/T..... There is some truth to that, but I'd argue that a significant amount of the userscripts and gadgets are continuously broken. They cause issues for people where their javascript sometimes doesn't load. They cause errors to show up in the WMF logging.

Additionally, gadgets can be enabled by default. That means that your code runs on ALL browsers that support (right now) ES6. This can create a lot of noise for all of us. And that is WITH the most basic of guards (syntax compatibility checks). You argue that people will be 'just fine' without guardrails. And maybe en.wp would be. But lots of the smaller wiki's definitely would not.

As someone who spent years maintaining user scripts and gadget on en.wp and providing support for users on WP:VP/T..... There is some truth to that, but I'd argue that a significant amount of the userscripts and gadgets are continuously broken. They cause issues for people where their javascript sometimes doesn't load. They cause errors to show up in the WMF logging.

Is that related, though? People always report on Discord or Village Pump first. Most people don't know how to use Phabricator. When I suggest to people to report problems here, they are usually reluctant to do so. So when things break, communities notice quite fast and help each other. The type of error doesn't really mater that much. The more severe error the sooner people notice ;)

Also, personally, I have access to logstash.wikimedia.org and... the most errors are on: en.wikipedia, en.m.wikipedia, fr.wikipedia, zh.m.wikipedia, commons.wikimedia, zh.wikipedia. Mostly TypeErrors.

(off-topic: I'm really sorry to hear that fr.wikipedia is still amongst the most error-throwing wikis. I had a look for requesting Logstash access, but that sounds just a bit too cumbersome to me. Though, would it be possible to provide me some information to spot the most problematic codes, without breaking the NDA?)

(off-topic: I'm really sorry to hear that fr.wikipedia is still amongst the most error-throwing wikis. I had a look for requesting Logstash access, but that sounds just a bit too cumbersome to me.

This is no more than logical. The most used wikis will generate the most errors. As such, the top 10 wikis will always be disproportionally be represented in the logging compared to the actual error surface.

Though, would it be possible to provide me some information to spot the most problematic codes, without breaking the NDA?)

I can copy paste a few stack traces to a pastebin. But please file a separate ticket for that and assign it to me, so that I don't forget. I can't do that right now, nor does it particularly have anything to do with this ticket.

Most gadgets are written for a limited audience; IMO the gadget author is usually best placed to know that audience and make productivity / inclusivity tradeoffs. The question is how much work on the platform itself it requires to make such tradeoffs safe to do, and is that a worthwhile investment compared to other things that could be done with that amount of effort. I think it would need

  • replacing requiresES6 with something more meaningful and accurate like requires=ES2018
  • having peast validate that the gadget really conforms to the given version
  • disabling minification for versions not supported by the minifier (and probably warning the gadget author about this fact)
  • having ResourceLoader detect the level of support in the browser and failing with a clear error message if an unsupported module (or a module with an unsupported dependency) is requested
  • having Gadgets detect the level of support in the browser and indicating to the user on the preferences page when they cannot use a gadget because their browser doesn't support it.

...which seems like a fair amount of work.