Change 751183 had a related patch set uploaded (by SD0001; author: SD0001):

[mediawiki/extensions/Gadgets@master] [WIP] User level gadget repositories

https://gerrit.wikimedia.org/r/751183

A user defined GadgetDefinition sounds interesting.

However, the exercise has two viewpoints:

1. I am defining my own userspace in my Gadgets-definition.
   1. There is a one and only global name (but perhaps localized??) as page identifier and just one Gadgets-definition per user. Similar to Special:MyPage/Common.css which has a globally unique meaning. Well, better no localization but Special:MyPage/Gadgets-definition for all gadgets.
   2. I can define bundles of resources, one gadget identifier will need a particular .js, a sub-.js, a .css resource.
   3. There might be a mixture of dependencies, some resources from my own user space and some from site gadgets.
   4. Special:Gadgets may be generated for this particular user name: Special:Gadgets/user:SD0001
   5. It might become necessary to apply for appropriate content model, but new pages should be equipped properly.
2. I am using gadgets of other people, perhaps from a different wiki.
   1. Using the URL format?
   2. Resource bundles, aka gadgets, are accessed by a gadget ID which is currently defined by MediaWiki:Gadgets-definition and which will need to be prefixed by user: and user nick and slash.
   3. Specifying my favoured default loadings even from other users and one day from other wikis might be a challenge. However, Special:Preferences#mw-prefsection-gadgets might be extended by a further chapter with sections taken from myself.

Why not extend to mw.loader.load|using("user:SD0001/doit") since that is just another gadget ID, and it is well defined how to create such module?

• No problem with cacheable modules. Cached compressed source code bundles are benefits from that infrastructure.

A major goal would be to keep as much as possible in line with the site gadgets infrastructure, and to reuse as much code as possible. This will make it easier for both developers and global code maintainers as well as users within wikis. Do not re-invent the wheel.

There is a one and only global name (but perhaps localized??) as page identifier and just one Gadgets-definition per user. Similar to Special:MyPage/Common.css which has a globally unique meaning. Well, better no localization but Special:MyPage/Gadgets-definition for all gadgets.

For user gadgets, we would not want to use MediaWiki:Gadgets-definition syntax – the gadget definition namespace and associated content model (which are in master, though not enabled in production) is preferable, since all site gadgets would one day be migrated to that. An additional advantage of this approach is that users would not have to add arcane syntax to a personal subpage – instead they get an intuitive GUI on Special:Preferences:

I am using gadgets of other people, perhaps from a different wiki. A. Using the URL format?

Site gadgets also don't support loading anything from a different wiki. I've filed T298561 to try resolve that. Whatever is the solution reached could be extended to apply for user gadgets too.

Why not extend to mw.loader.load|using("user:SD0001/doit") since that is just another gadget ID, and it is well defined how to create such module?

For an RL module to be loaded via mw.loader.using/load, it needs to be registered server-side. That's not something that can be done by a JavaScript function. Unlike site gadgets (all of which are always registered even if not enabled), it's not possible to register every single user gadget as there could be thousands which would blow up the size of module manifest in the startup module. Triggering server-side registration on demand from client side is essentially T29561.

I am thinking in three levels, and I could imagine that all three will be available one day:

• global
• site
• user myself

I would expect in Preferences that all three chapters are offered in future. However, I am not sure which century. The generation of entries in Preferences is identical for all three origins. However, global specifications will need translatable texts.

Global brief explanations and headlines would need to be multilingual for current user language. They shall advertise the promised global gadgets available in all WMF projects. Their gadget ID might be preceded by global: to avoid collisions with site ID.

Nobody will start to implement and maintain different systems and introduce a deviating definition language for users. The form derived of Special:MyPage/Gadgets-definition and presented on Special:Preferences is exactly the same as for site gadgets but a different chapter on the same page.

The current MediaWiki: location of MediaWiki:Gadgets-definition will be just shifted to its own namespace #2302 one day, together with description and headline text elements. The syntax and implementation will be kept, just page name and content model are subject for change. And resources are to reside in #2300 namespace.

Since I would expect gadget ID in mw.loader.load("enwiki:HotCat") it is obvious for RL how to resolve the request into a resource module from that wiki. All requests from that wiki might be collected and merged into one cacheable multi-module URL for that wiki. This is the same as for resources from current site.

A Gadgets-definition Editor tool may be offered by someone some day, which could make it easier to generate and modify lines in Gadgets-definition via interactive support: Select available resource pages from associated namespace via narrowing page selector drop box, tic various options on/off or use multi-select elements for some options. Link to every dependency page source even as redlink. Since site and user are sharing the same format it is the same approach for both interface editors and regular users.

So far I've been able to get user gadgets to work with group=user and group=private, but with two major limitations:

1. Dependencies don't load. It appears that both group=user and group=private embed modules in the page rather than link them, and both don't support dependencies (or just behave weirdly with them, see T299288)
2. If the docs are to be believed, in group=user caches aren't shared with other users – which is bad because a single user gadget is same for all users so caches should be shared. In group=private I think they aren't cached at all.

On trying to use 'site' or any other string as the group, everything falls apart – modules are now linked rather than embedded, so in the call to load.php?modules=startup the usergadgets must get registered. But as the session is not initialised in load.php requests, there's no way to know who the user is and hence no way to read their preferences.

So it appears some resourceloader changes would be required – specifically a new group for user gadgets in which: edit: not required, see T36958#7643239

1. Modules are embedded rather than linked (to overcome the above difficulty). Are there any performance issues with embedded modules?
2. Dependencies should load
3. Caches are shared across users. And ideally changes trigger cache misses like in group=user

Hm... Hi there. I was quoted on top 🙂, so two things:

I'm not sure, but I think this probably means we need to provide a way to set position = top on user scripts. Maybe it is related to the need of creating modules from wikipages, maybe not.

I actually do not need that as much any more. I use Stylus (a browser extension) if I want to make some changes quick. Also browsers are much faster then 10 years ago so that flickering effect when css is added is not as much as of a problem as back then. So AFAIK that part alone is probably not a big deal any more.

But anyway you seem to be doing something different I guess?

User-script authors define modules in their userspace, with page title having a .gadget suffix. These pages get the GadgetDefinition content model. Definitions are in the same format as the existing Gadget definition namespace (with some changes like not having a "default" and "hidden" keys).

Would that simply mean something like a package definition? Like for Gulp or Webpack? Because IIRC Resource Loader is basically like a Webpack from before Webpack existed, right? If so maybe it would be worth to allow devs to use Webpack on a remote server and publish a script that is already built? It would then be off your hands. No need to worry about build tools, and caching too.

I might have missed the point of the user-level repositories though. Sorry if that is so.

@Nux This task is essentially about T36958#387742, though I believe setting position is no longer an explicit RL feature (any module with just styles load without flicker).

Integrating Webpack into MediaWiki to replace or supplement RL's build steps is a different thing altogether warranting a separate ticket. Even an already built script still needs RL for caching and efficient delivery alongside other modules.

Change 756169 had a related patch set uploaded (by SD0001; author: SD0001):

[mediawiki/core@master] resourceloader: Minor tweaks to support user-level gadgets implementation

https://gerrit.wikimedia.org/r/756169

After doing some more research, I figured out a better approach (see linked patch, ready for review) that does not involve embedding modules on the page. This approach enables user gadgets to exist as normal modules, triggered by a user-level startup module. The issues with the earlier approaches are resolved, and only trivial resourceloader changes are needed. Contrary to bullet 5 in T36958#7594392, user gadgets can be loaded adhoc via load.php?modules=ext.usergadget.p12345 (where the number is the pageid of the definition page) though this would not be considered officially supported.

With the above patch, preferences of enabled user gadgets are public. This helps with caching of the gadget startup module (group=user which enables them to be cached, unlike group=private which adds them to the HTML on every page load) – they get exposed from load.php?modules=ext.gadgets.userstartup&user=<username>.

I suppose this is fine regardless of the caching benefit as it mirrors the public nature of personal common.js pages, enabling technical editors to help others and identify use of malicious scripts.

Test wiki created on Patch demo by SD0001 using patch(es) linked to this task:
https://patchdemo.wmflabs.org/wikis/3ced66066f/w/

Tagging Security-Team for review per Tgr's comment on patch (from last year). Have also updated the task description.

A high-level implementation overview is as follows: (some of this is repetition from T36958#7594392, apologies!)

• The "Gadgets" tab of Special:Preferences adds a multi-titleselect field in which user can select user gadget definitions they want to load.
  • The persisted preference contains the pageid of the gadget's definition page, so if the page is moved (such as when the user is renamed) the preference moves with it.
  • Unlike other preferences, user gadget selections are public, see T36958#7643240
• User gadget definitions are in the same format as the upcoming site gadget format (provided by GadgetDefinitionNamespaceRepo) – a JSON page with schema [0]. Some options – default, hidden and supportsUrlLoad – are not applicable for user gadgets.
• Each gadget can specify JS, CSS and JSON pages – which can be in MediaWiki or user namespaces. (Currently user script loading is more lenient – it also allows protected pages in any namespace[1])
• They can specify other RL modules as dependencies - including other user gadgets.

The intent overall would be to supersede user scripts - which is a quite less secure setup.

[0]: https://github.com/wikimedia/mediawiki-extensions-Gadgets/blob/master/includes/Content/schema.json
[1]: https://github.com/wikimedia/mediawiki/blob/84eb01e73b01765d117a7228f3f1a77780783fb5/includes/actions/RawAction.php#L168

In T36958#8878064, @SD0001 wrote:

• Each gadget can specify JS, CSS and JSON pages – which can be in MediaWiki or user namespaces. (Currently user script loading is more lenient – it also allows protected pages in any namespace[1])

I'm not certain this is that much better than what userJS allows right now, especially if user gadgets can pull any random resource from the user namespace.

The intent overall would be to supersede user scripts - which is a quite less secure setup.

Ok, is there any plan of action for this or any WMF interest/sponsorship? This would be an absolutely massive feature deprecation and I'm not certain that adding support for user gadgets alone would be anywhere close to the amount of effort required to eventually disable userJS. For now, this really just feels like a new set of features for ext:Gadgets.

The intent overall would be to supersede user scripts - which is a quite less secure setup.

Ok, is there any plan of action for this or any WMF interest/sponsorship? This would be an absolutely massive feature deprecation and I'm not certain that adding support for user gadgets alone would be anywhere close to the amount of effort required to eventually disable userJS.

No. What I meant is that if the feature is merged, communities may start using user gadgets instead of user scripts due to better performance, intuitive GUI for enabling/disabling, and somewhat better security. So user scripts get superseded, though I agree we may never be able to actually remove them.

As you've moved this to SecTeam-Processed, I take it the security team has no objection to this feature? (I was expecting to see a risk rating.)

In T36958#8891089, @SD0001 wrote:

As you've moved this to SecTeam-Processed, I take it the security team has no objection to this feature? (I was expecting to see a risk rating.)

No, that's not what that tag means. Per the Phabricator description, it basically implies that the Security-Team has acknowledged the task and may (or may not) work on it in some way. It's purely an internal tracking tag for our team and does not grant any sort of implicit approval or risk rating.

To have code more formally reviewed by our team, we use the current application security review process, which occurs on a quarterly cadence. We generally do not review every potentially-security-implicated feature in gerrit, as we are not resourced to handle manual code review of that much code. About 54,000 change sets were pushed through gerrit over the last year, which would be about 148 change sets per day for a team of about 3 people to 1) determine if they had anything to do with security or not and 2) manually review the changes, which span over 1,200 repositories of incredibly diverse and complex code. If you'd like a security-review of the Gadgets changes you're proposing, please feel free to submit a security-review request. But to be honest, without a plan to deprecate userJS, the proposed changes seem to just be introducing more complexity (eventual Technical-Debt) that might be marginally better than the current userJS attack surface. So the review would likely come back with a medium+ risk rating, which has to be accepted by a manager+ at the WMF for production deployment.

But to be honest, without a plan to deprecate userJS, the proposed changes seem to just be introducing more complexity (eventual Technical-Debt) that might be marginally better than the current userJS attack surface.

The key motivation here is performance rather than security (as user scripts don't enjoy any kind of ResourceLoader optimisations). Even when WMF employees used to work on Gadgets (the work on Gadgets 2 and 3), I don't think removing user scripts altogether was ever a part of the plan. Judging this solely on a "how does this help with mitigating userJS attack surface" metric doesn't seem fair.

@Krinkle Is any there any interest from performance team to steer this forward?

In T36958#8893133, @SD0001 wrote:

Judging this solely on a "how does this help with mitigating userJS attack surface" metric doesn't seem fair.

It may not seem fair, but that would be something we'd absolutely consider during a security review. Otherwise, again, this is just adding a novel attack surface to the general userJS/Gadgets ecosystem, even if it gets us to a marginally better place than userJS in isolation.

T272297: User script on user subpage doesn't work after user rename is something to keep in mind when deciding how user-level gadgets are referenced.

I want to note that not only this would be a helpful feature; the current user script architecture is, in fact, broken. It has been tacitly assumed since old times that user scripts are all enduser scripts and not modules to be reused by other scripts. But there is benefit to them being reused, and some indeed are, like libraries or utilities like this one I just wrote. They can be made into gadgets, but that would clutter the definitions, affect the overhead for regular users, and require a tedious wiki-bureaucratic process.

The basic issue is that user scripts have no straightforward way to be loaded as dependencies with their dependencies and expose their exports to the client, like this:

mw.loader.getScript('url, or maybe a page name (or oldid for versioning) with an interwiki prefix').then(({ exportedFunc }) => {
  // Use exportedFunc()
});

If the user script loads its own dependencies, you have to hook to the moment when they are ready, which generates additional clutter both in the user script and client code:

mw.loader.using([...]).then(() => {
  function exportedFunc(...) {
    // ...
  }
  mw.hook('userjs.someScript.ready').fire({ exportedFunc });
});

mw.loader.load('https://meta.wikimedia.org/w/index.php?title=User:Example/someScript.js&action=raw&ctype=text/javascript');
mw.hook('userjs.someScript.ready').add(({ exportedFunc }) => {
  // Use exportedFunc()
})

(You could probably even employ mw.loader internals instead of mw.hook and treat the script as a ResourceLoader module.)

That said, introducing user-level gadget repositories is not the only way to implement the support for what I described. My concerns with what is described in this task (and @SD0001's implementation?) are:

• Cross-wiki use of scripts. This issue is not even solved for gadgets, yes (search modules= in T204201), but user scripts are among the first candidates when it comes to streamlining cross-wiki usage. They are already loaded cross-wiki on a large scale! Take anyone's global.js. I'm wary that if gadget repositories are implemented as described here, cross-wiki usage will be left out of the picture.
• The ability to configure validation and minification. As I understand from reading T277675#9528940 by Krinkle, the validator already supports all ES versions. Meanwhile, it's not really critical for user JavaScript to be compatible with older browsers. So, some script authors could prefer to write in modern JS without transpiling and expect their users to follow.

I'm pondering a simpler, client-side, solution to the issue above with some magic inside mw.loader.getScript or other function that would return a promise that 1) would not resolve until the dependencies of the loaded script are ready and 2) would resolve with the script's export(s). (See the first code snippet above.)

As for user-level gadgets, among other things I see a potential security benefit in them. As I've demonstrated in T302769, we can successfully apply static analysis to find vulnerabilities in user JavaScript, and having it parsed into abstract syntax trees is the necessary first step in that direction.

The big problem I have with user scripts is loading dependencies and the incompatibility between gadgets and user scripts more generally.

I don't necessarily need my own gadgets-definition page or a way for people to add user scripts from the preferences, only a way to provide a list of dependencies and resource loader options for a script. A .gadget page would work for that, or even just some JSDoc or Greasemonkey-style comment headers with lines like @require mediawiki.api or @require [[User:Foo/bar.css]].

Not having a simple way to load dependencies is a huge pain. I end up with scripts with large blocks of embedded CSS and JSON because I can't easily put CSS, translations, configuration, etc, on a separate page. I end up copying and pasting helper functions between scripts because I can't easily load them from another page.

While it's possible to write something that works both as a gadget and a user script, it means not using any of the gadget features (since they can't be used in user scripts) and not using any modern Javascript features (since they can't or shouldn't be used in gadgets).

That makes creating and maintaining gadgets difficult: Either you make use of the resource loader options and ability to load multiple pages, but can't run a development version as a user script to test your changes, or you have a gadget that does all its own dependency handling (and probably puts everything on a single page to make it easier).

The proposed patch on Gerrit addresses precisely these problem - it introduces feature parity between gadgets and user scripts ("user gadgets"). All ResourceLoader features available to gadgets like loading dependencies, allowing multiple source pages, specifying peers for FOUC-free CSS loading, CommonJS module support, and conditional loading (based on namespaces, content models, skins, etc), would become available to 'user gadgets'.