Page MenuHomePhabricator

Load .json configuration files via ResourceLoaderWikiModule
Closed, ResolvedPublic

Description

We are trying to limit JS editing to as few accounts as possible to reduce the likelihood of XSS compromises (T190015); configuration should be moved to JSON files to remain widely available for editing. There is no way currently to load a JSON page with ResourceLoader though; loading with AJAX has suboptimal performance. There should be a more efficient way to load such files.

One concrete use case is the geonotice config on enwiki; this should be moved to JSON but loading it blocks banner display so performance is key.

Current AJAX-based method

$.getJSON( mw.util.getUrl( 'MediaWiki:Citoid-template-type-map.json', { action: 'raw' } ) ).then( function ( data ) {
  console.log( 'The data is:', data );
} );

Event Timeline

There are a very large number of changes, so older changes are hidden. Show Older Changes

As another example where this would be helpful: in Russian Wikipedia sysops and bureaucrats frequently edit a gadget for showing users with advanced groups so that it would stay up-to-date. If there won’t be a more comfortable solution such as JSON, quite a lot people might argue either for workarounds that undermine security (such as content model change?) or for giving those rights to them by default when they don’t really edit any other pages.

Two quick thoughts:

  • Interface: We'll need a find a transport method for the JSON file, and a way for the JS code to read its contents. It's probably best to build this on top of T133462, which would provide a built-in way for multiple files to be exported by a module (rather than concatenated). The read method could be something like require('./foo.json').
  • Edit rights: While inclusion of JSON files seems useful, I do not think we should have separate edit rights for the JS/CSS file of a gadget and the JSON files.

As another example where this would be helpful: in Russian Wikipedia sysops and bureaucrats frequently edit MediaWiki:Gadget-markadmins.js.

You might be interested in the bot that Wikimedia Commons uses for MediaWiki:Gadget-markAdmins-data.js.

You might be interested in the bot that Wikimedia Commons uses for MediaWiki:Gadget-markAdmins-data.js.

It is a viable solution, but there are some user groups for us that are present only in the name, like ArbCom members and clerks.

  • Edit rights: While inclusion of JSON files seems useful, I do not think we should have separate edit rights for the JS/CSS file of a gadget and the JSON files.

JS/CSS is dangerous in the hands of a malicious user, JSON not really. For Gadgets 2.0, not sure if there is any use case for different people editing the JSON files and the rest; current permissions are just how MediaWiki: pages are handled, and I think it's reasonable to make JSON editing available to more users there, as it tends to be used for bot control and such (not necessarily in gadgets).

Translation might be a use case for wider access, although T156210: Support translation of JSON blobs in Translate is the nicer solution there.

Right now many people who ask for interface-admin (see T190015) permissions in ruwiki are worried about the ability to edit gadgets data. But with the current state where the integration of JSON with gadgets is poor, the ability to load external JSON for gadgets is very limited. Hence, the efforts to integrate JSON with ResourceLoader as soon as possible are essential from the perspective of the interface-admin flag purpose.

For 99% of gadgets ResourceLoader integration makes little difference. geonotice is unusually performance-sensitive but something like the admin coloring gadget is not. Just loading it via a plain API query with a short smaxage should be fine.

Right now we have at least 3 widely used gadgets that would need such query, probably there will be more. Having 5–10 additional requests doesn't seem to make negligable difference. Many such gadgets also add elements that shift content on pages which in turn causes misclicks, so the earlier they run the better. Anyway, is it so difficult to implement? I thought splitting large JS files into modules according to the semantics of different code parts and loading them, together with other modules, in one package is exactly what ResourceLoader is for.

Krinkle changed the task status from Open to Stalled.Aug 2 2018, 4:19 PM

Please remember that I have accepted the feature request, but this will not be implemented in the short-term while other projects take priority.

Regarding the efficiency issues, nobody is being forced to use a JSON page fetched over AJAX. In fact, I would recommend against it for exactly the reasons you state. Fetching it would be inefficient, makes the code harder to maintain, and would render more slowly.

So, for today - use a JS page instead. That is the current best practice, and is supported by ResourceLoader. Place the data on a JS page (not a JSON page). They work the same way, in the same namespace, with the same protection level. The JSON would be prepended by a simple statement, such as MyGadget.data = { }, MyGadget.setData({ }) or mw.hook('myGadget.data').fire({ }). ResourceLoader, through Gadgets, can already combine these into a single bundle - together with the rest of the gadget - in a single request.

Anyway, is it so difficult to implement?

Concatenating the plain text as saved onto a .js page and a .json page together would be easy, but the result would be invalid syntax. It will not work the way you want. To do this, we need to build a communication channel between load.php (server-side) and mw.loader (client-side) for sending this data, and then we need to build a way for your gadget JS code to find the data.

Task T133462 is about building that channel (for multiple chunks of JS code). Once that exists, we can also use that channel for JSON data. To find the data, we will build a require() function that will work similar to what many web developers have seen in other environments (Node.js, AMD, UMD, etc.). This makes it easy to learn from tutorials that explain JavaScript in general (instead of new things to learn just for Gadgets on Wikipedia).

the efforts to integrate JSON with ResourceLoader as soon as possible are essential from the perspective of the interface-admin flag purpose.

Sorry, but I do not understand the link between JSON integration and interface-admin.

We currently have a single group ("sysop") that grants the ability to protect pages, block users, and modify and deploy software (Gadgets) saved to JS and CSS pages. This has many downsides: Communities have to entrust a user with all of it, or nothing. Hackers compromising an admin account will be capable of all the above, even if the admin only intended to use one or the other.

Separating these groups, has benifits:

  • Reduce security risk: Users will only ask the rights they wanted. Community stays in control of granting the rights.
  • Communities can give sysop rights without worrying about cybersecurity.
  • Communities can give interface-admin rights without worrying about deletion/blocking policy and trusted activity in that area. I have seen several times that very good volunteer developers was active as software developer, but not as general editor/content reviewer. This makes it difficult to become sysop.

They work the same way, in the same namespace, with the same protection level.

JS requires editsitejs which is soon going to be restricted to interface-admin. JSON requires editsitejson which sysops will keep (and it has a separate grant for OAuth that does not involve sitewide JS editing permissions).

@Krinkle Thanks for the detailed explanation.

@Tgr Yes, that's the link between JSON integration and interface-admin that I'm talking about. Right now we have 1) gadgets data on JS pages, 2) sysops actively editing some of this data. So, people ask for interface-admin flag that they could do without if the gadgets data was on JSON pages.

I guess you could attempt a horrible hack where you change the content model of a JS page to wikitext and transclude the JSON into it (and wrap with a variable assignment). No idea if it would actually work though...

en.wp has proposed to have a bot sync from an admin-protected page. Obviously, we need to make sure there is consensus, but once we establish that, I suspect that we may be able to run it on other wikis.

@Izno Please note, we think making a bot to sync json pages to js pages is far from ideal, and having this be able to work server-side in a way that "data" can be edited outside of "code" would be much preferred. For anyone following you can check on the bot build progress here: https://en.wikipedia.org/wiki/Wikipedia:Bots/Requests_for_approval/MusikBot_II_2

Krinkle changed the task status from Stalled to Open.Feb 19 2019, 9:59 PM
Krinkle edited projects, added Performance-Team (Radar); removed Performance-Team.

(No longer blocked/stalled per T133462. ) Moving to radar as this isn't currently resourced through a specific product need.

I don't know if this is useful, but I wrote a javascript function ("get_JSON" at https://en.wikipedia.org/wiki/User:DannyS712_test/JSON.js) to parse a JSON page and return a JSON object (as opposed to plain text).

@DannyS712, additionally to previous comment, there is a simple way to get raw page content - https://en.wikipedia.org/w/index.php?title=PAGE_TITLE&action=raw.

<s>Well nevermind then</s>

those don't appear to work on foreign wikis due to constraints about cross-wiki requests

Krinkle updated the task description. (Show Details)

Re-framing specifically about gadgets and RL support in the WikiModule class.

For other site/user scripts, this is unlikely to happen as it would require additional modules to be registered and has imho too little added value over action=raw imports. Note that if/when user scripts themselves can be a proper module, with a definition somewhere, then this would naturally work by basically being a "personal gadget".

Change 731283 had a related patch set uploaded (by SD0001; author: SD0001):

[mediawiki/core@master] Support JSON files in ResourceLoaderWikiModule

https://gerrit.wikimedia.org/r/731283

Change 731284 had a related patch set uploaded (by SD0001; author: SD0001):

[mediawiki/extensions/Gadgets@master] Support JSON files in gadgets

https://gerrit.wikimedia.org/r/731284

Change 731283 merged by jenkins-bot:

[mediawiki/core@master] resourceloader: Add support for JSON pages in ResourceLoaderWikiModule

https://gerrit.wikimedia.org/r/731283

Change 749207 had a related patch set uploaded (by SD0001; author: SD0001):

[mediawiki/core@master] resourceloader: Disable supportsURLLoading() for packaged wiki modules

https://gerrit.wikimedia.org/r/749207

Change 749207 merged by jenkins-bot:

[mediawiki/core@master] resourceloader: Disable supportsURLLoading() for packaged wiki modules

https://gerrit.wikimedia.org/r/749207

Notes on testing the patch:

Add gadget definition in MediaWiki:Gadgets-definition:

*json-test [ ResourceLoader | default | package ] | json-test.js | json-test.json

Create MediaWiki:Gadget-json-test.js:

console.log('in json-test.js');
var data = require('./json-test.json');
console.log(data);

Create MediaWiki:Gadget-json-test.json:

{ "key": "value", "key2": "value2"}

You should now see the json data showing up in the console.

reminder, we need to update the various documentation pages.

Change 731284 merged by jenkins-bot:

[mediawiki/extensions/Gadgets@master] Support JSON files in gadgets

https://gerrit.wikimedia.org/r/731284

This might be worth announcing in Tech News, something along the lines of:

"Gadgets will soon be able to have json files as explicit dependencies. This change allows basic configuration to be modified by administrators without the interface administrator permission, such as with the Geonotice gadget."

So, what's the right page name, mydata.json or Gadget-mydata.json as usual?

Might be a good idea to mention somewhere (in Tech News item?) that gadget writers should exercise caution and make sure not to use strings from these pages as unescaped HTML.

Now that this lucky day is near (great credit to SD0001), I want to remind that, with this patch, MediaWiki makes a small step to reduce the attack surface involving Interface admins. After data is moved to .json that is bundled with the gadget, people who hold this flag solely due to a need to edit some data stored in JS will no longer need it and can have their permissions revoked with no worries.

  1. Tech News entry - It currently says this in https://meta.wikimedia.org/wiki/Tech/News/2022/02 Please edit it, if needed, within the next 24 hours. (p.s. Thanks to Izno for drafting and JJMC89 for adding!)

Gadgets are now able to have JSON files as explicit dependencies. This change allows basic configuration to be modified by administrators without the interface administrator permission, such as with the Geonotice gadget.

  1. Docs link - I would like to link that Tech News entry to some specific (and translatable) docs, so that people can find more info. I am not sure which of these 3 (or other?) pages to link to. Please advise?: (p.s. Thanks to @TheDJ for updating the first two!)
  1. Docs updates - I see 2 questions above from IKhitron and stjn that possibly need to be answered, and the answers documented in the appropriate wikipage(s). Please could someone help with that?

I think I already get an answer from these links. Although, I coudn't make all this work yet.

Well, FYI. It works now, and the problem was misleading documentation of the new future. It's better to rewrite it, I think. To tell about file names, about try and catch, and more.

Well, FYI. It works now, and the problem was misleading documentation of the new future. It's better to rewrite it, I think. To tell about file names, about try and catch, and more.

Go right ahead.

  1. Docs link - I would like to link that Tech News entry to some specific (and translatable) docs, so that people can find more info. I am not sure which of these 3 (or other?) pages to link to. Please advise?: (p.s. Thanks to @TheDJ for updating the first two!)

I'd go for this one.

  1. Docs updates - I see 2 questions above from IKhitron and stjn that possibly need to be answered, and the answers documented in the appropriate wikipage(s). Please could someone help with that?

So, what's the right page name, mydata.json or Gadget-mydata.json as usual?

Page names are Gadget-mydata.json. But Gadget- is a prefix (almost a namespace) that the gadget system itself doesn't see. So in the gadget definitions and when using require() you refer to that page as mydata.json. This is all described on https://www.mediawiki.org/wiki/ResourceLoader/Migration_guide_(users)#Package_Gadgets and https://www.mediawiki.org/wiki/Extension:Gadgets

Might be a good idea to mention somewhere (in Tech News item?) that gadget writers should exercise caution and make sure not to use strings from these pages as unescaped HTML.

Something like: Interface admins should review that all data used from these json files is properly escaped where needed by the gadget.