Page MenuHomePhabricator

Re-consider module paths for access to Produnto
Closed, ResolvedPublic

Description

I looked at the current implementation at https://produnto-test.toolforge.org/wiki/Module:Produnto_test and see it happening like this:

  • require( 'databox' ), require( 'produnto_test' )
  • mw.loadJsonData( 'produnto-test/src/test.json' )

I think calling any external data using this syntax is extremely bad for the following reasons:

  1. It breaks the established convention on how the module authors can know where the source of the pages is directly from their code. We do not write require( 'String' ) for standard modules (we use require( 'Module:String' )), we should not write require( 'produnto-test' ) for external modules.
  2. It creates potential confusion for module authors with existing shared modules like require( 'strict' ).
  3. It makes it impossible for extensions like SyntaxHighlight (and potentially MediaWiki-extensions-CodeMirror) to actually know where these modules are coming from and link to their sources for usability.

I think, before it’s too late, there should be some sort of prefix introduced for calling these external modules, like Package: from T425752: Produnto repository viewer: Package namespace.

Event Timeline

stjn updated the task description. (Show Details)

I believe we can use something like require( 'global>strict' ) or require( 'global/strict' ) where global is a customizable prefix. (Why customizable? Produnto uses $wgProduntoServers which in concept supports multiple servers, and we may want some way to differentiate packages that comes from different source. 3rd party wikis may want to use own package repo in addition to Wikimedia one.)

The main reason it's like this is for compatibility/consistency with LuaRocks. If you allow packages to use any module path, then code can work unmodified from the Lua ecosystem, you don't need to modify or transpile it. There are big packages like busted that could be imported. The submodules within these packages require each other using e.g.

local utils = require 'busted.utils'
local exit = require 'busted.compatibility'.exit
local loadstring = require 'busted.compatibility'.loadstring

and this will all just work if we allow packages to define those names.

In future, Produnto could even directly parse a rockspec file and allow users to install unmodified rocks.

Code written for Scribunto hasn't been part of the Lua ecosystem in the past, because Scribunto is so weird. Allowing top-level package names is a necessary step towards becoming less weird so that these ecosystems can be integrated.

It breaks the established convention on how the module authors can know where the source of the pages is directly from their code.

True. It's complexity, but it's an abstraction. It stops being the module author's problem to define where exactly the code comes from.

It creates potential confusion for module authors with existing shared modules like require( 'strict' ).

Just don't deploy a package that defines a module called strict.

And more broadly, we could require a module name prefix for all packages that are native to Scribunto/Produnto, just as a policy. It could be enforced in CI. The rationale for allowing top-level names applies only to third-party code.

It makes it impossible for extensions like SyntaxHighlight (and potentially MediaWiki-extensions-CodeMirror) to actually know where these modules are coming from and link to their sources for usability.

CodeMirror can ask Produnto for the link.

Note that the sources are linked in templatelinks, with module-level granularity, so we do have deep links on edit and preview already.

Change #1332664 had a related patch set uploaded (by Tim Starling; author: Tim Starling):

[mediawiki/extensions/Scribunto@master] Require Package: prefix when loading a JSON file from a package

https://gerrit.wikimedia.org/r/1332664

Requiring a Package: prefix in mw.loadJsonData() makes sense, there's no reason to not do that.

For require, I would suggest documenting module prefixing conventions at https://www.mediawiki.org/wiki/Help:Extension:Produnto

Code written for Scribunto hasn't been part of the Lua ecosystem in the past, because Scribunto is so weird. Allowing top-level package names is a necessary step towards becoming less weird so that these ecosystems can be integrated.

Is the idea then that code written in Produnto would not use any MW-specific variables or have access to basic MW-specific conveniences? Because if the answer is ‘no’, how is having another MW-specific abstraction to not confuse the authors and re-users bad?

True. It's complexity, but it's an abstraction. It stops being the module author's problem to define where exactly the code comes from.

I don’t understand at all why you see this as a good thing. Module authors don’t like having code come from a god knows where god knows how. While having backlinks from an extremely inconvenient ‘Templates used on the page’ list is good, that shouldn’t be the only affordance on the matter. It should be self-evident how to get to the source, not hidden behind a list. In the last few years we have moved towards that (with tools like CodeMirror providing better UX on how to get to a page or to a template than our 20+-year old software does), this seems to take us backwards for what seems to be purely theoretical re-use concerns.

Is the idea then that code written in Produnto would not use any MW-specific variables or have access to basic MW-specific conveniences?

That depends on the package.

I think we can allow module access by RepoViewer title like require "Package:package_name/src/foo.lua", as well as by dot-separated module name like require "package_name.foo".

Then just don't define any modules in the manifest if you want to access all your modules by path.

Change #1332841 had a related patch set uploaded (by Tim Starling; author: Tim Starling):

[mediawiki/extensions/Produnto@master] Runtime: Add a facility for loading files by title

https://gerrit.wikimedia.org/r/1332841

Change #1332842 had a related patch set uploaded (by Tim Starling; author: Tim Starling):

[mediawiki/extensions/Scribunto@master] Allow title-like strings with Package: prefix in require()

https://gerrit.wikimedia.org/r/1332842

Change #1332664 merged by jenkins-bot:

[mediawiki/extensions/Scribunto@master] Require Package: prefix when loading a JSON file from a package

https://gerrit.wikimedia.org/r/1332664

Change #1335410 had a related patch set uploaded (by SD0001; author: SD0001):

[mediawiki/extensions/SyntaxHighlight_GeSHi@master] Link to package namespaces pages in require() statements

https://gerrit.wikimedia.org/r/1335410

Change #1332841 merged by jenkins-bot:

[mediawiki/extensions/Produnto@master] Runtime: Add a facility for loading files by title

https://gerrit.wikimedia.org/r/1332841

Change #1332842 merged by jenkins-bot:

[mediawiki/extensions/Scribunto@master] Allow title-like strings with Package: prefix in require()

https://gerrit.wikimedia.org/r/1332842

Change #1335410 merged by jenkins-bot:

[mediawiki/extensions/SyntaxHighlight_GeSHi@master] Link to package namespaces pages in require() statements

https://gerrit.wikimedia.org/r/1335410

Change #1340801 had a related patch set uploaded (by Tim Starling; author: Tim Starling):

[mediawiki/extensions/Scribunto@wmf/1.47.0-wmf.19] Allow title-like strings with Package: prefix in require()

https://gerrit.wikimedia.org/r/1340801

Change #1340802 had a related patch set uploaded (by Tim Starling; author: Tim Starling):

[mediawiki/extensions/Produnto@wmf/1.47.0-wmf.19] Runtime: Add a facility for loading files by title

https://gerrit.wikimedia.org/r/1340802

Change #1340802 merged by jenkins-bot:

[mediawiki/extensions/Produnto@wmf/1.47.0-wmf.19] Runtime: Add a facility for loading files by title

https://gerrit.wikimedia.org/r/1340802

Change #1340801 merged by jenkins-bot:

[mediawiki/extensions/Scribunto@wmf/1.47.0-wmf.19] Allow title-like strings with Package: prefix in require()

https://gerrit.wikimedia.org/r/1340801

Mentioned in SAL (#wikimedia-operations) [2026-09-14T02:59:50Z] <tstarling@deploy1003> Started scap sync-world: Backport for [[gerrit:1340801|Allow title-like strings with Package: prefix in require() (T430644)]], [[gerrit:1340802|Runtime: Add a facility for loading files by title (T430644)]]

Mentioned in SAL (#wikimedia-operations) [2026-09-14T06:56:53Z] <tstarling@deploy1003> Started scap sync-world: Backport for [[gerrit:1340801|Allow title-like strings with Package: prefix in require() (T430644)]], [[gerrit:1340802|Runtime: Add a facility for loading files by title (T430644)]]

Mentioned in SAL (#wikimedia-operations) [2026-09-14T07:17:46Z] <tstarling@deploy1003> tstarling: Backport for [[gerrit:1340801|Allow title-like strings with Package: prefix in require() (T430644)]], [[gerrit:1340802|Runtime: Add a facility for loading files by title (T430644)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.

Mentioned in SAL (#wikimedia-operations) [2026-09-14T07:31:23Z] <tstarling@deploy1003> Finished scap sync-world: Backport for [[gerrit:1340801|Allow title-like strings with Package: prefix in require() (T430644)]], [[gerrit:1340802|Runtime: Add a facility for loading files by title (T430644)]] (duration: 34m 30s)

tstarling claimed this task.