Page MenuHomePhabricator

RFC: Sharing templates and modules between wikis - poor man's version (investigation)
Open, NormalPublic

Description

Problem

Templates and Modules need to be shared between wikis. It was requested ages ago, but mostly due to technical limitations (cache invalidation), it has not been implemented yet.

Proposal

Until it gets solved in a more fundamental way, we can already introduce a very easy solution that can be done without any changes to the server-side code. This way users can start experimenting with it, build useful templates, and figure out the social aspect of this feature and its limitations. And developers will see what functionality is actually needed for this to work well, and have an easy migration path once that functionality is in place. To do this, we introduce a bot that copies any pages marked with a {{AutoSync}} from the central wiki (Meta - which is the central OAuth wiki - or Commons, MW.org, etc) to any wiki listed in the corresponding wikidata item, and only if the target also has an {{AutoSync}} template. This approach solves the biggest issue - stale cache invalidation, because copying a template automatically triggers dependent page refresh on the local wiki.

For example, Template:GraphChart on commons would have a template at the top {{AutoSync}}, which would add the page to Category:AutoSync, and also show a message explaining what auto-sync is. It will also transclude template documentation from the doc sub-pages, e.g. /doc/en. Lastly, the GraphChart template will also be added to the Wikidata, where users will list all wikis that should receive a copy of this template. The auto-sync bot will go through the Category:AutoSync, and copy GraphChart to all those wikis. The bot will also copy documentation in the proper language as a /doc page, and insert an appropriate info text at the top, explaining that this page should not be edited because it will be overwritten.

How to add a new target

If a user decides to add a new wiki to the list or the template recipients, they would need to create a new page in the target wiki, containing only some magic template like {{AutoSync}} (or the local language variant). Once the page is created, they will also need to add that page to Wikidata as a local version of the original template. On the next bot run, the template will be auto-synced to the new page.

Sync pausing

If at some point users want to stop or pause page cloning for a local wiki, they can edit the page and change the info template at the top - adding a parameter "pause=explanation" - in which case the infobox will change and will contain a warning that this page is not being synced at the moment because of the given reason.

Permissions

Lastly, the bot could automatically set certain edit permissions on a template page in case there are too many pages that use it to prevent large-scale page regeneration vandalism.

Event Timeline

Yurik created this task.Dec 21 2015, 10:02 PM
Yurik updated the task description. (Show Details)
Yurik raised the priority of this task from to Needs Triage.
Yurik added subscribers: Yurik, kaldari.
Restricted Application added subscribers: StudiesWorld, Aklapper. · View Herald TranscriptDec 21 2015, 10:02 PM
Yurik updated the task description. (Show Details)Dec 21 2015, 10:04 PM
Yurik set Security to None.

A mistake and voilà! Hundreds of templates and modules scattered on dozens of wikis :(

Yurik updated the task description. (Show Details)Dec 21 2015, 10:20 PM

@Ricordisamoa, how is this different from making a mistake on a single wiki? Actually, how is this different from any solution that attempts to solve T121470? And yes, templates with high usages will need to be auto-switched to a protected mode, where only experienced users will be able to edit them.

TTO awarded a token.Dec 21 2015, 11:04 PM
TTO added a subscriber: TTO.

Community Tech has been given the mandate to do this properly. I'd rather see that, than a messy stopgap solution.

Yurik added a subscriber: Catrope.EditedDec 21 2015, 11:09 PM

@TTO, we have been trying to implement this for as long as I have been around, so it might be a while - its not a trivial problem by far. With this stopgap solution, we can offer this functionality right away, and let community start collaborating already, figuring out how it will structure this feature socially. In the mean time, we will do it properly. This approach will not preclude us from doing it properly - instead it will actually offer us a good testing ground on what functionality is needed.

For me, an important part of the shadow namespace proposal is that it's possible to override the template locally. The hypothetical bot shouldn't take over pre-existing templates just because someone marked the Commons one as global.

@matmarex - fully agree - please see the Sync pausing section above.

Yurik updated the task description. (Show Details)Dec 21 2015, 11:16 PM
Yurik updated the task description. (Show Details)Dec 21 2015, 11:25 PM
Yurik updated the task description. (Show Details)
Rical added a subscriber: Rical.Dec 21 2015, 11:51 PM
Meirae added a subscriber: Meirae.Dec 22 2015, 6:24 AM
Base added a subscriber: Base.Dec 22 2015, 5:34 PM
Base added a comment.Dec 22 2015, 5:47 PM

I'm in general fine with the nuclear plant, though something about i18n must be thought, of both comments in code and support of different ways to translate docs, e.g. using Extension:Translate and not with possibility to switch from one system to another (perhaps some configuring parameters in the global template's the function's template), but now to the parking: I would rather like the global templates to be on Meta: that's where global userpage, .js and .css already are -- users would need to spend less time remembering where the certain piece of code is to be inserted depending on what it is.

Qgil added a subscriber: Qgil.Dec 22 2015, 6:08 PM
Yurik added a comment.Dec 22 2015, 6:18 PM

@Base, i think we should keep the comments in code in english only - after all, code needs to be understood internationally, but the documentation should be translated. Yes, the extension:translate is how I was envisioning it, but I don't know how it works exactly, I only saw those mystical <!--1--> comments - any help is welcome. As for Meta vs Commons - hard to say - commons is a shared repository of resources like images and video, so it seems like commons is a better location for other shared content resources like templates, whereas meta is for the discussions about the projects, or about its users. So it makes sense to keep the user-related info on meta (after all, users info is meta data about the project, not the project content itself).

Having now worked on 2 "global transclusion" projects (GlobalCssJs and GlobalUserPage) where bots were used first to do it manually, I do not envy the person who gets to clean all of this up. But maybe we need something like this to make MediaWiki developers realize how terrible the current solution of bots is and be inspired to fix it properly.

Jay8g added a subscriber: Jay8g.Dec 23 2015, 4:31 AM
putnik added a subscriber: putnik.Dec 25 2015, 11:44 AM
Yurik added a comment.Dec 27 2015, 7:08 PM

@Legoktm, what issues did you see with it? I am all for fixing it "the right way". Also, I think that bots will give us a good way to experiment. Otherwise, we may implement something that does not solve the problem, or solve it in an inconvenient way.

This should be closed and merged to T91162.

For 2 years, I work on a centralisable module.
The main goal is to convert any module as centralisable in any language.
Then any wiki in any language can use any centralised module, only by translation of some sentences.
See the documentation (where categories are displaid in a language and they link in another) and the usual test page.
A sub-module and some libraries support the main centralised module.

@Rical, hi, its great that you are building globally i18n-able modules for all wikis. Were you building it on-wiki only, or is there some other magical code/script somewhere/copy it manually?

Rical added a comment.EditedJan 20 2016, 9:04 AM

The first module was Author3 the main infobox.
MathRoman2 is a small converter digital<->roman with errors detection, usable as main or sub module.
ControlArgs1 to support others.
Support is: translate, manage categories and messages, manage arguments from calling template.
Manage is: collect, translate in wiki and in user langage, display, catégorize.
Any module can select versions of its submodules in the list of all known.
Now I group all submodules in ControlArgs1 and I convert them in libraries.
Each of these modules exists in other versions to test the versioning.
There is none other code. About 10.000 lines of 100 signs.

DannyH renamed this task from RFC: Sharing templates and modules between wikis - poor man's version to RFC: Sharing templates and modules between wikis - poor man's version (investigation).Feb 4 2016, 6:21 PM
DannyH added a subscriber: DannyH.

This is related to one of the top 10 wishes on the wishlist survey: T121470: Central Global Repository for Templates, Lua modules, and Gadgets

Connunity Tech will do some investigation on this, and report back with our findings.

DannyH triaged this task as Normal priority.Feb 4 2016, 6:23 PM
DannyH moved this task from Untriaged to To be estimated/discussed on the Community-Tech board.
Rical added a comment.Feb 6 2016, 7:09 PM

After enough debug, the next task I think to post could be "Convert any module as centralizable".

This means that the implementation can run from only one very small main module, in one wiki with only 3 languages, until for lots of main modules in a central repository, for many wikis, in many languages.
The guideline to implement that follows these steps:

  • Finish to debug the Module:Central which supports the process, defines libraries and contains a small main module, to accelerate the development, for one month.
  • Describe the basic libraries, and their uses for anybody.
  • Share this implementation and invite other developpers to test, to evaluate, to better define and to optimize the guideline, the ways chose and more.
  • A live example of use shows several aspects of the result in a complex infobox based on 3 modules before the migration to libraries.
  • Of course, this is not easy in the small edit window of mediawiki. I change the files in local JEdit.

There is a description of the project writed on 2015-08-09, with some simple examples.

Rical added a comment.Feb 7 2016, 9:37 AM

Where to find good examples of Test_cases ?
1- for a simple testcase, 2- for a list of testcases defined in a table, 3- for a hierarchical tree of testcases.

He7d3r updated the task description. (Show Details)Feb 11 2016, 2:03 PM

For a "poor man's version", wouldn't it be better to use mw.loader.load() to dynamically pull Gadget JS and CSS from a central wiki? This is how HotCat and ProveIt are currently shared:

Seems cleaner than syncing with a bot (although there's no option to pause updates).

Yurik added a comment.Jan 9 2017, 6:52 AM

@kaldari, I don't think this is possible - you cannot pull templates this way. You need to get template wiki markup into the target wiki, and make parser aware of it as a regular page - otherwise it cannot be used by any other page. In other words, you can do "client-side" hacks, but not server-side transclusions across wikis.

On the other hand, you CAN already use my TNT module approach (see comments at the top) - because it is server side. It allows a template to be copied to another wiki WITHOUT any modifications because you can "internationalize it" - you can store all i18n messages in a table dataset on Commons, and only use message keys in the template itself (just like we do in MediaWiki itself). Parameters are also supported.

In theory, bigger parts of the template could be stored in the datasets on Commons, but this will be even more hacky :(

@Yurik: Sorry, I was thinking about global gadgets, not global templates. You're right.

Restricted Application added a subscriber: pywikibot-bugs-list. · View Herald TranscriptMar 23 2019, 10:16 PM
Yurik added a comment.Apr 12 2019, 2:44 AM

A bot has been implemented and documented for this functionality. Needs a whole bunch of bot approvals, or a global bot flag. For now running it by hand for a few pages.