Page MenuHomePhabricator

RFC: Sharing templates and modules between wikis - poor man's version (investigation)
Open, MediumPublic

Description

Problem

Templates and Modules need to be shared between wikis. It was requested ages ago, but mostly due to technical limitations (cache invalidation), it has not been implemented yet.

Proposal

Until it gets solved in a more fundamental way, we can already introduce a very easy solution that can be done without any changes to the server-side code. This way users can start experimenting with it, build useful templates, and figure out the social aspect of this feature and its limitations. And developers will see what functionality is actually needed for this to work well, and have an easy migration path once that functionality is in place. To do this, we introduce a bot that copies any pages marked with a {{AutoSync}} from the central wiki (Meta - which is the central OAuth wiki - or Commons, MW.org, etc) to any wiki listed in the corresponding wikidata item, and only if the target also has an {{AutoSync}} template. This approach solves the biggest issue - stale cache invalidation, because copying a template automatically triggers dependent page refresh on the local wiki.

For example, Template:GraphChart on commons would have a template at the top {{AutoSync}}, which would add the page to Category:AutoSync, and also show a message explaining what auto-sync is. It will also transclude template documentation from the doc sub-pages, e.g. /doc/en. Lastly, the GraphChart template will also be added to the Wikidata, where users will list all wikis that should receive a copy of this template. The auto-sync bot will go through the Category:AutoSync, and copy GraphChart to all those wikis. The bot will also copy documentation in the proper language as a /doc page, and insert an appropriate info text at the top, explaining that this page should not be edited because it will be overwritten.

How to add a new target

If a user decides to add a new wiki to the list or the template recipients, they would need to create a new page in the target wiki, containing only some magic template like {{AutoSync}} (or the local language variant). Once the page is created, they will also need to add that page to Wikidata as a local version of the original template. On the next bot run, the template will be auto-synced to the new page.

Sync pausing

If at some point users want to stop or pause page cloning for a local wiki, they can edit the page and change the info template at the top - adding a parameter "pause=explanation" - in which case the infobox will change and will contain a warning that this page is not being synced at the moment because of the given reason.

Permissions

Lastly, the bot could automatically set certain edit permissions on a template page in case there are too many pages that use it to prevent large-scale page regeneration vandalism.

Event Timeline

There are a very large number of changes, so older changes are hidden. Show Older Changes

A mistake and voilà! Hundreds of templates and modules scattered on dozens of wikis :(

Yurik updated the task description. (Show Details)Dec 21 2015, 10:20 PM

@Ricordisamoa, how is this different from making a mistake on a single wiki? Actually, how is this different from any solution that attempts to solve T121470? And yes, templates with high usages will need to be auto-switched to a protected mode, where only experienced users will be able to edit them.

TTO awarded a token.Dec 21 2015, 11:04 PM
TTO added a subscriber: TTO.

Community Tech has been given the mandate to do this properly. I'd rather see that, than a messy stopgap solution.

Yurik added a subscriber: Catrope.EditedDec 21 2015, 11:09 PM

@TTO, we have been trying to implement this for as long as I have been around, so it might be a while - its not a trivial problem by far. With this stopgap solution, we can offer this functionality right away, and let community start collaborating already, figuring out how it will structure this feature socially. In the mean time, we will do it properly. This approach will not preclude us from doing it properly - instead it will actually offer us a good testing ground on what functionality is needed.

For me, an important part of the shadow namespace proposal is that it's possible to override the template locally. The hypothetical bot shouldn't take over pre-existing templates just because someone marked the Commons one as global.

@matmarex - fully agree - please see the Sync pausing section above.

Yurik updated the task description. (Show Details)Dec 21 2015, 11:16 PM
Yurik updated the task description. (Show Details)Dec 21 2015, 11:25 PM
Yurik updated the task description. (Show Details)
Rical added a subscriber: Rical.Dec 21 2015, 11:51 PM
Meirae added a subscriber: Meirae.Dec 22 2015, 6:24 AM
Base added a subscriber: Base.Dec 22 2015, 5:34 PM
Base added a comment.Dec 22 2015, 5:47 PM

I'm in general fine with the nuclear plant, though something about i18n must be thought, of both comments in code and support of different ways to translate docs, e.g. using Extension:Translate and not with possibility to switch from one system to another (perhaps some configuring parameters in the global template's the function's template), but now to the parking: I would rather like the global templates to be on Meta: that's where global userpage, .js and .css already are -- users would need to spend less time remembering where the certain piece of code is to be inserted depending on what it is.

Qgil added a subscriber: Qgil.Dec 22 2015, 6:08 PM
Yurik added a comment.Dec 22 2015, 6:18 PM

@Base, i think we should keep the comments in code in english only - after all, code needs to be understood internationally, but the documentation should be translated. Yes, the extension:translate is how I was envisioning it, but I don't know how it works exactly, I only saw those mystical <!--1--> comments - any help is welcome. As for Meta vs Commons - hard to say - commons is a shared repository of resources like images and video, so it seems like commons is a better location for other shared content resources like templates, whereas meta is for the discussions about the projects, or about its users. So it makes sense to keep the user-related info on meta (after all, users info is meta data about the project, not the project content itself).

Having now worked on 2 "global transclusion" projects (GlobalCssJs and GlobalUserPage) where bots were used first to do it manually, I do not envy the person who gets to clean all of this up. But maybe we need something like this to make MediaWiki developers realize how terrible the current solution of bots is and be inspired to fix it properly.

Jay8g added a subscriber: Jay8g.Dec 23 2015, 4:31 AM
putnik added a subscriber: putnik.Dec 25 2015, 11:44 AM
Yurik added a comment.Dec 27 2015, 7:08 PM

@Legoktm, what issues did you see with it? I am all for fixing it "the right way". Also, I think that bots will give us a good way to experiment. Otherwise, we may implement something that does not solve the problem, or solve it in an inconvenient way.

This should be closed and merged to T91162.

For 2 years, I work on a centralisable module.
The main goal is to convert any module as centralisable in any language.
Then any wiki in any language can use any centralised module, only by translation of some sentences.
See the documentation (where categories are displaid in a language and they link in another) and the usual test page.
A sub-module and some libraries support the main centralised module.

@Rical, hi, its great that you are building globally i18n-able modules for all wikis. Were you building it on-wiki only, or is there some other magical code/script somewhere/copy it manually?

Rical added a comment.EditedJan 20 2016, 9:04 AM

The first module was Author3 the main infobox.
MathRoman2 is a small converter digital<->roman with errors detection, usable as main or sub module.
ControlArgs1 to support others.
Support is: translate, manage categories and messages, manage arguments from calling template.
Manage is: collect, translate in wiki and in user langage, display, catégorize.
Any module can select versions of its submodules in the list of all known.
Now I group all submodules in ControlArgs1 and I convert them in libraries.
Each of these modules exists in other versions to test the versioning.
There is none other code. About 10.000 lines of 100 signs.

DannyH renamed this task from RFC: Sharing templates and modules between wikis - poor man's version to RFC: Sharing templates and modules between wikis - poor man's version (investigation).Feb 4 2016, 6:21 PM
DannyH added a subscriber: DannyH.

This is related to one of the top 10 wishes on the wishlist survey: T121470: Central Global Repository for Templates, Lua modules, and Gadgets

Connunity Tech will do some investigation on this, and report back with our findings.

DannyH triaged this task as Medium priority.Feb 4 2016, 6:23 PM
Rical added a comment.Feb 6 2016, 7:09 PM

After enough debug, the next task I think to post could be "Convert any module as centralizable".

This means that the implementation can run from only one very small main module, in one wiki with only 3 languages, until for lots of main modules in a central repository, for many wikis, in many languages.
The guideline to implement that follows these steps:

  • Finish to debug the Module:Central which supports the process, defines libraries and contains a small main module, to accelerate the development, for one month.
  • Describe the basic libraries, and their uses for anybody.
  • Share this implementation and invite other developpers to test, to evaluate, to better define and to optimize the guideline, the ways chose and more.
  • A live example of use shows several aspects of the result in a complex infobox based on 3 modules before the migration to libraries.
  • Of course, this is not easy in the small edit window of mediawiki. I change the files in local JEdit.

There is a description of the project writed on 2015-08-09, with some simple examples.

Rical added a comment.Feb 7 2016, 9:37 AM

Where to find good examples of Test_cases ?
1- for a simple testcase, 2- for a list of testcases defined in a table, 3- for a hierarchical tree of testcases.

He7d3r updated the task description. (Show Details)Feb 11 2016, 2:03 PM

For a "poor man's version", wouldn't it be better to use mw.loader.load() to dynamically pull Gadget JS and CSS from a central wiki? This is how HotCat and ProveIt are currently shared:

Seems cleaner than syncing with a bot (although there's no option to pause updates).

Yurik added a comment.Jan 9 2017, 6:52 AM

@kaldari, I don't think this is possible - you cannot pull templates this way. You need to get template wiki markup into the target wiki, and make parser aware of it as a regular page - otherwise it cannot be used by any other page. In other words, you can do "client-side" hacks, but not server-side transclusions across wikis.

On the other hand, you CAN already use my TNT module approach (see comments at the top) - because it is server side. It allows a template to be copied to another wiki WITHOUT any modifications because you can "internationalize it" - you can store all i18n messages in a table dataset on Commons, and only use message keys in the template itself (just like we do in MediaWiki itself). Parameters are also supported.

In theory, bigger parts of the template could be stored in the datasets on Commons, but this will be even more hacky :(

@Yurik: Sorry, I was thinking about global gadgets, not global templates. You're right.

Restricted Application added a subscriber: pywikibot-bugs-list. · View Herald TranscriptMar 23 2019, 10:16 PM
Yurik added a comment.Apr 12 2019, 2:44 AM

A bot has been implemented and documented for this functionality. Needs a whole bunch of bot approvals, or a global bot flag. For now running it by hand for a few pages.

MJL added a subscriber: MJL.Jul 5 2019, 6:48 PM
Viztor added a subscriber: Viztor.Aug 4 2019, 11:18 PM
Pppery awarded a token.Apr 4 2020, 1:40 AM
Tgr added a subscriber: Tgr.Apr 13 2020, 11:25 PM

I proposed building on this in T250065#6051620.

A bot has been implemented and documented for this functionality. Needs a whole bunch of bot approvals, or a global bot flag. For now running it by hand for a few pages.

It seems the bot stopped working in 2019 May. mw:Module:TNT for example is still documented as being synchronized by it, but it actually isn't. Was it a one-off proof of concept effort you are not maintaining anymore, or is there a different reason?

Qgil removed a subscriber: Qgil.Apr 28 2020, 1:32 PM
Yurik added a comment.EditedApr 28 2020, 7:35 PM

@Tgr I just ran it a bit more, but the issue is that the bot would need a bot flag with admin rights (not possible globally, so one would have to apply to every wiki... painful). The bot source code is in https://github.com/nyurik/dibabel

I think the next step, as discussed with some members of the ruwiki (cc: @Ghuron), is to implement a web tool instead of a bot:

  • this web tool would work as an automated helper under the account of a specific user
  • the tool would show a list of all "auto-sync - enabled" modules and templates, and show all the changes that need to happen
  • the user will be able to click on the "sync" button for each page/site, allowing identical change to happen on multiple wikis in one go.

This will remove the need for a flag, will give the power of editing to the community, and will allow much more visual inspection of editing. Plus if the template/module is locked because it is widely used, only approved editors will be able to do the sync for those sites.

Tgr added a comment.Apr 28 2020, 9:07 PM

An opt-in model for wikis does not seem particularly painful to me. Wikis need to actively choose using those modules in their articles (or other templates/modules), there's no value in them being there otherwise. Compared to that a bot approval process doesn't seem like a huge extra effort. (Also, why admin? Modules can be edited by anyone, and most wikis don't regularly protect them.)

A web tool would make sense, but it needs to be written, the bot already exists and it's trivial to switch from one to the other later. So is there any reason no to use the bot with an opt-in wiki list right now? Or is that too much effort to administer?

Yurik added a subscriber: Ghuron.Apr 28 2020, 9:47 PM
Yurik added a comment.Apr 28 2020, 9:51 PM

Most wikis do want to protect highly used templates/modules. E.g. the Module:TNT would be used by most pages - you never want to make it editable by novice users. Thus, the bot would need to have the rights to edit that page.

Another big issue is simply applying for the bot flag to 500 wikis. If that can be somehow automated...? :)

I could already automate the distribution of the content only to wikis that have granted bot user a bot flag... I guess that would be a good indication that the bot is welcomed there... I am just not sure where i should run it (its been a while since i set up any automation processes on tools or wmflabs)

The bot recently got approvals in Kn, vec and te. I have applied on some more wikis for approvals.

Tgr added a comment.Apr 29 2020, 1:57 PM

Most wikis do want to protect highly used templates/modules.

Seems like most medium/large wikis do that but not for many templates (150 wikis, about 7 templates per wiki on average, not counting semiprotection which wouldn't affect a bot). So that doesn't seem like a huge obstacle. The numbers exclude template protection (apparently the stats backend does not track it correctly: T251411: page_restrictions field incomplete in current and historical dumps) but not that many wikis use that, I think.

Another big issue is simply applying for the bot flag to 500 wikis. If that can be somehow automated...? :)

With a global bot flag. The current policy is limited to interwiki bots, but generalizing that into bots syncing global change seems like a reasonable change.

I could already automate the distribution of the content only to wikis that have granted bot user a bot flag...

Or have an opt-in process via some config page like MediaWiki:SharedTemplates.json which the bot parses before touching that wiki.

matmarex removed a subscriber: matmarex.Apr 29 2020, 5:41 PM
Dvorapa added a subscriber: Dvorapa.May 6 2020, 4:52 PM

It seems the bot stopped working in 2019 May. mw:Module:TNT for example is still documented as being synchronized by it, but it actually isn't. Was it a one-off proof of concept effort you are not maintaining anymore, or is there a different reason?

Have it been restarted? On Czech Wikipedia there are some issues with importing Covid-19 related templates from enwiki and also there is a proposal to share some basic templates (like Ping) from Czech Wikipedia to Czech Wikibooks, Wikisource, etc...

Tgr added a comment.May 11 2020, 8:14 PM

I think the next step, as discussed with some members of the ruwiki (cc: @Ghuron), is to implement a web tool instead of a bot

Filed as T252453: Create a Toolforge tool for syncing templates and Lua modules between wikis.

Iniquity moved this task from Backlog to RFC on the Tech-Ambassadors board.
Iniquity added a subscriber: Iniquity.
Yurik added a comment.Aug 25 2020, 6:56 AM

There is now a tool for this... https://dibabel.toolforge.org/

I think this can be closed?

There is now a tool for this... https://dibabel.toolforge.org/

Looks great! @Yurik, will you send a mass message to all communities to inform about this tool?

Yurik added a comment.Aug 25 2020, 1:57 PM

@Candalua there is a number of bugs and features I would like to fix/add before advertising it further. For example, the service does not refresh page state too often enough, it does not show dependency status (e.g. if one module depends on another which is stale, it doesn't warn about it). Everyone is welcome to use it though - shouldn't cause any major issues :)