Page MenuHomePhabricator

Central Global Repository for Templates, Lua modules, and Gadgets
Open, MediumPublic

Subscribers
Tokens
"Like" token, awarded by D3r1ck01."Like" token, awarded by Rical."Like" token, awarded by Billinghurst."Love" token, awarded by KuboF."Love" token, awarded by Liuxinyu970226."Love" token, awarded by Capankajsmilyo."Love" token, awarded by robkam."Like" token, awarded by MarcoAurelio."Like" token, awarded by daniel."Barnstar" token, awarded by Zppix."Love" token, awarded by Shizhao.
Assigned To
None
Authored By
DannyH, Dec 15 2015

Description

This card tracks a top 10 wish from the Community Wishlist Survey: https://meta.wikimedia.org/wiki/2015_Community_Wishlist_Survey

Original proposal: We could use a single location where to keep templates, Lua modules, and Gadgets that are used on all the wikipedia projects. Just like images from commons can be used on other projects, code from such site would be visible to all the projects. The current system of 100's of out of sync copies of the same templates or Lua modules occasionally synchronized with the original is very hard to maintain. --@Jarekt (talk) 20:06, 10 November 2015 (UTC)

Community Tech preliminary analysis:

Support: High. Strong support, but there are some important concerns expressed in the voting about the difficulty of using templates across wikis. Templates are already complicated, and this project could be offering a technical solution to a problem that needs social, consensus-based solutions.

Impact: High. Existing templates, gadgets and modules (across all wikis) will probably be affected.

Feasibility: Difficult. We're currently working (mid-January) on getting the Gadgets 2.0 Gadget Manager ready for review (see T31398). That doesn't include global gadgets, but it's a step in that direction. Creating a global repository will depend on using shadow namespaces -- there's currently an RfC about shadow namespaces on mediawiki.org. There are also notes and discussions from the Developer Summit in early January here: T115762.

Risk: High. Templates, gadgets, and modules are implemented differently and may need different solutions. Templates often transclude other templates. Adding translation support to templates will make them harder to maintain. We'll need to figure out how permissions will work, and how template and module editors can test on multiple wikis or provide a warning of changes. We need to also think about Gerrit-like code review needs for templates, gadgets and modules and possibly even a versioning system. It will be difficult to come up with a solution that will truly work across projects and will fit into the social structures of diverse communities.

Status: This work will probably depend on the shadow namespaces work, which is still being discussed, and probably won't be complete by the end of this year.

Project page: https://meta.wikimedia.org/wiki/Community_Tech/Central_repository_for_gadgets,_templates_and_Lua_modules

Related Objects

StatusAssignedTask
OpenNone
OpenNone
OpenNone
OpenNone
ResolvedSamwilson
OpenNone
ResolvedNone
StalledNone
OpenNone
StalledNone
ResolvedLegoktm
OpenNone
OpenNone
ResolvedTTO
OpenNone
ResolvedNone
ResolvedNone
OpenNone
OpenNone
OpenNone
ResolvedWhatamidoing-WMF
OpenNone
OpenNone
OpenRical
DeclinedNone
OpenNone
OpenNone
OpenNone
StalledNone
ResolvedNone
OpenNone
OpenNone
InvalidJackmcbarn
OpenNone
DeclinedRical
OpenRical
OpenNone
OpenNone
OpenNone
OpenNone
OpenNone
OpenNone
OpenNone

Event Timeline

There are a very large number of changes, so older changes are hidden. Show Older Changes
Izno added a subscriber: Izno.Jul 17 2018, 5:35 AM
KuboF awarded a token.Oct 26 2018, 8:18 AM
KuboF added a subscriber: KuboF.

This one seems like a popular demand, but not much has changed since I first saw this ticket. Can anyone please update on the status and whats the direction in which we are moving on this one?

Yurik added a comment.EditedNov 5 2018, 6:38 PM

@Capankajsmilyo AFAIK, WMF is not working on this. When I have some time, I will try to set up a bot to make this possible with the existing technology. A typical workflow:

  • A template or a module is created on mediawiki.org (MW is better because its community is more dev-focused, whereas Commons tends to be content-focused)
    • all localization strings are placed in the tabular data on Commons to simplify translation
    • template parameters are also placed in a tabular data on Commons
    • All strings are used via the Module:TNT
  • Some well known infobox is placed at the top of the template/module documentation to indicate that this module is shared between multiple wikis, and should not be changed anywhere else.
  • A bot looks for all modules/templates on MW.org that have that infobox, and copies them automatically to all other wikis using the sitelink list in Wikidata
  • If someone wants to use that template/module in a wiki X, they just need to copy/paste it to the wiki X, and add the sitelink - the bot will automatically keep it up to date from there on.
  • If a wiki decides to "fork" their version, they can simply remove the infobox from the docs.

@Capankajsmilyo AFAIK, WMF is not working on this. When I have some time, I will try to set up a bot to make this possible with the existing technology. A typical workflow:

  • A template or a module is created on mediawiki.org (MW is better because its community is more dev-focused, whereas Commons tends to be content-focused)
    • all localization strings are placed in the tabular data on Commons to simplify translation
    • template parameters are also placed in a tabular data on Commons
    • All strings are used via the Module:TNT
  • Some well known infobox is placed at the top of the template/module documentation to indicate that this module is shared between multiple wikis, and should not be changed anywhere else.
  • A bot looks for all modules/templates on MW.org that have that infobox, and copies them automatically to all other wikis using the sitelink list in Wikidata
  • If someone wants to use that template/module in a wiki X, they just need to copy/paste it to the wiki X, and add the sitelink - the bot will automatically keep it up to date from there on.
  • If a wiki decides to "fork" their version, they can simply remove the infobox from the docs.

Seems like a good solution for now. Nice thought.

Qgil removed a subscriber: Qgil.Nov 5 2018, 7:58 PM
Elitre removed a subscriber: Elitre.Nov 8 2018, 4:43 PM
Rical awarded a token.Feb 11 2019, 8:55 AM
D3r1ck01 added a subscriber: D3r1ck01.
daniel changed the status of subtask T91162: RFC: Shadow namespaces from Open to Stalled.Mar 28 2019, 10:57 AM

What's taking this so long? Has any decision been made on how to implement this? There are a lot of wikis like sawiki which don't have enough tech contributors. They will be more than happy to adopt it as compared to frwiki which has a huge army of tech people. We can start with allowing sawiki to use modules of enwiki from mediawiki directly. Any comments?

Yurik added a comment.EditedApr 10 2019, 2:03 AM

It's not about "allowing", it's just a matter of me (or someone else) to sit down and implement it :) Afterwards, I will have to document it in detail, and apply for a bot flag for my YurikBot (it has been dormant for a while now), preferably to be allowed on all wikis at once.

P.S. This only applies to my bot-idea above, not the general MW-core supported functionality that would take far longer to implement, and require much more collaboration.

How can I help you? I really need such a functionality at the earliest.

daniel added a comment.EditedApr 10 2019, 9:22 AM

What's taking this so long? Has any decision been made on how to implement this? There are a lot of wikis like sawiki which don't have enough tech contributors. They will be more than happy to adopt it as compared to frwiki which has a huge army of tech people. We can start with allowing sawiki to use modules of enwiki from mediawiki directly. Any comments?

Cross-wiki code sharing is difficult, for security reasons (immediate deployment on 1 thousand websites, code gets run in the browsers of millions of people, with no code review) but also for because it requires the code to use proper internationalization mechanisms, which are currently not in place or not easy to use or simply not used.

There is so far no agreement on how exactly this should work. I mean what exactly it should do. Here is the summary from a discussion we had at the Wikimedia Technical Conference last year:

  • Making it easier to localize gadgets. Descoped from "tools".
  • As a volunteer, I want to reuse gadgets without rebuilding them, in my own project and my own language.
  • Research with community needed, on how it would work for them, and how governance would work.
  • Need to create a prototype in which we decide where gadgets would be stored and where their definitions will be
  • Determine if that fits all use-cases, and then build prototype.
  • Build gadget repository, perhaps searchable?
  • Update most/all gadgets to use it.
  • At the same time, deal with translations issues -- where to store them, how to allow as many as possible to edit, try to standardize use by gadgets
  • Resources: 1-1.5 years to work with community engagement. 6months of Eng time.
  • Concerns: perhaps not big enough for WMF approval. Resourcing perhaps from Platform Evolution or CommTech Wishlist.

See https://www.mediawiki.org/wiki/Wikimedia_Technical_Conference/2018/Session_notes/Working_together_to_develop_our_roadmap. @TheDJ was leading the discussion, perhaps he has some thoughts. @Tgr wrote a prototype for maintaining Gadgets on git a while ago. I really like that idea.

daniel added a subscriber: TheDJ.Apr 10 2019, 9:26 AM
daniel added a subscriber: Tgr.

What's taking this so long? Has any decision been made on how to implement this? There are a lot of wikis like sawiki which don't have enough tech contributors. They will be more than happy to adopt it as compared to frwiki which has a huge army of tech people. We can start with allowing sawiki to use modules of enwiki from mediawiki directly. Any comments?

Cross-wiki code sharing is difficult, for security reasons (immediate deployment on 1 thousand websites, code gets run in the browsers of millions of people, with no code review) but also for because it requires the code to use proper internationalization mechanisms, which are currently not in place or not easy to use or simply not used.
There is so far no agreement on how exactly this should work. I mean what exactly it should do. Here is the summary from a discussion we had at the Wikimedia Technical Conference last year:

  • Making it easier to localize gadgets. Descoped from "tools".
  • As a volunteer, I want to reuse gadgets without rebuilding them, in my own project and my own language.
  • Research with community needed, on how it would work for them, and how governance would work.
  • Need to create a prototype in which we decide where gadgets would be stored and where their definitions will be
  • Determine if that fits all use-cases, and then build prototype.
  • Build gadget repository, perhaps searchable?
  • Update most/all gadgets to use it.
  • At the same time, deal with translations issues -- where to store them, how to allow as many as possible to edit, try to standardize use by gadgets
  • Resources: 1-1.5 years to work with community engagement. 6months of Eng time.
  • Concerns: perhaps not big enough for WMF approval. Resourcing perhaps from Platform Evolution or CommTech Wishlist.

See https://www.mediawiki.org/wiki/Wikimedia_Technical_Conference/2018/Session_notes/Working_together_to_develop_our_roadmap. @TheDJ was leading the discussion, perhaps he has some thoughts. @Tgr wrote a prototype for maintaining Gadgets on git a while ago. I really like that idea.

Most of your points are about gadgets. What about modules? What about templates? What about infoboxes? Where is the discussion? It has been 4 years since this was proposed. Is it stalled? Is it being worked upon? Or is it simply ignored for unknown reasons? There's nothing clear in status for this.

TheDJ added a comment.Apr 10 2019, 9:47 AM

@Capankajsmilyo When there is no patch and no planning mentioned, you can assume that it is only being discussed at this time. There is no fixed planning to work on this (As there isn't for most of our 20 000 open tickets).

@Capankajsmilyo When there is no patch and no planning mentioned, you can assume that it is only being discussed at this time. There is no fixed planning to work on this (As there isn't for most of our 20 000 open tickets).

In that case, can we please think of something to resolve such a huge backlog or atleast bring such backlog to attention of developers here? Which are the top 10 tasks being done? Which are top 100? Which are top 1000?

Keeping it as a black well where task is generated and is left forever, might not be of much help.

Internationalization issues, cultural aspects, script and language, world wide impact goes in similar way for templates and modules.

As long as very low level functionality, utilized by programmers only to build other templates on top, global basic software may be offered. In Lua this is possible today by libraries, some are used now.

As soon it is some kind of text or figures or graphics visible to a reader, that global stuff is not easy since that needs to be adopted to many many requests and formatting details by local culture and even community policy.

Sounds pretty: There is a big bag of global things, and a local project has nothing to work on and may use it without taking care of anything. But this is not how it works in reality. Just displaying a number is a challenge, even more a number with measuring unit, which needs a lot of adaption per project.

Internationalization issues, cultural aspects, script and language, world wide impact goes in similar way for templates and modules.
As long as very low level functionality, utilized by programmers only to build other templates on top, global basic software may be offered. In Lua this is possible today by libraries, some are used now.
As soon it is some kind of text or figures or graphics visible to a reader, that global stuff is not easy since that needs to be adopted to many many requests and formatting details by local culture and even community policy.
Sounds pretty: There is a big bag of global things, and a local project has nothing to work on and may use it without taking care of anything. But this is not how it works in reality. Just displaying a number is a challenge, even more a number with measuring unit, which needs a lot of adaption per project.

If that is the case, I guess this issue deserves a dedicated discussion. I can't even find a list of issues being faced while completing this project. The task says "Status: This work will probably depend on the shadow namespaces work, which is still being discussed, and probably won't be complete by the end of this year." It does not even mention which year. There is no guidance on how to go about it. How can it be achieved in such a manner?

TheDJ added a comment.EditedApr 10 2019, 10:00 AM

atleast bring such backlog to attention of developers here?

I think it needed to be brought to YOUR attention. The rest of us here are pretty much aware of this ;) Remember, this is a non-profit, aided by a few volunteers. There is no resourcing as with some other major commercial websites.

Anything that isn't part of the annual planning of the Wikimedia foundation is not scheduled and funded and depends on volunteers to work on (even though some of those volunteers are WMF employees). That's almost all the tickets which are thus not resourced and funded and everything has a gazillion dependencies, short and longterm ideas and/or required skillsets. It is difficult to present a 'top 100' for that reason.

atleast bring such backlog to attention of developers here?

I think it needed to be brought to YOUR attention. The rest of us here are pretty much aware of this ;) Remember, this is a non-profit, aided by a few volunteers. There is no resourcing as with some other major commercial websites.
Anything that isn't part of the annual planning of the Wikimedia foundation is not scheduled and funded and depends on volunteers to work on (even though some of those volunteers are WMF employees). That's almost all the tickets which are thus not resourced and funded and everything has a gazillion dependencies, short and longterm ideas and/or required skillsets. It is difficult to present a 'top 100' for that reason.

Has there been any discussion on the need for a top 100 list ever then? Can you please refer me to the same? If not, can such a discussion be started? If yes, how?

greg removed a subscriber: greg.Apr 10 2019, 6:48 PM
Yurik added a comment.Apr 12 2019, 2:46 AM

A bot has been implemented and documented for this functionality. Needs a whole bunch of bot approvals, or a global bot flag. For now running it by hand for a few pages. Anyone interested in this functionality, we can now start building cross-site templates and lua modules...

Please do not post to this ticket, instead lets discuss it on the project's talk page.

Amire80 added a comment.EditedOct 13 2019, 12:59 PM

As a pet project I wrote something like a requirements doc for this from a user perspective (less PHP classes, storage and caching; more user experience description). And it's only for templates and modules, not for gadgets.

Short version: https://www.mediawiki.org/wiki/User:Amire80/Global_templates_draft_spec/TLDR

Long version: https://www.mediawiki.org/wiki/User:Amire80/Global_templates_draft_spec

Yupik added a subscriber: Yupik.Nov 3 2019, 1:28 PM
Sophivorus added a comment.EditedNov 4 2019, 9:10 PM

I've been reading around this issue and doing some tests in my private wiki, and I fail to understand the reasons for not implementing centralized templates (not talking about gadgets or modules in this comment) using $wgEnableScaryTransclusion

  • Is it because of cache issues? I think we users would be much happier going around manually purging templates and pages, than copying and updating wikitext again and again as we currently are. It would definitely be easier, less time consuming and we would eventually come up with a bot for it (as was suggested in T122086). Plus, what's the worst that could happen? Slightly outdated templates?
  • Is it because of localization issues? I'm 99% sure we can sort this out with /i18n.json subpages or a similar in-wiki solution, either at the central wiki or at the periphery wikis, plus some special-purpose module or template for getting the messages. Again, much easier to do and maintain than the current situation.
  • Is it because of performance issues? Do those computational resources cost more than the volunteer time that is costing us to not centralize templates?

Or is there some other reason I haven't considered? Or maybe I'm missing some key fact? I would really appreciate someone telling me.

This issue has been quite a drag for years and is a frequent community request. Are the risks and downsides of enabling $wgEnableScaryTransclusion serious enough to justify more years of wasted volunteer time?

Tgr added a comment.Tue, Nov 5, 8:29 PM

I've been reading around this issue and doing some tests in my private wiki, and I fail to understand the reasons for not implementing centralized templates (...) using $wgEnableScaryTransclusion

From that link:

This will display the results of the wikicode on your wiki. Templates function to the extent required pages exist locally: i.e. where the template displays a link that doesn't exist on the local wiki, the link will be red, where the template includes other templates nested within its code, local templates by those names will be called, and unless the required templates exist in the required form, the template will be broken.

@Tgr That's only with "raw" transclusions. Subtemplates are not required to exist locally when "raw" is not used. But even if using "raw" were necessary, we'd just create the required subtemplates using "raw", and so on. Still fail to understand the issue.

Tgr added a comment.Wed, Nov 6, 12:09 AM

Without raw you get back a HTML blob rendered by a different parser, there is no end to the exciting bugs that could cause (local interface changes would get ignored, for example; links would be blue or red based on whether they exist on the templatecommons wiki). If you prefix all template names with raw:central_wiki_name, they won't work on the central wiki itself (I guess that would be not super hard to fix). It will also break cache invalidation and maybe performance, plus Parsoid would have to be updated to support it. Functionally, this is more or less what the shadow namespace proposal was aiming at (but it was a wishlist item and wasn't finished within the year, so I imagine there must be much more complexity to it than I can see at a glance).

Sophivorus added a comment.EditedWed, Nov 6, 12:50 AM

@Tgr Thanks for taking time off to explain some of the difficulties! I guess all I wanted to say is "perfect is the enemy of good". Whether it's shadow namespaces, bots or scary transclusions, at this point I think any half-working solution would be better than the current situation. Cheers!

Tgr added a comment.Wed, Nov 6, 1:07 AM

Well, a bot has been written with very much the same mindset.

Amire80 added a comment.EditedWed, Nov 13, 10:23 PM

I'm trying to clean up the tasks around this topic. It's time for another round of merging duplicates.

My proposal is:

  1. This one (T121470) should probably remain as the main reference point for global templates, modules, and gadgets, simply because it appears to be linked to from so many places (correct me if I'm wrong). Duplicates:
  2. There should be one subtask for global gadgets. Probably T22153: Implement global gadgets (WMF-wide)
    1. Duplicates:
      1. T153263: Allow easier downloading of userscripts/gadgets (from enwiki and such)
    2. Subtasks:
      1. T117540: Introduce global registry for gadget module identifiers
  3. There should be one subtask for global templates and modules, because they are mostly similar (but see below). It can be T52329: We need a common repository for Scribunto modules and templates.
  4. The common task for templates and modules should have further subtasks for things that different for templates and for modules. Examples:
    1. Modules:
      1. T135845: Convert any module as central or centralisable
      2. T41610: Scribunto should support global module invocations
    2. Templates:
      1. T3126: Interwiki templates
      2. T6547: Support crosswiki template inclusion (transclusion => interwiki templates, etc.)
  5. I'm not sure what to do with even more generic tasks, such as these:
    1. T66475: Make crosswiki bits and pieces truly global (tracking) - This task (T121470) can probably be its subtask.
    2. T11890: Reasonably efficient interwiki transclusion
    3. T91162: RFC: Shadow namespaces - this should probably a blocker for T121470 if @Legoktm and other relevant people still think that it's the way forward as the internal technical solution.

This list is probably not complete, but it's a beginning.

Furthermore, I suggest to decline the tasks that ask for global user scripts. We already have global personal Common.js, and (hacky) cross-wiki loading of user scripts. So it's enough to have global gadgets at some future point, and when this becomes possible, if anyone wants to make a user script global, it won't be too hard to convert it into a gadget and make it global.

Nikki removed a subscriber: Nikki.Fri, Nov 15, 4:51 PM
Amire80 moved this task from Backlog to Transclusions on the Crosswiki board.