Page MenuHomePhabricator

Central Global Repository for Templates, Lua modules, and Gadgets
Open, NormalPublic

Tokens
"Like" token, awarded by D3r1ck01."Like" token, awarded by Rical."Like" token, awarded by Billinghurst."Love" token, awarded by KuboF."Love" token, awarded by Liuxinyu970226."Love" token, awarded by Capankajsmilyo."Love" token, awarded by robkam."Like" token, awarded by MarcoAurelio."Like" token, awarded by daniel."Barnstar" token, awarded by Zppix."Love" token, awarded by Shizhao.
Assigned To
None
Authored By
DannyH, Dec 15 2015

Description

This card tracks a top 10 wish from the Community Wishlist Survey: https://meta.wikimedia.org/wiki/2015_Community_Wishlist_Survey

Original proposal: We could use a single location where to keep templates, Lua modules, and Gadgets that are used on all the wikipedia projects. Just like images from commons can be used on other projects, code from such site would be visible to all the projects. The current system of 100's of out of sync copies of the same templates or Lua modules occasionally synchronized with the original is very hard to maintain. --@Jarekt (talk) 20:06, 10 November 2015 (UTC)

Community Tech preliminary analysis:

Support: High. Strong support, but there are some important concerns expressed in the voting about the difficulty of using templates across wikis. Templates are already complicated, and this project could be offering a technical solution to a problem that needs social, consensus-based solutions.

Impact: High. Existing templates, gadgets and modules (across all wikis) will probably be affected.

Feasibility: Difficult. We're currently working (mid-January) on getting the Gadgets 2.0 Gadget Manager ready for review (see T31398). That doesn't include global gadgets, but it's a step in that direction. Creating a global repository will depend on using shadow namespaces -- there's currently an RfC about shadow namespaces on mediawiki.org. There are also notes and discussions from the Developer Summit in early January here: T115762.

Risk: High. Templates, gadgets, and modules are implemented differently and may need different solutions. Templates often transclude other templates. Adding translation support to templates will make them harder to maintain. We'll need to figure out how permissions will work, and how template and module editors can test on multiple wikis or provide a warning of changes. We need to also think about Gerrit-like code review needs for templates, gadgets and modules and possibly even a versioning system. It will be difficult to come up with a solution that will truly work across projects and will fit into the social structures of diverse communities.

Status: This work will probably depend on the shadow namespaces work, which is still being discussed, and probably won't be complete by the end of this year.

Project page: https://meta.wikimedia.org/wiki/Community_Tech/Central_repository_for_gadgets,_templates_and_Lua_modules

Related Objects

StatusAssignedTask
DuplicateNone
OpenNone
OpenNone
OpenNone
Declined Reguyla
ResolvedNemo_bis
OpenNone
OpenNone
DuplicateNone
StalledNone
OpenNone
OpenNone
ResolvedTTO
ResolvedNone
OpenNone
ResolvedNone
ResolvedNone
StalledNone
ResolvedLegoktm
OpenNone
DuplicateNone
ResolvedSamwilson
ResolvedNone
OpenNone
OpenNone
OpenRical
DeclinedNone
OpenNone
OpenNone
OpenNone
StalledNone
ResolvedNone
OpenNone
OpenNone
InvalidJackmcbarn
Openthiemowmde
DeclinedRical
OpenRical
OpenNone
OpenNone
OpenNone
OpenNone
OpenNone
OpenNone

Event Timeline

There are a very large number of changes, so older changes are hidden. Show Older Changes
robkam added a comment.Dec 9 2017, 7:55 PM

This should be treated as a MW bug, see T182504

I'm afraid it won't change a thing if it's treated as "a MW bug" or not. :)

CCicalese_WMF moved this task from Inbox to Watching on the MediaWiki-Platform-Team board.
Liuxinyu970226 removed a subscriber: Liuxinyu970226.

It's not totally clear to me how this task is distinct from older tasks such as T66475: Make crosswiki bits and pieces truly global (tracking). Is there something new here?

We started to move in this direction, for at least some use-cases, with https://www.mediawiki.org/wiki/Requests_for_comment/Shadow_namespaces#Applications.

Yurik added a comment.Mar 21 2018, 4:37 PM

@MZMcBride I think this task is the implementation of one aspect of T66475 . Also, I don't think shadow namespaces are a good approach, as discussed some time ago at a conference. Shadow implies local (per-language) override, whereas most of the time you don't want to override per language, you want to override per usecase, and that override may be useful for more than one language. To give an example, lets say you have an infobox that most wikis find useful, but it breaks in RTL wikis. You have two options - either fix the infobox so that it works on both (preferable, but might be very hard), or you could create a new infobox for RTL wikis, in which case it is useful to all RTL wikis, not just a single one. So instead of having a global + several local overrides, there should be two globals.

Also, I don't think shadow namespaces are a good approach, as discussed some time ago at a conference.

It depends on the use-case. Some of the past discussions focused on the difference between use-cases like Help pages v. User pages v. Template and Module pages. They may require different solutions, even though they can feel like similar problems.

You have two options - either fix the infobox so that it works on both (preferable, but might be very hard), or you could create a new infobox for RTL wikis, in which case it is useful to all RTL wikis, not just a single one. So instead of having a global + several local overrides, there should be two globals.

Two globals here could mean a central template repository with one page "Template:Infobox" and another page "Template:Infobox RTL", right? It does not necessarily mean multiple central template repositories.

Rical added a comment.Mar 21 2018, 5:36 PM

To try to solve the languages questions, I could propose a dedicated i18n table or even a strict rule.
In Central modules, see T135845, I use i18n tables to translate sentencies, which could be aproximative.
Here you probably need accurate translations in a dedicated i18n table.
My way to name modules in any wikis is:
Module:Central is for the central one.
Module:Central-s-fr is for fr.wikisource.org ... and so on.

Yurik added a comment.Mar 21 2018, 7:25 PM

Also, I don't think shadow namespaces are a good approach, as discussed some time ago at a conference.

It depends on the use-case. Some of the past discussions focused on the difference between use-cases like Help pages v. User pages v. Template and Module pages. They may require different solutions, even though they can feel like similar problems.

Correct, there is a clear "shadow" use case for localization, where the content is purely language-specific. I don't think Lua modules should have this feature at all. So lets not conflate two concepts - per-language separation (localization) and content structures (like templates and lua modules). The user page might be a special usecase, but it might be outside of the scope for this one.

You have two options - either fix the infobox so that it works on both (preferable, but might be very hard), or you could create a new infobox for RTL wikis, in which case it is useful to all RTL wikis, not just a single one. So instead of having a global + several local overrides, there should be two globals.

Two globals here could mean a central template repository with one page "Template:Infobox" and another page "Template:Infobox RTL", right? It does not necessarily mean multiple central template repositories.

Correct, that's what I meant - two templates in the same global repository.

Yurik added a comment.Mar 21 2018, 7:27 PM

@Rical take a look at the https://www.mediawiki.org/wiki/Module:TNT module -- it already allows one global translation table that can be used in any template anywhere. This allows a template or a module to be copy/pasted to any wiki without a single modification. Only requirement - the TNT module must be present on the wiki (many wikis already have it).

Rical added a comment.EditedMar 21 2018, 11:08 PM

@Yurik where to find an example of module already shared using TNT?
To see what include the module to share it. And for a template also. Thanks in advance.

@Rical take a look at Template:Graph:Lines -- that template's localized parameter documentation is stored in one place at the single location, and can be changed there without going to each wiki and copy/pasting things inside the complex template. Beyond that, the caption under the graph is also localized -- the string "See or edit raw graph data." is localized using the {{#invoke:TNT|msg|I18n/Template:Graphs.tab|source_table|{{#invoke:TNT|link|{{{table}}}}}}} (a bit cryptic, I know - it uses TNT twice, once to render the link, and once to render the whole message). The localized messages are actually stored here.

Using TNT from Lua modules should be very similar, and should be even easier to read than wiki markup.

Mvolz added a subscriber: Mvolz.Mar 26 2018, 4:51 PM
Izno added a subscriber: Izno.Jul 17 2018, 5:35 AM
KuboF awarded a token.Oct 26 2018, 8:18 AM
KuboF added a subscriber: KuboF.

This one seems like a popular demand, but not much has changed since I first saw this ticket. Can anyone please update on the status and whats the direction in which we are moving on this one?

Yurik added a comment.EditedNov 5 2018, 6:38 PM

@Capankajsmilyo AFAIK, WMF is not working on this. When I have some time, I will try to set up a bot to make this possible with the existing technology. A typical workflow:

  • A template or a module is created on mediawiki.org (MW is better because its community is more dev-focused, whereas Commons tends to be content-focused)
    • all localization strings are placed in the tabular data on Commons to simplify translation
    • template parameters are also placed in a tabular data on Commons
    • All strings are used via the Module:TNT
  • Some well known infobox is placed at the top of the template/module documentation to indicate that this module is shared between multiple wikis, and should not be changed anywhere else.
  • A bot looks for all modules/templates on MW.org that have that infobox, and copies them automatically to all other wikis using the sitelink list in Wikidata
  • If someone wants to use that template/module in a wiki X, they just need to copy/paste it to the wiki X, and add the sitelink - the bot will automatically keep it up to date from there on.
  • If a wiki decides to "fork" their version, they can simply remove the infobox from the docs.

@Capankajsmilyo AFAIK, WMF is not working on this. When I have some time, I will try to set up a bot to make this possible with the existing technology. A typical workflow:

  • A template or a module is created on mediawiki.org (MW is better because its community is more dev-focused, whereas Commons tends to be content-focused)
    • all localization strings are placed in the tabular data on Commons to simplify translation
    • template parameters are also placed in a tabular data on Commons
    • All strings are used via the Module:TNT
  • Some well known infobox is placed at the top of the template/module documentation to indicate that this module is shared between multiple wikis, and should not be changed anywhere else.
  • A bot looks for all modules/templates on MW.org that have that infobox, and copies them automatically to all other wikis using the sitelink list in Wikidata
  • If someone wants to use that template/module in a wiki X, they just need to copy/paste it to the wiki X, and add the sitelink - the bot will automatically keep it up to date from there on.
  • If a wiki decides to "fork" their version, they can simply remove the infobox from the docs.

Seems like a good solution for now. Nice thought.

Qgil removed a subscriber: Qgil.Nov 5 2018, 7:58 PM
Elitre removed a subscriber: Elitre.Nov 8 2018, 4:43 PM
Rical awarded a token.Feb 11 2019, 8:55 AM
D3r1ck01 added a subscriber: D3r1ck01.
daniel changed the status of subtask T91162: RFC: Shadow namespaces from Open to Stalled.Mar 28 2019, 10:57 AM

What's taking this so long? Has any decision been made on how to implement this? There are a lot of wikis like sawiki which don't have enough tech contributors. They will be more than happy to adopt it as compared to frwiki which has a huge army of tech people. We can start with allowing sawiki to use modules of enwiki from mediawiki directly. Any comments?

Yurik added a comment.EditedApr 10 2019, 2:03 AM

It's not about "allowing", it's just a matter of me (or someone else) to sit down and implement it :) Afterwards, I will have to document it in detail, and apply for a bot flag for my YurikBot (it has been dormant for a while now), preferably to be allowed on all wikis at once.

P.S. This only applies to my bot-idea above, not the general MW-core supported functionality that would take far longer to implement, and require much more collaboration.

How can I help you? I really need such a functionality at the earliest.

daniel added a comment.EditedApr 10 2019, 9:22 AM

What's taking this so long? Has any decision been made on how to implement this? There are a lot of wikis like sawiki which don't have enough tech contributors. They will be more than happy to adopt it as compared to frwiki which has a huge army of tech people. We can start with allowing sawiki to use modules of enwiki from mediawiki directly. Any comments?

Cross-wiki code sharing is difficult, for security reasons (immediate deployment on 1 thousand websites, code gets run in the browsers of millions of people, with no code review) but also for because it requires the code to use proper internationalization mechanisms, which are currently not in place or not easy to use or simply not used.

There is so far no agreement on how exactly this should work. I mean what exactly it should do. Here is the summary from a discussion we had at the Wikimedia Technical Conference last year:

  • Making it easier to localize gadgets. Descoped from "tools".
  • As a volunteer, I want to reuse gadgets without rebuilding them, in my own project and my own language.
  • Research with community needed, on how it would work for them, and how governance would work.
  • Need to create a prototype in which we decide where gadgets would be stored and where their definitions will be
  • Determine if that fits all use-cases, and then build prototype.
  • Build gadget repository, perhaps searchable?
  • Update most/all gadgets to use it.
  • At the same time, deal with translations issues -- where to store them, how to allow as many as possible to edit, try to standardize use by gadgets
  • Resources: 1-1.5 years to work with community engagement. 6months of Eng time.
  • Concerns: perhaps not big enough for WMF approval. Resourcing perhaps from Platform Evolution or CommTech Wishlist.

See https://www.mediawiki.org/wiki/Wikimedia_Technical_Conference/2018/Session_notes/Working_together_to_develop_our_roadmap. @TheDJ was leading the discussion, perhaps he has some thoughts. @Tgr wrote a prototype for maintaining Gadgets on git a while ago. I really like that idea.

daniel added a subscriber: TheDJ.Apr 10 2019, 9:26 AM
daniel added a subscriber: Tgr.

What's taking this so long? Has any decision been made on how to implement this? There are a lot of wikis like sawiki which don't have enough tech contributors. They will be more than happy to adopt it as compared to frwiki which has a huge army of tech people. We can start with allowing sawiki to use modules of enwiki from mediawiki directly. Any comments?

Cross-wiki code sharing is difficult, for security reasons (immediate deployment on 1 thousand websites, code gets run in the browsers of millions of people, with no code review) but also for because it requires the code to use proper internationalization mechanisms, which are currently not in place or not easy to use or simply not used.
There is so far no agreement on how exactly this should work. I mean what exactly it should do. Here is the summary from a discussion we had at the Wikimedia Technical Conference last year:

  • Making it easier to localize gadgets. Descoped from "tools".
  • As a volunteer, I want to reuse gadgets without rebuilding them, in my own project and my own language.
  • Research with community needed, on how it would work for them, and how governance would work.
  • Need to create a prototype in which we decide where gadgets would be stored and where their definitions will be
  • Determine if that fits all use-cases, and then build prototype.
  • Build gadget repository, perhaps searchable?
  • Update most/all gadgets to use it.
  • At the same time, deal with translations issues -- where to store them, how to allow as many as possible to edit, try to standardize use by gadgets
  • Resources: 1-1.5 years to work with community engagement. 6months of Eng time.
  • Concerns: perhaps not big enough for WMF approval. Resourcing perhaps from Platform Evolution or CommTech Wishlist.

See https://www.mediawiki.org/wiki/Wikimedia_Technical_Conference/2018/Session_notes/Working_together_to_develop_our_roadmap. @TheDJ was leading the discussion, perhaps he has some thoughts. @Tgr wrote a prototype for maintaining Gadgets on git a while ago. I really like that idea.

Most of your points are about gadgets. What about modules? What about templates? What about infoboxes? Where is the discussion? It has been 4 years since this was proposed. Is it stalled? Is it being worked upon? Or is it simply ignored for unknown reasons? There's nothing clear in status for this.

TheDJ added a comment.Apr 10 2019, 9:47 AM

@Capankajsmilyo When there is no patch and no planning mentioned, you can assume that it is only being discussed at this time. There is no fixed planning to work on this (As there isn't for most of our 20 000 open tickets).

@Capankajsmilyo When there is no patch and no planning mentioned, you can assume that it is only being discussed at this time. There is no fixed planning to work on this (As there isn't for most of our 20 000 open tickets).

In that case, can we please think of something to resolve such a huge backlog or atleast bring such backlog to attention of developers here? Which are the top 10 tasks being done? Which are top 100? Which are top 1000?

Keeping it as a black well where task is generated and is left forever, might not be of much help.

Internationalization issues, cultural aspects, script and language, world wide impact goes in similar way for templates and modules.

As long as very low level functionality, utilized by programmers only to build other templates on top, global basic software may be offered. In Lua this is possible today by libraries, some are used now.

As soon it is some kind of text or figures or graphics visible to a reader, that global stuff is not easy since that needs to be adopted to many many requests and formatting details by local culture and even community policy.

Sounds pretty: There is a big bag of global things, and a local project has nothing to work on and may use it without taking care of anything. But this is not how it works in reality. Just displaying a number is a challenge, even more a number with measuring unit, which needs a lot of adaption per project.

Internationalization issues, cultural aspects, script and language, world wide impact goes in similar way for templates and modules.
As long as very low level functionality, utilized by programmers only to build other templates on top, global basic software may be offered. In Lua this is possible today by libraries, some are used now.
As soon it is some kind of text or figures or graphics visible to a reader, that global stuff is not easy since that needs to be adopted to many many requests and formatting details by local culture and even community policy.
Sounds pretty: There is a big bag of global things, and a local project has nothing to work on and may use it without taking care of anything. But this is not how it works in reality. Just displaying a number is a challenge, even more a number with measuring unit, which needs a lot of adaption per project.

If that is the case, I guess this issue deserves a dedicated discussion. I can't even find a list of issues being faced while completing this project. The task says "Status: This work will probably depend on the shadow namespaces work, which is still being discussed, and probably won't be complete by the end of this year." It does not even mention which year. There is no guidance on how to go about it. How can it be achieved in such a manner?

TheDJ added a comment.EditedApr 10 2019, 10:00 AM

atleast bring such backlog to attention of developers here?

I think it needed to be brought to YOUR attention. The rest of us here are pretty much aware of this ;) Remember, this is a non-profit, aided by a few volunteers. There is no resourcing as with some other major commercial websites.

Anything that isn't part of the annual planning of the Wikimedia foundation is not scheduled and funded and depends on volunteers to work on (even though some of those volunteers are WMF employees). That's almost all the tickets which are thus not resourced and funded and everything has a gazillion dependencies, short and longterm ideas and/or required skillsets. It is difficult to present a 'top 100' for that reason.

atleast bring such backlog to attention of developers here?

I think it needed to be brought to YOUR attention. The rest of us here are pretty much aware of this ;) Remember, this is a non-profit, aided by a few volunteers. There is no resourcing as with some other major commercial websites.
Anything that isn't part of the annual planning of the Wikimedia foundation is not scheduled and funded and depends on volunteers to work on (even though some of those volunteers are WMF employees). That's almost all the tickets which are thus not resourced and funded and everything has a gazillion dependencies, short and longterm ideas and/or required skillsets. It is difficult to present a 'top 100' for that reason.

Has there been any discussion on the need for a top 100 list ever then? Can you please refer me to the same? If not, can such a discussion be started? If yes, how?

greg removed a subscriber: greg.Apr 10 2019, 6:48 PM
Yurik added a comment.Apr 12 2019, 2:46 AM

A bot has been implemented and documented for this functionality. Needs a whole bunch of bot approvals, or a global bot flag. For now running it by hand for a few pages. Anyone interested in this functionality, we can now start building cross-site templates and lua modules...

Please do not post to this ticket, instead lets discuss it on the project's talk page.