Shadow namespaces at the 2016 Wikimedia Developer Summit
Closed, ResolvedPublic

Description

I'm splitting the WMDS-specific details from the parent task.

RfC: https://www.mediawiki.org/wiki/Requests_for_comment/Shadow_namespaces

Here's what I'd like to discuss at the developer summit for the Shadow namespaces RfC.

  1. At what layer should shadow namespaces be integrated into the code? Should we continue to subclass Article? How do we efficiently implement things like batch lookups? (T88644)
  1. How will we integrate remote content with search and other discovery mechanisms? We currently have Commons file results integrated with normal search results, but it's usefulness is questionable (T96535). Will it be possible to implement this for people using remote content via the API?
  1. How will we keep track of usage and invalidate caches? Should we just have a central link table like GlobalUsage? What about API users?
  1. Should we allow people to chain "repositories" like we currently allow with file repos?

The outcome I'd like to see is having a plan ready on how to migrate the foreign file system to a generic arbitrary namespace one, which should completely obsolete the GlobalUserPage extension.

Legoktm created this task.Oct 16 2015, 10:35 PM
Legoktm updated the task description. (Show Details)
Legoktm raised the priority of this task from to Needs Triage.
Legoktm claimed this task.
Legoktm added subscribers: Niharika, brion, Addshore and 12 others.
Isarra added a subscriber: Isarra.Oct 28 2015, 2:35 AM
Qgil added a comment.Oct 28 2015, 10:33 AM

Is this a Summit goal for this proposal that you agree with?

By the end of the Summit, we want to have the RfC resolved (aka a plan for the technical implementation) and also a task/plan to implement this feature assigned to a WMF team, as part of their FY2015-16 roadmap.

This would de-block the current situation, it would allow for a team to start finding the budget / resources for the implementation, and the future of this feature would not depend anymore on enthusiastic individuals having to lobby about it almost in their free time.

I keep wondering which team should in charge of implementing this feature, if only to associate their project to this task.

Restricted Application added a subscriber: StudiesWorld. · View Herald TranscriptNov 4 2015, 11:53 AM
Legoktm updated the task description. (Show Details)Nov 6 2015, 8:07 AM
Legoktm set Security to None.

Sorry about being super behind on planning this. I posted the updated task description to wikitech-l: https://lists.wikimedia.org/pipermail/wikitech-l/2015-November/083884.html

Is this a Summit goal for this proposal that you agree with?

By the end of the Summit, we want to have the RfC resolved (aka a plan for the technical implementation) and also a task/plan to implement this feature assigned to a WMF team, as part of their FY2015-16 roadmap.

I agree with the first part (and identified 4 specific areas that need to be addressed), but I don't think it's reasonable (for me at least) to commit to having a WMF team implement this. There are tentative plans for me to work on this in 2016, but nothing is confirmed, so I don't think it's appropriate to commit to it yet.

This would de-block the current situation, it would allow for a team to start finding the budget / resources for the implementation, and the future of this feature would not depend anymore on enthusiastic individuals having to lobby about it almost in their free time.

Even if no WMF team picks it up, having a clear technical plan of attack is going to be invaluable for moving forward. I don't see anything wrong with individuals getting stuff done in their free time, that's how most important stuff seems to get done these days anyways.

I don't think it's reasonable (for me at least) to commit to having a WMF team implement this. There are tentative plans for me to work on this in 2016, but nothing is confirmed, so I don't think it's appropriate to commit to it yet.

To be clear, I was not suggesting that you or someone else commits already to the plan. While you focus on the technical plan (where I cannot help), I will try to be helpful by bringing the attention of WMF product/engineering management.

The good thing about this proposal is that it is clearly a technical stepping stone that needs to be discussed and agreed in order to have any progress.

The bad thing is that, as the stepping stone it is, it may be difficult for "Product" to even understand the benefits, and it is definitely a too complex topic for most of the stakeholders to promote it directly (template users and developers, etc).

@DannyH, can we take you as main contact with Product? @kaldari, can we consider Community-Tech as another WMF stakeholder in addition to Developer-Relations?

This would de-block the current situation, it would allow for a team to start finding the budget / resources for the implementation, and the future of this feature would not depend anymore on enthusiastic individuals having to lobby about it almost in their free time.

Even if no WMF team picks it up, having a clear technical plan of attack is going to be invaluable for moving forward. I don't see anything wrong with individuals getting stuff done in their free time, that's how most important stuff seems to get done these days anyways.

Yes, agreed. You can just forget about quarterly goals, etc, but I will do my best making sure that this proposal is taken into account when planning for next quarters / next fiscal year. It cannot harm. :)

bd808 added a subscriber: bd808.Nov 17 2015, 11:08 PM
Tgr added a subscriber: Tgr.Nov 18 2015, 8:37 AM
Qgil added a comment.Dec 16 2015, 11:44 PM

Central Global Repository for Templates, Lua modules, and Gadgets has been the third most voted proposal at the Community Wishlist Survey. This is a very good display of community interest!

Central Global Repository for Templates, Lua modules, and Gadgets has been the third most voted proposal at the Community Wishlist Survey. This is a very good display of community interest!

Yup, agreed. I brought this up at T121470: Central Global Repository for Templates, Lua modules, and Gadgets. @Qgil (and @Krenair in T119593), thank you for highlighting this! @Legoktm, I think your list of questions is superb, and the right questions for large group discussions. No need to apologize for lack of planning!

I think the main bit of planning involved here is going to be about followup. I'm hoping someone in Community Tech is available and interested in helping out with this. @kaldari, @Fhocutt, @DannyH, and/or @NiharikaKohli: how will you need the outcome of the conversation that LegoKtm has laid out to go in order for it to be most useful for your post WikiDev work? Do you just need good notes based on the current set of questions, or are there specific (currently missing) questions that need to be discussed in order for the notes to be a useful document?

Rical added a subscriber: Rical.Dec 17 2015, 10:36 PM

@RobLa-WMF, @Legoktm: One other question we'll want to address is:

  • Is our current user permissions system adequate or will it need to be changed to deal with widespread cross-wiki dependencies?

@RobLa-WMF, @Legoktm: One other question we'll want to address is:

  • Is our current user permissions system adequate or will it need to be changed to deal with widespread cross-wiki dependencies?

I think the answer is definitely no, based on the existence of KrinkleBot/Commons:Auto-protected files.

However, for the summit I'd like to focus on generalizing the current foreign file system and merging in the GlobalUserPage feature set, which currently doesn't touch protection/rights.

Etherpad copy:


Topics for discussion:

  • Extending core namespaces so things like global user pages can be in core
  • Should we continue to subclass article
  • How will we integrate remote content with search and other discovery mechanisms? We currently have Commons file results integrated with normal search results, but it's usefulness is questionable (T96535). Will it be possible to implement this for people using remote content via the API?
  • How will we keep track of usage and invalidate caches? Should we just have a central link table like GlobalUsage? What about API users?
  • Should we allow people to chain "repositories" like we currently allow with file repos?

General notes

  • Batch lookups of foreign names very hacky right now (Title::isAlwaysKnown hacky)
  • What is this for? https://www.mediawiki.org/wiki/Requests_for_comment/Shadow_namespaces#Applications
  • Local parsing vs foreign parsing. How do parameters and different configs of foreign repos affect output
  • What about translatable content. For instance, we have a City infobox. The content is in one language, but how to show the labels in the local language? **Mostly TBD, but use user language for parsing (?). Also need to match translate extension with auto-rendering in user language.

*forward compatibility is a big ?

  • If a remote module calls i.e. a FlaggedRevs module and the local wki doesn't have it, what happens?
  • Versioning of shared templates. Should users be able to pin to a specific version. We haven't thought about that yet.
  • Search is also a problem. How not to replicate the search index in local wikis, making their search unusable.

We need a way to tracking usage. Who is using it (so we know when making changes). How do we do cache-invalidation.

  • If we do versioning, then maybe we don't need to rely in cache validation.

Proposal for dealing with this is to use pinned revisions (this revision of this page from this wiki), rather than just {{Foo}}.

Vandalism or accidental mistakes: how to avoid that a breaking change affects hundreds of wikis. Some kind of validation of changes before propagating?

Q (Amir): How much of it is already developed?
A: Most of the remote parsing exists, but needs to be generalized and make sure it fits in the architecture of Core.

ScaryTransclusion should be superseded by this feature.

Q: What is the opposition to this?
A: No opposition? Probably people wants to se an implementation.
A: We should be aware of i.e. telling en.wiki that they no longer rely on their local templates but need to maintain templates in a global namespace instead. (It is similar to Wikidata in a way.)
Maybe opt-in system. If people want it great, if not, then they don't have to.

End users using the templates shouldn't care about where it somes from. This should be transparent for them.

(Discussion on templates that shouldn't be part of content and controlled by editors but of UI and controlled via software.)

Q: Any idea of implementation process?
A: The plan is that legoktm will start working on this in April, in sync with the Community Tech team.

Make template sandbox work cross-wiki

Q: How do we decide which wikis we use as a source wiki?
A: Meta is the wiki for most of the existing WMF cluster-wide things; Commons is the only 'global' wiki so far; MediaWiki.org is the most common community of people that work on things for third party wikis; enwiki has the largest selection of existing templates for content etc..

Q (Amir): If a page is translatable, will the transculsion be able to pull the translated content on the targeted wiki in the appropriate language? (Relevant for any kind of page, but for Help in particular.)
A: Yes, it should do that, although it's not clear how at the moment. Special:MyLanguage is not quite right.

Q: How to augment global pages with local content?
A: Not good answer for that. Render the global template and append local content?

Nobody seems to object? In the communities, nobody is forced to change their current way of working. Opt-out is simple: don't use it.

In terms of community engagement, there is something to be learned from Wikidata adoption. This is similar, although more complex.

Possibly useful homework: Learn which templates are used in a lot of languages (many language links), and also the import log.

(Some potential problem with Tidy and 3rd party MediaWikis?)
Templates might have invalid HTML. On WMF we use Tidy to fix the html, on 3rd parties, usually html is not fixed, and wiki ends up looking totally broken

Maybe we will need something like Flagged Revisions, receiving a notification when a global template has changed and accepting the change or not?

JavaScript, another source of problems?

Action items with owners: (reflecting existing actions; no new actions came really out of this meeting)

  • Lego to get some answers to the questions
  • Lego will implement this feature, focusing on Templates
  • (Kaldari) Community Tech will research on which templates are already being copied across many wikis.
Rical added a comment.EditedJan 5 2016, 8:04 PM

The versionning could have 2 meanings. One is successive states of a page or a module.
Another is a group of modules with similar names and functions. Example: job, job01, job02.

The versionning of group management that I experiment is managed by a user, which defines in a main_module a list of sought_versions included in a list of known_versions.
Example: sought_versions = "Author3 MathRoman2 ControlArgs1"
known_versions = " Author,Author3 * MathRoman,MathRoman2 * ControlArgs,ControlArgs1 "
The user changes sought and known at module level or at Args level.
And a versions_library supports the module and the user.
The main_module can display messages like "MathRoman44 is not in known versions, normal MathRoman replaces it." or "ControlArgs345 is missing".

At the same time translations of messages linked to each of these modules are in their /I18N sub_modules.
The versions_library or the ControlArgs sub_module mixes translations from all active modules.
Also, to support users who help a foreign wiki, categories are displayed in user language and are linked to the categories in local wiki language.

Each /I18N sub_module can be local or central.
Each module version can be local or central.

Legoktm closed this task as Resolved.Jan 19 2016, 11:25 PM

Lego will implement this feature, focusing on Templates

To clarify, the first part of the shadow namespaces project is still going to be unifying the current implementations of global user pages / foreignfilerepo (remote parsing), and then we'll look at local parsing (Lua modules, templates). I don't know how "focusing on templates" got in that action item.

@Legoktm Are you going to create a project or a group of tasks that we can follow?

Rical removed a subscriber: Rical.Jul 21 2017, 9:40 AM