WIP RFC: Hygienic transclusions for WYSIWYG, incremental parsing & composition: Options and trade-offs
Open, NormalPublic

Description

This is a task to explore and summarize the trade-offs inherent in different possible paths towards hygienic transclusions.

Problem statement / motivation

Transclusions can currently affect arbitrary parts of the page by producing unbalanced HTML. This causes problems:

  • Editing: Visual editing of templates is unreliable and not truly WYSIWYG. The ergonomics of editing typical multi-template content are relatively poor in the wikitext editor, and even worse in VisualEditor.
  • Performance: Unpredictable side effects prevent many important optimizations in the parser, which means that users need to wait longer than necessary, and more CPU cycles, energy and money are spent.
  • Composability: In order to better adapt the user experience to different devices, network conditions and use cases, we are interested in composing content dynamically. Doing this efficiently requires the definition of clear components, and limits on how a component can affect its surrounding content. Current transclusions don't satisfy these requirements.

Requirements

  • Transclusions result in a well-defined DOM forest.
    • Single-rooted trees would be even more efficient to match.
  • Transclusions do not affect surrounding content.
    • Corollary: Transclusion scopes are stable across transclusion re-expansions.
  • Minimal breakage of existing content. Remaining issues ideally fixable with template changes, not transclusion changes.
  • Simple mental model, predictable behavior.
  • Ideally, works consistently for old & new revisions.

Ideas

The three main approaches currently discussed are:

a) Opt-in: No balancing by default, special syntax for explicitly requesting balancing of specific transclusions.
b) Opt-out: All transclusions are balanced by default, special syntax can be used to explicitly widen the balancing scope to support multi-transclusion content with unbalanced parts.
c) Inference: All transclusions are balanced by default, but statistics and templatedata are used to identify templates that are normally unbalanced, as well as the normal end tags. DOM scopes are established by typical end templates or tags.

Opt-in

Advantages
  • Relatively easy to implement.
  • Can be gradually phased in.
Disadvantages
  • Results in low coverage out of the box.
  • Requires changes to most transclusion sites to enable balancing.
  • Does not support old revisions.

Opt-out

  • DOM notes, proposing a <domparse> extension tag that causes its content to be balanced as one unit. Use case: Multi-template content, like table start / row / end combinations.
Advantages
  • Matches common transclusion behavior, and yields predictable behavior for new citations.
Disadvantages
  • Risk of breaking existing multi-part content.
  • Need to add explicit markup for muti-part transclusions.
  • No support for old revisions.

Inference

Automatic inference aims to establish the scope of transclusion balancing using statistics across template uses. It is based on the observation that most templates are balanced (infoboxes, navboxes, citations etc), and a few (such as table start or end templates) are not balanced by design. Based on experience, there are very few ambiguous top-level templates. Parsoid has rich information about which templates are typically balanced / unbalanced, so it is conceivable that we could use these statistics to identify templates that are normally not balanced.

The challenge with inference is establishing a stable transclusion scope. We have not spent a lot of time investigating possible options for this problem yet. Here are some possible ideas:

Thesis 1: Basically all multi-template content is started by a specific template.
  • How to verify assumption:
    • Collect stats on templated content with data-mw.parts.length > 1.
    • Look for multi-template content that only had *end tags* (but not start tags) supplied by transclusions.
  • Some possible counter-examples I could think of
    • Partially-templated list -> balanced, constraint: list item
    • Plain table with templated rows -> balanced, constraint: table row
  • Alternative: Can find limited look-ahead that lets us establish start token by backtracking, for example a table token triggered by a table cell template.
Thesis 2: Start template implies a stable DOM parsing scope.

How to verify: Define some scopes & compare with actual scope?

Implementation sketch: Start template defined DOM scope

Goal: For a given start template, figure out what should be included in the
<domparse> action.

  • Identify multi-transclusion start templates like "table start" and mark them up in templatedata, based on statistics.
  • For each start template, define transclusion block boundary. Examples:
    • anything up to </table>" for table start template
    • anything up to <li> or </li> for list item template
  • When encountering transclusions of start templates:
    • Parse tokens from start transclusion to end-of-scope as separate DOM scope.
    • Enforce use site content model constraints on transclusion content.
  • All other transclusions: Parse as self-contained units, enforce use site content model constraints.
Advantages
  • Avoids the need to modify transclusion sites.
  • Makes the machines do most of the work.
  • Works consistently for both new & old revisions.
  • Achieves high coverage.
Disadvantages
  • Open question if stable DOM scopes can be established based on template classification.
  • No visual indication of DOM scopes in wikitext.

See also

GWicke created this task.Mar 22 2016, 12:34 AM
Restricted Application added a subscriber: Aklapper. · View Herald TranscriptMar 22 2016, 12:34 AM
GWicke edited subscribers, added: ssastry, cscott, tstarling, Arlolra; removed: Aklapper.
GWicke renamed this task from Hygienic transclusions for WYSIWYG, incremental parsing & composition to Hygienic transclusions for WYSIWYG, incremental parsing & composition: Options and trade-offs.Mar 22 2016, 1:22 AM
GWicke renamed this task from Hygienic transclusions for WYSIWYG, incremental parsing & composition: Options and trade-offs to WIP RFC: Hygienic transclusions for WYSIWYG, incremental parsing & composition: Options and trade-offs.Mar 23 2016, 9:01 PM
GWicke added a project: TechCom-RfC.
daniel added a subscriber: daniel.Apr 5 2016, 2:10 PM

I would add to the requirements that the mechanism must be independent of wikitext. The transclusion must be agnostic of the original content model, and should operate entirely on the level of the HTML DOM (respectively ParserOutput objects). This would allow us to transclude content freely across content models: a page containing tabular data could be transcluded on a wikitext page, a JSON page could be transcluded into a JS page, a page defining a graph can be transcluded on a page written in Markdown or TeX.

We already have a hack that provides such a mechanism for transcluding special pages. We also allow parser functions to return HTML, which can then be used from wikitext. It would be nice to consolidate all these mechanisms. I started some code experiments for this a while a go, but I can't find the relevant ticket right now.

The the future, perhaps even the system for placing images and other media on a page could use the same interface.

The transclusion must be agnostic of the original content model, and should operate entirely on the level of the HTML DOM (respectively ParserOutput objects).

Yes. I suspect there is broad agreement about this requirement.

Also, see #3 in https://www.mediawiki.org/wiki/User:SSastry_%28WMF%29/Notes/Wikitext#Core_ideas which proposes a slight variation, i.e. treat the output as typed, which allows for some variation, i.e. plain text, DOM, and couple other types to capture existing use cases.

tstarling claimed this task.Apr 6 2016, 8:41 PM
GWicke updated the task description. (Show Details)Apr 6 2016, 9:49 PM
GWicke updated the task description. (Show Details)Apr 7 2016, 12:51 AM
GWicke updated the task description. (Show Details)Apr 7 2016, 12:59 AM

This RFC is on the agenda for E159: RFC Meeting: Hygienic transclusions and balanced templates (2016-04-13, #wikimedia-office) (next Wednesday for most of us), where the discussion of this RFC will be paired with discussion of T114445: [RFC] Balanced templates

I have been writing up a wiki page @ https://www.mediawiki.org/wiki/User:SSastry_%28WMF%29/Notes/Document_Composability ... I could extract the relevant high level summary for RFC with pointers back to that document.

For every proper Wikipedia article, this would be the ideal:

  • Every significant claim made by an article should be citable
  • Every significant claim made by an article should be translatable
  • The presence of a claim in an article should be the subject of thoughtful consideration of the relevance of that claim for the audience of the article
  • Articles should be easy to view and edit
  • The underlying markup for an article should be intuitive and easy to understand
  • Our software should be only as complicated as it needs to be, and no more

There's a lot of tradeoffs to balance in that set of assertions. The reason why I bring these up is that there is currently a debate about translation markup started by @brion's filing of T131516: Reduce or eliminate the need for the user to touch <translate> tags and unit markers. If/how should we be considering translation extension markup as we think through the issues discussed here?

In the context of "smart" transclusion, we should also keep in mind that the content we transclude may be written in a language different from the page that does the transclude. So the template (transclusion unit) acts as a context that defines the content language, and the parser should use that content language when processing the transcluded resource. See also T114640: RFC: make Parser::getTargetLanguage aware of multilingual wikis and the session notes on T119022: WikiDev 16 working area: Content format.

RobLa-WMF mentioned this in Unknown Object (Event).Apr 13 2016, 6:54 PM
GWicke updated the task description. (Show Details)Apr 13 2016, 7:33 PM
RobLa-WMF mentioned this in Unknown Object (Event).Apr 13 2016, 7:34 PM

Quite a while ago, I started a patch to allow non-wikitext content to be transcluded on wikitext pages. I called it "HTML based transclusion". The (stale) patch is here: https://gerrit.wikimedia.org/r/#/c/132710/16

In Subbu's Document Composability writeup, he suggests these are the two questions we should resolve:

Question 1: How are DOM fragments identified during parse?

a. Specified via wikitext semantics
b. Opt-in via special markup
c. Opt-out via special markup
d. Automatically inferred

Question 2: Possible approaches for handling nesting constraints:
a. No-op
b. Modify fragment
c. Change insertion context
d. Expand the fragment scope
e. Turn off fragment composition

This comment is an attempt to summarize what @ssastry was suggesting in E159

While I'm in the summarizing business, here's a couple of other attempts from E159 at defining the key questions:
@cscott's 6 questions:

  1. there's a choice between a whitelist and blacklist approach for {{#balance:block}} -- in the whitelist approach the only tags allowed to remain open are <div> and <section> (that last is forward-looking). thoughts about that choice would be welcome.
  2. whether the set of tags closed by block/inline is sufficient to allow interesting stuff. ie, "but I want <a> inside my inline templates"
  3. whether there should be additional modes (again, maybe an inline mode which allowed <a> tags but closed them in the context.)
  4. tables are problematic. there could in theory be three separate table modes (at least) for outer scope, row, and cell context. do we need all that?
  5. whether silently stripping bad tags is good enough, or do we want a more obvious "error" mechanism.
  6. stealing from gwicke: is there an alternative to manually specifying the context constraints?

...to which @GWicke added:

  1. how will this work with old revisions?

@GWicke's version

a) how modular do we want transclusions to be?
b) should we go with the proposed opt-in direction, and does it get us the benefits we are hoping for?
c) if we go with opt-in, should we use constraints from the context, or make the user specify the intention manually?

Restricted Application added a subscriber: TerraCodes. · View Herald TranscriptApr 19 2016, 3:11 PM
RobLa-WMF mentioned this in Unknown Object (Event).May 11 2016, 12:09 AM
RobLa-WMF triaged this task as Normal priority.May 11 2016, 8:24 PM

Per E170