Page MenuHomePhabricator

Improved editability, tooling, reasoning, and performance by adopting DOM-based semantics for wikitext
Open, Needs TriagePublic

Description

Type of activity: Pre-scheduled session
Main topic: Handling wiki content beyond plaintext

The problem

Wikitext's processing model is based on generating snippets of HTML strings for wikitext markup which are then concatenated to yield a HTML string for the entire document. This string is then parsed to yield well-formed HTML markup (typically, Tidy has been used to do this cleanup, but any HTML5 parser could also be used to parse to DOM and then serialize it to well-formed HTML markup.). So, in this string concatenation model, in the general case, you cannot know how a piece of wikitext markup is going to render without processing the entire document.

This string-concatenation based model has a number of issues:

  1. It is a poor fit for tools that operate at a structural level on the document and need to precisely map those structures back to the wikitext that generated them. VisualEditor (VE) is the best known example of such a tool. VE operates on the DOM, and the edits on the DOM need to be converted back to wikitext without introducing spurious diffs elsewhere in the document. To enable this, Parsoid does a lot of analysis and hard work to map a DOM node to its associated wikitext string that generated it. Parsoid relies on a lot of (some ugly) hacks to provide this support.
  2. When a template is edited, it triggers a reparse of all pages that use that template. This is a fairly expensive operation which is unavoidable since it is hard to know how the change is going to affect the rendering of the page.
  3. Given a piece of markup, it is not always possible to know how that markup is going to render. For example, in this markup '''foo {{some-template}} bar''', you cannot know if bar is going to be bolded or not without looking into the source of some-template.
  4. Given a page, it is not possible for multiple editors to edit the same document at a subpage level finer than sections and also guarantee their edits will render the same way as they intended. While this guarantee doesn't exist for sections either, the probability is likely very high for sections, and the probability gets worse as the granularity gets finer.
  5. When an editor makes a minor edit (say a typo) on the page, the parser still reparses the entire page because, in the general case, it cannot guarantee that the minor edit will not affect rendering elsewhere on the page.

However, with some mostly small changes to semantics of wikitext to move away from a string-concatenation based processing model to a DOM-composition based processing model, all of the above limitations can be addressed. It can provide improved ability to reason about wikitext, improved editing experience in visual editor, improved performance on template edits (and rerenders) and regular edits, as well as ability to potentially support sub-page editing at granularities much finer than a section (which also has the benefit of reducing edit conflicts).

Since this can be potential source of misunderstanding, to be explicit, this proposal is not focused on VisualEditor only nor is it about getting rid of wikitext editing. If anything, we are interested in improving the edit experience for both visual and wikitext editing.

In this session, I am going to talk about some of these changes to wikitext semantics, and how we might get there from where we are.

Expected outcome

  • Increased awareness among others of the benefits of moving to DOM-composition based processing model for wikitext.
  • Receiving concrete feedback about some of the specific proposals to get there.
  • Identifying expectations regarding usability, performance, editability, consistency, backward compatibility, and ability to transition smoothly.

Current status of the discussion

Right now, there have been a few scattered in-person discussions with other members of the parsing team. The proposal exists in a somewhat rough draft form on mediawiki.org. But, beyond that, it hasn't had much discussion or exposure.

Links

Related Objects

Event Timeline

ssastry created this task.Oct 27 2016, 4:39 AM
Restricted Application added a subscriber: Aklapper. · View Herald TranscriptOct 27 2016, 4:39 AM
ssastry renamed this task from [WIP] Improved editablity, tooling, reasoning, and performance by adopting DOM-based semantics for wikitext to [WIP] Improved editability, tooling, reasoning, and performance by adopting DOM-based semantics for wikitext.Oct 27 2016, 4:39 AM
ssastry added a project: Parsing-Team.
ssastry updated the task description. (Show Details)Oct 27 2016, 4:50 AM
Elitre added a subscriber: Elitre.
ssastry updated the task description. (Show Details)Oct 27 2016, 6:02 PM
ssastry renamed this task from [WIP] Improved editability, tooling, reasoning, and performance by adopting DOM-based semantics for wikitext to Improved editability, tooling, reasoning, and performance by adopting DOM-based semantics for wikitext.Oct 28 2016, 4:01 AM
ssastry updated the task description. (Show Details)
Siznax added a subscriber: Siznax.Nov 15 2016, 7:52 PM
ssastry updated the task description. (Show Details)Nov 22 2016, 5:41 PM
Mholloway updated the task description. (Show Details)Nov 23 2016, 2:24 PM
Anomie added a subscriber: Anomie.Nov 23 2016, 4:47 PM
  1. When a template is edited, it triggers a reparse of all pages that use that template. This is a fairly expensive operation which is unavoidable since it is hard to know how the change is going to affect the rendering of the page.
  1. When an editor makes a minor edit (say a typo) on the page, the parser still reparses the entire page because, in the general case, it cannot guarantee that the minor edit will not affect rendering elsewhere on the page.

Your goal behind these problems seems to be that you could unplug the old template's output DOM fragment and plug in the new template's output in its place. I see some things that don't seem to really be addressed in the linked document. I don't think any of these are insurmountable, and you may already have solutions for some of them, and many could probably be solved by sufficiently clever propagation of metadata, but they can't be ignored.

  • Post-processing for Cite's <ref> and <references> tags is mentioned, but I don't see consideration of how to handle cases when multiple things need to do post-processing. See T4700 for a similar issue with the existing parser.
  • Extensions like Variables violate the basic "DOM-shaped hole" assumption here. WMF wikis don't use that extension and Scribunto explicitly tries to avoid that use case (T67258), but we should be clear about breaking it.
  • Assuming that various limitations remain, for example the "expensive parser function" limitation or Scribunto's "total Lua execution time" limitation, it may be possible for a series of edits to produce an output via merging DOM fragments that cannot be reconstructed by parsing the resulting page from scratch.
  • Metadata in general is currently handwaved. The use cases need analysis so a sufficiently expressive representation can be designed when the time comes.
    • Magic words like __NOINDEX__, __HIDDENCAT__, or {{DISPLAYTITLE:}} that affect page properties need to be tracked in metadata, probably with a DOM crawl to merge this metadata from all the replaceable nodes.
    • Content can also affect cache fragmentation, see ParserOutput::getUsedOptions(). Again, probably a metadata crawl-and-merge is sufficient to track it. Note this can affect the DOM output, for example whenever the {{int:}} magic word is used.
    • The links tables (categorylinks, pagelinks, templatelinks, imagelinks, externallinks, and so on) need to be maintained. Metadata crawl-and-merge may be sufficient to track it, but the metadata here will probably have to be more complicated than just a list of values for each links table (see below).
    • Parser functions like {{#if:}} need their own reprocessing if certain of their input parameters change, not just a replacing of DOM fragments in their output. For example, {{#if:{{foo}}|{{bar}}|{{baz}}}} could get by with the normal handling for whichever of {{bar}} or {{baz}} is output, but metadata from {{foo}} would have to be selectively copied to the #if's node (e.g. a normal categorylink wouldn't be copied, but one for a tracking category would have to be), it would have to signal a templatelink on Template:Foo itself, and the replacer would have to be smart enough to read that metadata and reprocess the {{#if:}} when Template:Foo changes.
    • Parser functions like {{#ifexist:}} or {{#invoke:}} have even more complex dependencies, they may need reprocessing if any of multiple arbitrary other pages change, sometimes only in certain ways (e.g. {{#ifexist:}} only cares if its target is created or deleted).
  1. Given a page, it is not possible for multiple editors to edit the same document at a subpage level finer than sections and also guarantee their edits will render the same way as they intended.

How do you guarantee that at the section level, since sections can currently be inside other block-level markup?

Lord_Farin added a subscriber: Lord_Farin.EditedNov 23 2016, 6:10 PM

Wikitext's processing model is based on generating snippets of HTML strings for wikitext markup [...] So, in this string concatenation model, in the general case, you cannot know how a piece of wikitext markup is going to render without processing the entire document.
This string-concatenation based model has a number of issues:

  1. It is a poor fit for tools that operate at a structural level on the document and need to precisely map those structures back to the wikitext that generated them. VisualEditor (VE) is the best known example of such a tool. VE operates on the DOM, and the edits on the DOM need to be converted back to wikitext without introducing spurious diffs elsewhere in the document. To enable this, Parsoid does a lot of analysis and hard work to map a DOM node to its associated wikitext string that generated it. Parsoid relies on a lot of (some ugly) hacks to provide this support.

From my use-case (a mathematical wiki with a lot of MathJax rendering in place) I have trouble seeing VE going to work - ever. At the very least the VE paradigm should not be too aggressive. It should be able to derive, as and when I choose to start editing somewhere, if there is a surrounding local context that can be isolated from the entire page (this would necessitate some scoping mechanism for templates/transclusions/etc). Within this context, it might be possible to edit in-line. If so, please go ahead. If not, one should be relegated to text-based editing.

VE is nice and all, but precise layout control, predictable rendering, and intelligible wikitext override the niceties that come with inline visual editing.
In general my interest lies with retaining the low-level power that full-fledged text-based wiki editing currently provides (parser functions, tag extensions, hooks, and all). The bundled creativity of the thousands of programmers and other MediaWiki enthusiasts is too valuable to consider deprecating it to make our rendering and parsing life easier. Also e.g. table templates will continue to serve a purpose, and it is our job to handle them in a user-friendly way.

  1. When a template is edited, it triggers a reparse of all pages that use that template. This is a fairly expensive operation which is unavoidable since it is hard to know how the change is going to affect the rendering of the page.
  2. Given a piece of markup, it is not always possible to know how that markup is going to render. For example, in this markup '''foo {{some-template}} bar''', you cannot know if bar is going to be bolded or not without looking into the source of some-template.

These points seem more related to the desired scope of templates, which is to be taken into consideration when coming up with a DOM- or string-based rendering, and not to string-based rendering as such.

  1. Given a page, it is not possible for multiple editors to edit the same document at a subpage level finer than sections and also guarantee their edits will render the same way as they intended.
  2. When an editor makes a minor edit (say a typo) on the page, the parser still reparses the entire page because, in the general case, it cannot guarantee that the minor edit will not affect rendering elsewhere on the page.

However, with some mostly small changes to semantics of wikitext to move away from a string-concatenation based processing model to a DOM-composition based processing model, all of the above limitations can be addressed. It can provide improved ability to reason about wikitext, improved editing experience in visual editor, improved performance on template edits (and rerenders) and regular edits, as well as ability to potentially support sub-page editing at granularities much finer than a section (which also has the benefit of reducing edit conflicts).

The overall impression I get is that we are looking for some way to identify scope-limiting opportunities in the DOM (whether the DOM is imagined as an intermediate within text-based parsing, or considered the central processing mechanism). That is to say, we want to be able to discern which portions of the source code give rise to a more-or-less isolated segment of the DOM (comparable to the current, very crude heuristic "section" implemented in the edit section links).
All of this seems to be aimed at identifying editable portions of the DOM. But a DOM can give us much, much more.

A full parse will conceptually yield a purely syntactic DOM, which can subsequently be used for all kinds of interesting things, before the rendering phase (based on said syntactic DOM) starts. Once the content of a DOM node has been established, we can call upon semantic postprocessing routines to enhance the display/content/anything -- the prime example of semantic postprocessing being ToC generation.
But one can also imagine automatic truncation/collapsing of transcluded pages, automation of tedious prettifying wikitext constructs, insertion of widgets (table search/sorting, "random related image", or other external services). If the semantic model is complete, possibilities are endless, and there is a decoupling from rendering techniques.

It is easy to make a new markup language that readily supports inline editing, performs great, and is machine-intelligible. We could call it markdown. But that is not the challenge we are facing -- we're facing the challenge to keep the virtues of wikitext, while making maintenance and future enhancements easier and more accessible.

Thanks for taking the time to provided your detailed comments. Some responses below.

Wikitext's processing model is based on generating snippets of HTML strings for wikitext markup [...] So, in this string concatenation model, in the general case, you cannot know how a piece of wikitext markup is going to render without processing the entire document.
This string-concatenation based model has a number of issues:

  1. It is a poor fit for tools that operate at a structural level on the document and need to precisely map those structures back to the wikitext that generated them. VisualEditor (VE) is the best known example of such a tool. VE operates on the DOM, and the edits on the DOM need to be converted back to wikitext without introducing spurious diffs elsewhere in the document. To enable this, Parsoid does a lot of analysis and hard work to map a DOM node to its associated wikitext string that generated it. Parsoid relies on a lot of (some ugly) hacks to provide this support.

From my use-case (a mathematical wiki with a lot of MathJax rendering in place) I have trouble seeing VE going to work - ever. At the very least the VE paradigm should not be too aggressive. It should be able to derive, as and when I choose to start editing somewhere, if there is a surrounding local context that can be isolated from the entire page (this would necessitate some scoping mechanism for templates/transclusions/etc). Within this context, it might be possible to edit in-line. If so, please go ahead. If not, one should be relegated to text-based editing.
VE is nice and all, but precise layout control, predictable rendering, and intelligible wikitext override the niceties that come with inline visual editing.
In general my interest lies with retaining the low-level power that full-fledged text-based wiki editing currently provides (parser functions, tag extensions, hooks, and all). The bundled creativity of the thousands of programmers and other MediaWiki enthusiasts is too valuable to consider deprecating it to make our rendering and parsing life easier. Also e.g. table templates will continue to serve a purpose, and it is our job to handle them in a user-friendly way.

To be explicitly very clear, we are not proposing to get rid of wikitext editing. To also be explicit once again, I agree with your last 2 statements about the creativity of template editors and table templates. I used VE as an example of one editor that can benefit from DOM semantics. However, non-VE editing can benefit from DOM scopes.

  1. When a template is edited, it triggers a reparse of all pages that use that template. This is a fairly expensive operation which is unavoidable since it is hard to know how the change is going to affect the rendering of the page.
  2. Given a piece of markup, it is not always possible to know how that markup is going to render. For example, in this markup '''foo {{some-template}} bar''', you cannot know if bar is going to be bolded or not without looking into the source of some-template.

These points seem more related to the desired scope of templates, which is to be taken into consideration when coming up with a DOM- or string-based rendering, and not to string-based rendering as such.

Indeed. This is about bounding the scope of template output and parsing them independently on their own separate from the top level page. But, how the template produces its content itself is not as relevant. For practical purposes, it makes sense to support the existing model as long as we can treat its output as a DOM structure.

  1. Given a page, it is not possible for multiple editors to edit the same document at a subpage level finer than sections and also guarantee their edits will render the same way as they intended.
  2. When an editor makes a minor edit (say a typo) on the page, the parser still reparses the entire page because, in the general case, it cannot guarantee that the minor edit will not affect rendering elsewhere on the page.

However, with some mostly small changes to semantics of wikitext to move away from a string-concatenation based processing model to a DOM-composition based processing model, all of the above limitations can be addressed. It can provide improved ability to reason about wikitext, improved editing experience in visual editor, improved performance on template edits (and rerenders) and regular edits, as well as ability to potentially support sub-page editing at granularities much finer than a section (which also has the benefit of reducing edit conflicts).

The overall impression I get is that we are looking for some way to identify scope-limiting opportunities in the DOM (whether the DOM is imagined as an intermediate within text-based parsing, or considered the central processing mechanism). That is to say, we want to be able to discern which portions of the source code give rise to a more-or-less isolated segment of the DOM (comparable to the current, very crude heuristic "section" implemented in the edit section links).
All of this seems to be aimed at identifying editable portions of the DOM. But a DOM can give us much, much more.
A full parse will conceptually yield a purely syntactic DOM, which can subsequently be used for all kinds of interesting things, before the rendering phase (based on said syntactic DOM) starts. Once the content of a DOM node has been established, we can call upon semantic postprocessing routines to enhance the display/content/anything -- the prime example of semantic postprocessing being ToC generation.
But one can also imagine automatic truncation/collapsing of transcluded pages, automation of tedious prettifying wikitext constructs, insertion of widgets (table search/sorting, "random related image", or other external services). If the semantic model is complete, possibilities are endless, and there is a decoupling from rendering techniques.

Aboslutely. This proposal (phab task, wiki page) is basically a generalization of the idea of DOM scopes as in T114444. Also see this and this. The idea is to be able to generate an AST that lends itself to tooling of all sorts (I allude to an AST in https://www.mediawiki.org/wiki/Parsing/Notes/Wikitext_2.0/Strawman_Spec which is not fully developed yet, just one afternoon's thoughts dumped on the wiki).

It is easy to make a new markup language that readily supports inline editing, performs great, and is machine-intelligible. We could call it markdown. But that is not the challenge we are facing -- we're facing the challenge to keep the virtues of wikitext, while making maintenance and future enhancements easier and more accessible.

Agreed. We are on the same page here.

Thanks for the detailed comments and feedback. I'll cherry pick a few comments that I can respond to right away and leave the rest for later once I have had a chance to digest your comment and see whether they are addressed already or how they could be addressed.

  1. When a template is edited, it triggers a reparse of all pages that use that template. This is a fairly expensive operation which is unavoidable since it is hard to know how the change is going to affect the rendering of the page.
  2. When an editor makes a minor edit (say a typo) on the page, the parser still reparses the entire page because, in the general case, it cannot guarantee that the minor edit will not affect rendering elsewhere on the page.

Your goal behind these problems seems to be that you could unplug the old template's output DOM fragment and plug in the new template's output in its place. I see some things that don't seem to really be addressed in the linked document. I don't think any of these are insurmountable, and you may already have solutions for some of them, and many could probably be solved by sufficiently clever propagation of metadata, but they can't be ignored.

  • Post-processing for Cite's <ref> and <references> tags is mentioned, but I don't see consideration of how to handle cases when multiple things need to do post-processing. See T4700 for a similar issue with the existing parser.

I don't fully understand this. Is the concern here about ordering or post-processing passes where one post-processing pass might make changes that could trigger something else? If there are extensions / passes that require global state, then they would have to be run on the final document no matter how it is produced (whether by a full page parse, or whether it is produced by incrementally updating previous output). Of course, you can do all kinds of clever optimizations to reduce work, but, the principle remains.

Ideally, post-processing passes are a lot cheaper than the original page parse which is the scenario where something like this makes sense.

  • Extensions like Variables violate the basic "DOM-shaped hole" assumption here. WMF wikis don't use that extension and Scribunto explicitly tries to avoid that use case (T67258), but we should be clear about breaking it.

You are correct. The plan is to move extensions away from relying on ordered global state. And, yes, we can be explicit about breaking this functionality.

  • Assuming that various limitations remain, for example the "expensive parser function" limitation or Scribunto's "total Lua execution time" limitation, it may be possible for a series of edits to produce an output via merging DOM fragments that cannot be reconstructed by parsing the resulting page from scratch.

True. I haven't considered this edge case.

  • Metadata in general is currently handwaved. The use cases need analysis so a sufficiently expressive representation can be designed when the time comes.
    • Magic words like __NOINDEX__, __HIDDENCAT__, or {{DISPLAYTITLE:}} that affect page properties need to be tracked in metadata, probably with a DOM crawl to merge this metadata from all the replaceable nodes.

In this section, I have proposed treating metadata as annotations on the DOM tree. If this can be represented and tracked in this fashion, then replacing one DOM tree with another would also let us update metadata properly. Does that cover what you were thinking?

  • Content can also affect cache fragmentation, see ParserOutput::getUsedOptions(). Again, probably a metadata crawl-and-merge is sufficient to track it. Note this can affect the DOM output, for example whenever the {{int:}} magic word is used.
  • The links tables (categorylinks, pagelinks, templatelinks, imagelinks, externallinks, and so on) need to be maintained. Metadata crawl-and-merge may be sufficient to track it, but the metadata here will probably have to be more complicated than just a list of values for each links table (see below).
  • Parser functions like {{#if:}} need their own reprocessing if certain of their input parameters change, not just a replacing of DOM fragments in their output. For example, {{#if:{{foo}}|{{bar}}|{{baz}}}} could get by with the normal handling for whichever of {{bar}} or {{baz}} is output, but metadata from {{foo}} would have to be selectively copied to the #if's node (e.g. a normal categorylink wouldn't be copied, but one for a tracking category would have to be), it would have to signal a templatelink on Template:Foo itself, and the replacer would have to be smart enough to read that metadata and reprocess the {{#if:}} when Template:Foo changes.
  • Parser functions like {{#ifexist:}} or {{#invoke:}} have even more complex dependencies, they may need reprocessing if any of multiple arbitrary other pages change, sometimes only in certain ways (e.g. {{#ifexist:}} only cares if its target is created or deleted).

The common theme in some of these cases is to be able to determine when a full page (re)parse is required. The goal of this proposal is not to completely eliminate full page parses. We need a functionally correct solution that works in all cases (by doing a full page parse where necessary), but that, in a vast majority of cases can actually get away with composing DOM fragments from old pages and new edits. So, as long as we can design this to let us fall back to full page reparse, I think we are covered.

  1. Given a page, it is not possible for multiple editors to edit the same document at a subpage level finer than sections and also guarantee their edits will render the same way as they intended.

How do you guarantee that at the section level, since sections can currently be inside other block-level markup?

Think of the structure being edited as a DOM tree. There is nothing special about sections there. Since the atomic structure you are editing (what these atomic structure are is something that the new spec will propose: but, for example, tables, lists, list items, table cells, table rows, paragraphs, etc. are all game) is always going to reparsed to a DOM tree, you cannot insert a "</div>" in that section and corrupt the outer block level markup. However, because HTML5 has constraints on how structures can be nested, it can indeed affect structure higher up in the tree. How this will be handled is determined by a spec we come up with for how to compose DOM fragments properly. These notes here talk about this a bit. Does that answer your question?

Thanks for taking the time to provided your detailed comments. Some responses below.

I'm happy to note the constructive attitude, thanks :).

In general my interest lies with retaining the low-level power that full-fledged text-based wiki editing currently provides (parser functions, tag extensions, hooks, and all). The bundled creativity of the thousands of programmers and other MediaWiki enthusiasts is too valuable to consider deprecating it to make our rendering and parsing life easier. Also e.g. table templates will continue to serve a purpose, and it is our job to handle them in a user-friendly way.

To be explicitly very clear, we are not proposing to get rid of wikitext editing. To also be explicit once again, I agree with your last 2 statements about the creativity of template editors and table templates. I used VE as an example of one editor that can benefit from DOM semantics. However, non-VE editing can benefit from DOM scopes.

It is a relief to read this, because all the previous discussions I read about the "future of parsing" (notably the closely related T142803 and what is linked from there) somehow kept this in the middle or came across as seeing the source code editor and current parser as second class citizens.

  1. [...]
  2. [...]

However, with some mostly small changes to semantics of wikitext to move away from a string-concatenation based processing model to a DOM-composition based processing model, all of the above limitations can be addressed. It can provide improved ability to reason about wikitext, improved editing experience in visual editor, improved performance on template edits (and rerenders) and regular edits, as well as ability to potentially support sub-page editing at granularities much finer than a section (which also has the benefit of reducing edit conflicts).

The overall impression I get is that we are looking for some way to identify scope-limiting opportunities in the DOM [...]
A full parse will conceptually yield a purely syntactic DOM, which can subsequently be used for all kinds of interesting things, before the rendering phase (based on said syntactic DOM) starts. Once the content of a DOM node has been established, we can call upon semantic postprocessing routines to enhance the display/content/anything -- the prime example of semantic postprocessing being ToC generation.
But one can also imagine automatic truncation/collapsing of transcluded pages, automation of tedious prettifying wikitext constructs, insertion of widgets (table search/sorting, "random related image", or other external services). If the semantic model is complete, possibilities are endless, and there is a decoupling from rendering techniques.

Aboslutely. This proposal (phab task, wiki page) is basically a generalization of the idea of DOM scopes as in T114444. Also see this and this. The idea is to be able to generate an AST that lends itself to tooling of all sorts (I allude to an AST in https://www.mediawiki.org/wiki/Parsing/Notes/Wikitext_2.0/Strawman_Spec which is not fully developed yet, just one afternoon's thoughts dumped on the wiki).

Now that I took the effort to also read the strawman spec, I'm more convinced about the general direction we're going here.
If I may sneak in one thing: Please consider support for nested extension tags, and improving support for nesting in general. It is a pain to work around the limitation of choosing between no-parse and full-parse-then-mask in the current text-based parser. A DOM/AST approach provides the opportunity to do better and I would be sad if we missed out on it.

It is easy to make a new markup language that readily supports inline editing, performs great, and is machine-intelligible. We could call it markdown. But that is not the challenge we are facing -- we're facing the challenge to keep the virtues of wikitext, while making maintenance and future enhancements easier and more accessible.

Agreed. We are on the same page here.

Good. Because when I started reading the top post, that was not what came to mind :).
I will pay close attention to the follow-up; for that it'd be nice if there is a post on wikitech-l every once in a while.

Thanks for your efforts!

I don't fully understand this. Is the concern here about ordering or post-processing passes where one post-processing pass might make changes that could trigger something else?

The second, "post-processing passes where one post-processing pass might make changes that could trigger something else".

probably with a DOM crawl to merge this metadata from all the replaceable nodes.

In this section, I have proposed treating metadata as annotations on the DOM tree. If this can be represented and tracked in this fashion, then replacing one DOM tree with another would also let us update metadata properly. Does that cover what you were thinking?

I think we're saying the same thing here, yes.

The common theme in some of these cases is to be able to determine when a full page (re)parse is required. The goal of this proposal is not to completely eliminate full page parses. We need a functionally correct solution that works in all cases (by doing a full page parse where necessary), but that, in a vast majority of cases can actually get away with composing DOM fragments from old pages and new edits. So, as long as we can design this to let us fall back to full page reparse, I think we are covered.

None of the cases I pointed out necessarily require a full page reparse, just that things could be more complicated than looking for the DOM subtree produced by a simple {{foo}} and replacing it with the new subtree.

My main motivation here, though, was to start listing out the different metadata that would need to be tracked for existing use cases.

  1. Given a page, it is not possible for multiple editors to edit the same document at a subpage level finer than sections and also guarantee their edits will render the same way as they intended.

How do you guarantee that at the section level, since sections can currently be inside other block-level markup?

Think of the structure being edited as a DOM tree. There is nothing special about sections there. Since the atomic structure you are editing (what these atomic structure are is something that the new spec will propose: but, for example, tables, lists, list items, table cells, table rows, paragraphs, etc. are all game) is always going to reparsed to a DOM tree, you cannot insert a "</div>" in that section and corrupt the outer block level markup. However, because HTML5 has constraints on how structures can be nested, it can indeed affect structure higher up in the tree. How this will be handled is determined by a spec we come up with for how to compose DOM fragments properly. These notes here talk about this a bit. Does that answer your question?

I think you misunderstood. The proposal states that it's not possible in the string-based model to allow multiple editors to edit the same document at a subpage level finer than sections and also guarantee their edits will render the same way as they intended. This seems to imply that it is possible in the string-based model to make that guarantee when editing at the section level. I don't think the string-based model can even guarantee that, although often it works out.

I don't fully understand this. Is the concern here about ordering or post-processing passes where one post-processing pass might make changes that could trigger something else?

The second, "post-processing passes where one post-processing pass might make changes that could trigger something else".

This will require some thought by looking at use cases where this currently happens.

The common theme in some of these cases is to be able to determine when a full page (re)parse is required. The goal of this proposal is not to completely eliminate full page parses. We need a functionally correct solution that works in all cases (by doing a full page parse where necessary), but that, in a vast majority of cases can actually get away with composing DOM fragments from old pages and new edits. So, as long as we can design this to let us fall back to full page reparse, I think we are covered.

None of the cases I pointed out necessarily require a full page reparse, just that things could be more complicated than looking for the DOM subtree produced by a simple {{foo}} and replacing it with the new subtree.
My main motivation here, though, was to start listing out the different metadata that would need to be tracked for existing use cases.

Got it. Thanks for listing them out. But, yes, in some cases, the tracking might be too complicated in which cases, it might be simpler to fall back to full page parse.

  1. Given a page, it is not possible for multiple editors to edit the same document at a subpage level finer than sections and also guarantee their edits will render the same way as they intended.

How do you guarantee that at the section level, since sections can currently be inside other block-level markup?

Think of the structure being edited as a DOM tree. There is nothing special about sections there. Since the atomic structure you are editing (what these atomic structure are is something that the new spec will propose: but, for example, tables, lists, list items, table cells, table rows, paragraphs, etc. are all game) is always going to reparsed to a DOM tree, you cannot insert a "</div>" in that section and corrupt the outer block level markup. However, because HTML5 has constraints on how structures can be nested, it can indeed affect structure higher up in the tree. How this will be handled is determined by a spec we come up with for how to compose DOM fragments properly. These notes here talk about this a bit. Does that answer your question?

I think you misunderstood. The proposal states that it's not possible in the string-based model to allow multiple editors to edit the same document at a subpage level finer than sections and also guarantee their edits will render the same way as they intended. This seems to imply that it is possible in the string-based model to make that guarantee when editing at the section level. I don't think the string-based model can even guarantee that, although often it works out.

Ah indeed, I misunderstood. And you are right about the string-based model about section level edits.

ssastry updated the task description. (Show Details)Nov 23 2016, 8:48 PM
Ltrlg added a subscriber: Ltrlg.Nov 26 2016, 12:40 AM
cscott added a subscriber: cscott.Nov 29 2016, 10:01 PM

I'm proposing to include this in a merged session: T151950: Wikitext 2.0 Session at Wikidev'17.

@ssastry Hey! As developer summit is less than four weeks from now, we are working on a plan to incorporate the ‘unconference sessions’ that have been proposed so far and would be generated on the spot. Thus, could you confirm if you plan to facilitate this session at the summit? Also, if your answer is 'YES,' I would like to encourage you to update/ arrange the task description fields to appear in the following format:

Session title
Main topic
Type of activity
Description Move ‘The Problem,' ‘Expected Outcome,' ‘Current status of the discussion’ and ‘Links’ to this section
Proposed by Your name linked to your MediaWiki URL, or profile elsewhere on the internet
Preferred group size
Any supplies that you would need to run the session e.g. post-its
Interested attendees (sign up below)

  1. Add your name here

We will be reaching out to the summit participants next week asking them to express their interest in unconference sessions by signing up.

To maintain the consistency, please consider referring to the template of the following task description: https://phabricator.wikimedia.org/T149564.

@ssastry Hey! As developer summit is less than four weeks from now, we are working on a plan to incorporate the ‘unconference sessions’ that have been proposed so far and would be generated on the spot. Thus, could you confirm if you plan to facilitate this session at the summit?

This session has been merged into a common session. See below.

I'm proposing to include this in a merged session: T151950: Wikitext 2.0 Session at Wikidev'17.

So, this won't be part of the unconference.

Qgil reopened this task as Open.Dec 20 2016, 7:13 AM
Qgil added a subscriber: Qgil.

Actually... As a Summit proposal, this topic will be part of T151952: Media, Visualizations, and Layout at Wikidev'17.

This task has a live of its own though, so I am just removing it from the Summit workboard.

Actually... As a Summit proposal, this topic will be part of T151952: Media, Visualizations, and Layout at Wikidev'17.

@Qgil, as far as I know, it is part of T151950: Wikitext 2.0 Session at Wikidev'17.

Qgil added a comment.EditedDec 21 2016, 9:42 AM

Argh, sorry, quick ugly copy&paste yesterday while going through dozens of Unconference proposals.

If you want to simply merge the task (and therefore close this discussion, please do so. I just thought that there was a lot of (unresolved?) discussion here and didn't dare to simply merge the task.

Tgr awarded a token.Dec 23 2016, 1:37 AM
Tgr added a subscriber: Tgr.
Boshomi added a subscriber: Boshomi.Jan 1 2017, 9:21 AM

https://www.mediawiki.org/wiki/Topic:Ti4be2qa4i6pq3o8 has some new discussion relevant to this topic.

ssastry updated the task description. (Show Details)Jan 9 2017, 5:53 AM
Magol added a subscriber: Magol.Aug 19 2019, 8:59 AM
Restricted Application added a subscriber: Liuxinyu970226. · View Herald TranscriptAug 19 2019, 8:59 AM