From @Tgr:

I've been thinking about the Parsing email. AIUI we have four options:

• Make the recommendation service return Parsoid documents (or some annotation that can be applied to Parsoid documents). AIUI what Djellel is doing, this would not be hard (processing HTML is probably a bit slower / more resource-intensive than processing wikitext, because the tree is deeper, OTOH it's a very standard task so there will be native implementations while mwparserfromhell is pure Python); there are no HTML dumps but there likely will be by the time we'd actually need them. But, there are many HTML representations of the same article wikitext (due to user options, desktop vs mobile, templates changing over time etc), so the recommendation would have to be applied to a slightly different document from what it was generated for. If we can commit to always generate recommendations just-in-time, this would not be an issue, otherwise it seems problematic.
• Make the recommendation service return some annotation for the wikitext that can be converted into annotation for the specific Parsoid document. I can't imagine any other way of doing this than offsets, and the Parsing team is not keen on committing to maintain offset mapping as a public API.
• Make the recommendation service return something that applies to wikitext, and interject in the rendering process somehow to replace the real wikitext with a modified one that will result in the extra markup we want after it's transformed by the parser.
  • a) Do this for the old parser (ie. render the extra markup into the normal pageview, not VE - we'd have to skip the editing step entirely, provide some review dialog thingy, and apply back the changes to wikitext via the web API). How? ContentGetParserOutput is too early, ContentAlterParserOutput is too late. We could add a new hook I suppose. We'd also have to split the parser cache. The resulting markup would show up in normal page views, not VE, but maybe that's not bad - we can provide some review dialog thingy, and apply back the changes to wikitext.
  • b) Do this for Parsoid (ie. render the extra markup into the VE DOM, not normal pageviews). I guess an integration point for doing that could be built into ApiVisualEditor, and that could also interact with the suggested-edits session flag somehow? We'd also have to get rid of the extra markup during save. I suppose we can use ParserPreSaveTransformComplete for that, but it seems messy.

From @cscott:

Briefly: if you could engineer your tool to use the Parsoid HTML as a basis, instead of mwparserfromhell, I think your job would be much easier. You can use Element and other python libraries for interacting with the HTML, and restbase or the new dumps service for getting the HTML.

Then you can use an xpath query to identify the particular segment in the parsoid html you are interested in, and I believe that could be translated to VE pretty directly, since it is working with the same DOM tree structure. Or as you suggest, feeding VE the Parsoid HTML with an added class added to the link text. I think you could do that by stuffing the new HTML into the 'stash' mechanism as a "not-yet-saved proposed edit".

Parsoid does maintain a mapping of wikitext offsets to particular bits of DOM tree structure, but we don't export that as a public API and I think we would be very reluctant to do so. We did discuss something similar as a way to support the "new wikitext editor", where Parsoid would export the character boundaries of certain wikitext constructs to guide the wikitext source editor, but we did not end up going that way.

I don't think the Text Fragments API will do what you want. VE constructs its content-editable canvas on the fly as it loads content from Parsoid, so it won't be in the browser's DOM tree at the time the Text Fragments API would want to scroll to it. Visual Editor uses https://developer.mozilla.org/en-US/docs/Web/API/Range internally, and the DiscussionTools work does as well; that would be the way to communicate regions with VE.

From IRC discussion yesterday with @DED, it sounds like a challenge is to ignore already linked items; the algorithm in the tool would like to have plain sentences (without links) to operate on. As C. Scott mentioned it should be possible to use a library to operate on plain text from the sentence, while also retaining the ability to manipulate the sentence to add a new mw:WikiLink on the relevant word(s).

Example Parsoid HTML for @MGerlach and @DED to reference

And the same content in Wikitext

@cscott @ssastry we're considering an implementation where, instead of attempting to compute the offsets where we want text to be linked, we just know a list of phrases that should be linked, as well as how many times that phrase has occurred in the document before the link is supposed to be added.

For example, given an article with text like this:

In contrast, an Arabic qanun measures a bit larger as mentioned.

The qanun is played on the lap while sitting or squatting, or sometimes on trestle support.

The link recommender might propose that the following phrases should be linked: "Arabic qanun", "qanun", "trestle support", with some metadata like:

[ 
{
  phrase: "Arabic qanun",
  occurrence: 0,
  linkTarget: "Arabic_Qanun"  
},
{
  phrase: "qanun",
  occurrence: 1,
  linkTarget: "Qanun"
}
]

Our frontend tool would use VisualEditor's content editable area (see ve.ui.FindAndReplaceDialog for a rough idea of what we'd be doing) to find e.g. "qanun", and since we know that there is 1 occurrence before the phrase we are going to create a link suggestion for, it would propose "qanun" in the second paragraph and not "qanun" in "Arabic qanun" in the first paragraph.

@Tgr and @Esanders pointed out that we need to be careful about what kind of text conversion happens between wikitext and Parsoid HTML (entity decoding, Unicode normalization, more?). Do you think if the mwaddlink tool is generating phrase recommendations with wikitext, that we will then run into problems in locating those phrases in VisualEditor due to text conversion? Should we use the transform-wikitext-to-html API for phrases that the mwaddlink tool provides to work around any potential issues?