More discussion from slack:

@daniel By the way, I'm inclined to replace ?stash=true with a format, html_for_editing or some such. Or we could have both: html_for_editing has parsoid data, but if you combine it with stash, it has IDs. The stash param would do nothing with normal /html.
@cscott I think my preference is for a "format" option, because I think there will be (eventually) lots of different post processing combinations.
@cscott (Although I'd /like/ to avoid that.)
@cscott for example, "mobile HTML" is a thing
@cscott and "TOC included, yes/no" is already a thing
@cscott basically all the options in ParserOutput::getText() and then some are probably going to show up in the API endpoint at some point.
@cscott although one hopes that they are all coming from the same cached content/ParserOutput and are just various postprocessing done on it
@daniel ParserOutput::getText needs to be factored out urgently, before it accumulates more logic
@cscott Yes, I've even got a phab task for it
@cscott https://phabricator.wikimedia.org/T293512
@cscott any bikeshedding you want to do about the name/location/etc of the future "post processing parserOutput" class/hierarchy would be welcome. I've speculated that the Content hierarcy might be a good place, but I'm not really familiar with everything that lives there now.
@cscott I think my vague idea is that we might have a small number of different "format" options like "read", "edit", "mobile", "discussiontools" (!) which each map internally into a set of postprocessing options/steps like "discardDataParsoid", "no-toc", etc, to avoid exposing the entire postprocessing chain as part of the API.
@daniel Yes, that sounds reasonable.
@cscott The other option is just to allow an extensible number of "transforms" and ask for them explicitly. That is a bit more future-proof, in that it allows you to (say) transition the mobile app to no-toc content by having the latest version of the app explicitly request options=mobile,no-toc or whatever, while not breaking older versions of the app which expect toc inline in the HTML.
@daniel I'd not put the post processing in the content hierarchy. I'd prefer them to be orthogonal. The same post-processing could be applied to output generated from various kinds of content.
@daniel The thought experiment I'd make is "if someone wrote extensive markdown support for mediawiki, would they want this transformation to work on their output"?
@cscott maybe something under MediaWiki\Parser then?
@cscott Red link tagging is a good thought experiment, you'd probably want that to work on markdown.
@cscott and it's specific to mediawiki & requires database access, eg, so fits better in core than in Parsoid.
@daniel I'd want variant conversion to work on markdown too.
@daniel In my mind, these transformations have nothing to do with wikitext. They are html-to-html transformations. They are independent of the content type, and should be oblivious of whatever generates the HTML, or what from.
@cscott "strip infoboxes" is something that happens in mobile HTML, I guess that's something you'd do for markdown as well? It's really an artifact of the fact that we never fully pursued MCR so the infoboxes are still "inline" with the main article content instead of being a separate content type.
@daniel Not sure MCR would remove infoboxes completely from the content. The layout gets tricky if you want to do it generically, especially if there are many kinds of infobox on the same page
@cscott (i'll just say that "page layout" wants to be an MCR type of its own.)
@daniel I'd probably start a new top level namespace for "output transformation".
@cscott MediaWiki\OutputTransform ?
@daniel MediaWiki\OutputTransform sounds good to me
@ssastry ( oh ...lots of dicsussion here ... will catch up later) .. but MediaWiki\html2html is less cumbersome?
@ssastry but, i suppose that ties it to html as a format.
@cscott (it's also a bit tricky by the specific way we use "html" and "dom" in parsoid, where "html" is usually "html as a string" and "dom" is "html as a parsed DOM tree"; to be consistent I'd hope this would be "dom2dom")
@ssastry I will also add another variable into this space ... does a html2html transform always have to live as core code? Are they / can they be "micro"services?
@daniel html2html would also work for me. I want it to be tied to HTML as a format.
@daniel Other kinds of transformations are conceptually different.
@daniel DOMTransform also works for me. Anything that doesn't bind it to wikitext or parsing :slightly_smiling_face:
@ssastry scott dom/html is a detail can be combined in htm2html in my view.
@cscott the word "output" is more-or-less specific in mediawiki: ParserOutput/OutputPage/etc.
@daniel ParserOutputTransform would be nice and specific, but maybe overly so.
@cscott the api question is just "does the set of postprocessing transforms want to be exposed as a limited number of 'format's, or as an infinitely-extensible set of boolean flags"
@daniel for wikidata rdf exports, we invented a system of "flavors" for this purpose
@cscott (subbu doesn't like ParserOutputTransform, but I will submit that it is extremely meaningful to a Mediawiki core developer, and it would be 100% clear that ParserOutput::getText() should live there.)
@daniel I agree that it's both very clear and very ugly 😉
@cscott yes, it shares the "very ugly" (but not the "very clear") with most parts of core. :slightly_smiling_face:
@daniel hehe...
@cscott My gut feeling is that "format" should refer to "cachable or database" things, eg wikitext from the db, html from the cache, future-expensive-to-recompute-derived-content-type.
@cscott and some other word should be used for the set of transformations you can make to that 'format'. I like 'flavor', fwiw.
@cscott format=html,flavor=edit or format=html,flavor=mobile,edit
@cscott those dimensions seem to make intuitive sense

Some discussion copied from slack:

@cscott: The key refactor there is ParserOutput::getText() https://phabricator.wikimedia.org/T293512 which doesn't belong there -- it does a bunch of content post-processing that shouldn't belong in a "dumb array of keys".
@ihurbain: and where would that belong, ultimately?
@cscott: Not sure! The phab task speculates "somewhere in the Content* hierarchy"? But maybe we'll want a Postprocess.php class on par with Parser.php? Or else this goes into OutputPage, which generally contains that sort of thing. That's a big question mark. But it's definitely "not in ParserOutput".

See also: https://gerrit.wikimedia.org/r/plugins/gitiles/mediawiki/core/+blame/refs/heads/REL1_38/includes/parser/Sanitizer.php#567

Should be fixed in MW1.39 due to T305108: Zero Config Install of VE + Parsoid for MW 1.39.

This shouldn't be an issue in MW 1.39, which doesn't require a separate REST request for zeroconf VE: T305108: Zero Config Install of VE + Parsoid for MW 1.39

It seems like we're not *necessarily* saying that the redirect shouldn't happen, but that instead perhaps it ought to include a language parameter? T301372#8011652 discusses other places where the redirect ought to depend on the user language.

Global user pages are another instance which *perhaps* should be handled via the generic redirect service, despite the title of this task: T153801: File and global user pages should not be redirected. (I think the title is mostly referring to the fact that the language settings should be taken into account when the redirect is done, which was mentioned in the comment above.)

I think current plan is to do this in the redlink-marking pass (or in a new similar postprocessing pass) so that we can reuse core parsoid html as fragments in other pages, but the selflinks are replaced "at the last minute".

Note that the redirects functionality in RESTBase has some known bugs as well. There are the following separate aspects to how redirects "should" work:

HTML dumps are already available in https://dumps.wikimedia.org/other/enterprise_html/ ; see also T302237: Outreachy Project (Round 24): Build Python library to work with html-dumps.

This has been requested by the kiwix team multiple times over the years. Hopefully this would be parsoid-format HTML dumps.

I think this part of the task can be unblocked, as we've got some consensus around the ParserOutput portion of the task I think. We need a patch which sets the ParserOptions flag to finish the work on T307691, and I've got an initial version here: https://gerrit.wikimedia.org/r/c/mediawiki/core/+/805897

Regression in 1.39: T310378: Messages don't get localized in the updater.

(off topic: why does it say "developer account not linked to phabricator" by my name? I'm @cscott here... is there some hidden preference I need to update to make that correspondence?)

The bug seems to be in lib/transformations/wrapSections.js in mobileapps, which is trying to recreate section information from the legacy HTML output.
https://gerrit.wikimedia.org/r/plugins/gitiles/mediawiki/services/mobileapps/+/refs/heads/master/lib/transformations/wrapSections.js#33

const makeSection = ( heading, id, text ) => {
	const headingLabel = heading ? heading.querySelector( 'span[id]' ) : null;
	return {
		id,
		anchor: headingLabel ? headingLabel.getAttribute( 'id' ) : '',
		toclevel: heading ? getTocLevel( heading ) : undefined,
		line: heading ? heading.textContent : undefined,
		text
	};
};

I'm pretty sure the heading.querySelector( 'span[id]' ) should be heading.querySelector( 'span.mw-headline[id]' ) or something like that.

This appears correctly on desktop, eg: https://zh.wikipedia.org/w/index.php?title=2011%E5%B9%B4%E5%A4%A7%E8%A5%BF%E6%B4%8B%E9%A3%93%E9%A3%8E%E5%AD%A3%E6%97%B6%E9%97%B4%E8%BD%B4&action=history

Thanks so much @AlexisJazz !

The Content-Transform-Team maintains the HTML format specifications (informally "Parsoid HTML" as opposed to the HTML currently displayed on the web site), and may be a useful resource for questions about (eg) how templates are represented in the HTML dump. Without distracting too much, the following projects might be an inspiration for how an "easy to use" API might look:

• https://github.com/wikimedia/parsoid-jsapi
• https://github.com/earwig/mwparserfromhell
• https://github.com/magnusmanske/mediawiki_rust
• https://git.legoktm.com/legoktm/parsoid-rs

The Kiwix project also uses "Parsoid HTML" format dumps: https://www.kiwix.org/en/

@AlexisJazz thanks, the tech news entry looks good. I'd probably add another sentence or two mentioning that this has been "in process" since July 2020 (maybe link to the previous tech news entries) and giving credit to the many editors who helped make this happen: https://meta.wikimedia.org/wiki/Equals_sign_parser_function_template_conflicts

In T307251#7899853, @Esanders wrote:

I think this should be handled similarly to how we handle categories, specifically:

• Reading web should add a tochtml param to action=parse (like categorieshtml), that returns the ToC HTML iff the ToC is not present in the content (i.e. in Vector-2022)
• When VE sees data in tochtml result, we can render it to a specific skin element (again, like categorieshtml).
• wikipage.content hook fires and Vector adds interactivity back

Could someone clarify the code path here? We're rendering as Wikitext and generating a ParserOutput for category/backlink/etc purposes, then throwing away the generated HTML and rendering the code as preformatted text (or some such) instead?

Because there are some legacy extensions that may deliberately register a parser function or extension tag *only* in a specific context, our "expand this extension tag/parser function" API should explicitly have a "skip this registration and look for another" return value. That way those legacy extensions can (in the new system) *always* register the extension tag/parser function, but return the "skip this registration" value in all cases except those of the specific context in which it wants to be active.

Some additional thoughts: it would be desirable to have a single implementation of the core parser functions (in includes/parser/CoreParserFunctions.php) that can be used by both the legacy parser and Parsoid. This shouldn't be *too* hard: most are simple text-in/text-out functions, so a wrapper which massaged the input arguments (converted from Text node or Tokens to a flat string), called the implementation function, and then converted the output from a string into a Text Node should probably be sufficient to provide for Parsoid compatibility.

A lot of existing code implements this by playing games with strip state, ie asking the user to pass the argument surrounded by <nowiki> and then pulling the raw text out of the strip state.

Did we identify the root cause? (ie, what was the change being deployed that caused this to trigger?)

Somewhat related to T299316 (w/r/t rendering JS as wikitext), T307691 (TOC generated for wikitext). A Content based way to disable TOC/PST on certain content types would be useful.

This is an interesting issue.

In T307720#7908304, @ssastry wrote:

https://codesearch.wmcloud.org/deployed/?q=!!%5Cs*config%5Cs*%24&i=nope&files=&excludeFiles=&repos= shows that it is Parsoid, core, Cite, ImageMap, SyntaxHighlight, Kartographer, TimedMdiaHandler that will need updates for Wikimedia-deployed code (and possible syncs to/from Parsoid repo to ensure CI continues to pass).

Dump from chat transcript outlining the issues:

The problematic code is in Graph:includes/ParserTag.php

There's a known-failures mechanism for broken tests. I'll look into it.

@Krinkle would it make sense for https://gerrit.wikimedia.org/r/c/mediawiki/services/mobileapps/+/701574 to do an HTTP request to the production ResourceLoader for /just/ the minerva module, and use that as the basis for the compiled-in styles that mobileapps ships, instead of trying to fetch the various .less files from checked out sources and manually build them?

Four-ish options, all not great:

• Do a simple client-side script that runs ?action=purge on every title in georgian wikipedia over the course of a week or so. "Big hammer", but reasonable since it's a small-ish wiki. (We could be fancy and only purge titles with problematic letters.)
• Use a patch like https://github.com/wikimedia/restbase/pull/1297/commits/10aa15501e5f747bce891b174ca7fb12f46c4179 which effectively blacklists content from a particular timestamp range in restbase. (So a "rolling purge" as it were.)
• Do some more network snooping to try to figure out exactly what requests are being made from the mobile client. (Either run android studio in a VM and use the VM facilities to capture traffic from the VM, or use pcap on linux, or figure out if Android Studio has built-in network capture utilities...)
• Throw this over to the mobile team and try to figure out if this is an app-side caching issue, or at least narrow this down to a specific request which is failing, so we can further dig into the precise nature of the problem (aka try to figure out *why* action=purge works).

Restbase content doesn't have a TTL, so waiting for the TTL expire won't work. Generally pages are edited pretty quickly, which has the effect of purging RESTBase, but Georgian wiki is low-traffic so that probably won't work either (at least not in a reasonable time frame).

I reviewed @Fomafix's patch shortly after it was written and made some comments, but they were never followed up on. Further, after my initial review some new code landed on master (and made it to 1.38) which would simplify what he was doing. So the patch needs to be updated, either by @Fomafix or @me or someone else.

Yep, I tested locally, but I'm glad it works for at least one other person as well! Thanks, Mark!

(Might be worth considering adding more "pages of this type" (whatever that is) to rt testing?)

Yeah, I'd expect language converter markup to be affected too (as another source of "a DOM tree serialized and embedded in data-mw"). There's a similar issue w/r/t traveresal -- some users will want to be sure to traverse even embedded HTML. I have a vague recollection there's a phab task and/or a generic traversal mechanism somewhere for that.

I created a "proper" hook interface in core for TemplateData in 1c3216bf92342e9f141946693f27c75e9c78a646 and filed T304899: Clean up the ParserFetchTemplateData hook because the interface didn't seem quite right. Then a week later I found this task. :)

Yeah, it took a while to get the trains all aligned, so I decided I'm not going to try to backport this. Thanks.

Arguably we should not be adding display space in <pre> tags. I don't know if that avoids the bug, but displayspace is a formatting hack for french : style " punctuation and is completely unnecessary for preformatted text.

Caption is last one wins, but also remember that "anything not recognized as a valid option is a caption". That behavior actually makes the syntax very hard to extend & very hard to deprecate no-longer-acception options without creating a bunch of inadvertent captions.

Turns out I accomplishes this task by getting rid of the feature in core! :)

See also T25138: Create a flag to setFunctionHook to force the parser not to parse parameters ("raw" parameter mode) and T204307: Parser Functions should support named parameters (named parameters).

There's also the /transform/html/to/wikitext endpoint exported by RESTBase (and by Parsoid if you turn on $wgParsoidEnableREST, although that endpoint is experimental and subject to change). That has the benefit of using the wiki configuration of the wiki which is exporting that endpoint. You can write a simple curl or other script to hit the REST endpoint; I think the rate limits are quite generous.

In T305254#7828768, @Arlolra wrote:

Parsoid itself has a bin/parse.php that takes an argument, --html2wt
https://github.com/wikimedia/parsoid/blob/master/bin/parse.php#L59

Does that suffice for your purposes?

In T302114#7829212, @Arlolra wrote:

when you use a static image thumb for a video, we're not *really* doing <a href="video"><img src="thumb"></a>, we're using a hack that works only for video-(and-audio?)-with-static-image-thumbnails: <video src="video" poster="thumb">.

To be clear, that's not something that the legacy parser does, it was just something I did in the Parsoid implementation because it seems reasonable and I hadn't explored how manualthumb worked in the timedmedia case.