Page MenuHomePhabricator

Flow <pre> tags garbled in old posts
Closed, DeclinedPublic

Description

(First off, the usual disclaimer - I know Flow is very janky and in the process of being undeployed, and don't really care if this is deemed not a problem and not worth fixing - I'll just work around it in the Flow export script I'm using, filing for the record anyway)

The Flow API seems to be garbling <pre> tags when asked to return wikitext in some situations. For example, https://www.mediawiki.org/wiki/Special:ApiSandbox#action=flow&format=json&submodule=view-topic&page=Topic%3AStixxiicok0zbyzn&formatversion=2&vtformat=wikitext says that the content of the opening post of https://www.mediawiki.org/wiki/Topic:Stixxiicok0zbyzn converts to wikitext as:

Would somebody please add for MyWikis, they now offer VisualEditor:

&lt;pre&gt;
| [http://www.mywikis.com/ MyWikis]
| USA
| Yes
|
| no
| paid
| Yes (included)
| FTP and SSH
| Ticketed support system, email, and phone
| Yes
| Unlimited storage, unlimited bandwidth. VisualEditor available with MyWikis Pro and above plans.<br />[https://www.mywikis.com/hostedparsoid A dirt-cheap hosted Parsoid plan] is available to non-MyWikis wikis.
&lt;/pre&gt;

If you would like to corroborate this with MyWikis Support, please feel free to email them at contact [at] mywikis [dot] com. Thanks!

Even though if you look at the posts the <pre> tags shouldn't be escaped - it renders with HTML <pre>. You can also see this when editing the post as wikitext. You can also see this as https://www.mediawiki.org/wiki/Topic:Ritum7nezwfd3g6a, which seems to have gotten garbled during the LQT->Flow import, and is now getting garbled again when its content is exported as wikitext.

This can also be seen when editing the posts using the wikitext editor.

Event Timeline

Restricted Application added a subscriber: Aklapper. · View Herald Transcript
Pppery updated the task description. (Show Details)

The saved html that's being serialized predates emitting multiline-pre (ie. not inline-pre) as an extension tag, with mw:Extenstion/pre and data-mw->body->extSrc,
https://github.com/wikimedia/mediawiki-services-parsoid/commit/79ccfb9372cb57afa569036ef39ead13abfba673

so it has stx: html as a literal tag marker,

<pre data-parsoid="{&quot;stx&quot;:&quot;html&quot;,&quot;strippedNL&quot;:true,&quot;dsr&quot;:[69,436,5,6]}">| [http://www.mywikis.com/ MyWikis]
| USA
| Yes
|
| no
| paid
| Yes (included)
| FTP and SSH
| Ticketed support system, email, and phone
| Yes
| Unlimited storage, unlimited bandwidth. VisualEditor available with MyWikis Pro and above plans.&lt;br /&gt;[https://www.mywikis.com/hostedparsoid A dirt-cheap hosted Parsoid plan] is available to non-MyWikis wikis.
</pre>

We've more recently added escaping for literal html tags that mask extension tags,
https://github.com/wikimedia/mediawiki-services-parsoid/commit/362648790a718454b6cbf65477367e53ea09ae98

There's a couple ways a script could work around the issue. Dropping the stx: html from data-parsoid would serialize as indent-pre,

| <nowiki>[http://www.mywikis.com/ MyWikis]</nowiki>
| USA
| Yes
|
| no
| paid
| Yes (included)
| FTP and SSH
| Ticketed support system, email, and phone
| Yes
| Unlimited storage, unlimited bandwidth. VisualEditor available with MyWikis Pro and above plans.<nowiki><br /></nowiki><nowiki>[https://www.mywikis.com/hostedparsoid A dirt-cheap hosted Parsoid plan]</nowiki> is available to non-MyWikis wikis.

From https://github.com/wikimedia/mediawiki-services-parsoid/commit/6d658a4844d6a71cae8a2c2a556d71a999d216a7 it could add autoInsertedEnd: true to data-parsoid and that would give,

<pre>
| [http://www.mywikis.com/ MyWikis]
| USA
| Yes
|
| no
| paid
| Yes (included)
| FTP and SSH
| Ticketed support system, email, and phone
| Yes
| Unlimited storage, unlimited bandwidth. VisualEditor available with MyWikis Pro and above plans.<br />[https://www.mywikis.com/hostedparsoid A dirt-cheap hosted Parsoid plan] is available to non-MyWikis wikis.

which is better but missing the closing tag.

First off, the usual disclaimer - I know Flow is very janky and in the process of being undeployed, and don't really care if this is deemed not a problem and not worth fixing - I'll just work around it in the Flow export script I'm using, filing for the record anyway)

We could remove the requirement that autoInsertedEnd: true be set for pre and it should work. I'll let you decide if you want us to deploy that.

Generating wikitext without a closing tag would be worse for my purposes, given that the script I'm runnng calls the Flow API to produce multiple distinct blocks of wikitext and concatenates them together (as well as doing other modificatons of the content in a lossy way that I suspect you wouldn't approve of). The code is at https://gitlab.wikimedia.org/pppery/flow-export-with-history. And it tells the Flow API to produce Wikitext (for all but one irrelevant edge case) rather than talking to Parsoid itself, so any on-the-fly mutations of the HTML can't be stored there and would need to be in the Flow repository.

I'll probably just add another lossy transformation to deal with this sumewhere in the convertTopic function - the goal of the script is not to produce a 100% faithful representation of what the Flow board looked like (which is impossible), but instead to extract the content into something like what it would have looked like if the same discussion had taken place on a wikitext talk page instead, and (importantly) to ensure people's edits still show up in contributions etc.

Worked around in https://gitlab.wikimedia.org/pppery/flow-export-with-history

And probably nothing other than that script cares, so I'm going to just decline this. Thanks for investigating nevertheless.