We can simply move it to ParserOutput as one more derived output property and have clients do what they want to do with it. Between Platform Engineering and us, we are going to be working on T287216 which effectively will create a base class / interface for the current ParserOutput class and that would be a good time to slot it there.

Thanks for the reduced test case. Here are example scenarios that all merit handling the same way.

: <div>Hi
:: Ho</div>

But, looks like if we are going to down this class alias route to support Dodo, we will need to figure out how to address non-Parsoid-repo uses of DOMCompat class from Parsoid's repo.

This is not on production yet. So, all good there. But yes will impact beta cluster. Will make sure @cscott sees this.

Okay, tagged a new version of Parsoid as -a10 and that seems to have fixed the problem.

I am confused by these failures. That reverted patch on vendor should have completely removed all references to Dodo from vendor, so i don't know why those CI jobs are even running into Dodo at all!

This is the domain of the html2html transformations piece (see these preliminary slides -- sorry non-WMF folks, I'll work to make these accessible soonish by updating them and uploading them somewhere accessible). The question of where html2html transforms ought to sit and what is cached or not and whether they are extensions, services, or in core or in Parsoid is all up for discussion. Whether there ought to be a framework / libraries to support ease of spinning these up is also up for discussion.

I am in the bikeshedding mode, and in that vein, here goes:

• WikitextOutput -- doesn't work because there are other content models
• PageOutput -- doesn't work because you can have page-independent content transformation
• Output -- doesn't work because it is far too generic
• PageBundle -- Parsoid already has this concept but probably misnamed for the same reason PageOutput is not
• ContentBundle -- this or probably Content + some synonym of Bundle ?

We need do a quick name bikeshedding for this and maybe not call it ParsoidOutput.

Any reason not to decline this.? This is probably a wontfix and a waste of time to try to sync output.

In T287001#7224300, @Jdforrester-WMF wrote:

Might have been caused by https://gerrit.wikimedia.org/r/c/integration/quibble/+/703182 which could have changed the way Parsoid requests made by Flow go, possibly?

@Jgiannelos should be able to devote some time to this next week -- the team has been a bit heads down on the maps rollout preparations and trying to cram things in between availability of others that are needed for this. Thanks for the heads up about your requirements around this.

Sample test case based on a metawiki page that triggered this "warning" in rt-testing:

<p>                                                                                                      
<translate> foo</translate>                                                                              
</p>

Ah, ok.

In T286786#7217633, @Arlolra wrote:

We added some logging for this in T280449

https://logstash.wikimedia.org/app/discover#/doc/logstash-*/logstash-mediawiki-2021.07.13?id=S-v8oXoBCxLmWkI6RaEy

In T286402#7207555, @matmarex wrote:

All of the en.wp examples involve using {{tq2|{{#section:WT:COUNTRIES|summary}}}} inside nested lists (someone really likes quoting that policy), which I guess ends up generating misnested <blockquote> tags and wikitext list items. Reduced test case: https://en.wikipedia.org/w/index.php?title=User:Matma_Rex/sandbox&oldid=1033331490. It renders wrong in the old parser too!

At first glance, my hunch is that it could potentially be bad DSR values - curious is there is broken wikitext in any of those lists. Will take a look later this week.

Aha .... curious why we have that exception there for specific node names.

But, this should impact any stripped closing tag in a list item, not just in this misnested scenario ... so there is that piece as well.

The placeholder has zero width there, so, the DSR should line up properly.

The expanded wikitext is This user has [[m::Special:MyLanguage/Main Page|near native speaker]] knowledge of [[m::Category:User_en|English]]. and so Parsoid doesn't recognize the m:: prefix as a valid interwiki prefix whereas the core parser seems to let the :: pass.

There is misnested <i><b>...</i></b> (HTML tags, not wikitetx) in there and I suspect something is broken with DSR computation in the wt->html direction causing this. This is some edge case but my recommendation is to just fix up the wikitext and the bad nesting.

Reopening to follow up on the failure to fully serve all the static files.

In T73962#4466772, @Izno wrote:

I was wondering if perhaps Parsoid handles this correctly if it's the p-wrap logic at fault.

T268785: IDEA: Move parallel tag parsing logic from Math to core is related.

In T284921#7156629, @Nirmos wrote:

In T284921#7156564, @ssastry wrote:

span.tocnumber { display: none; } should do it right?

Nah, this has to be a misunderstanding. This task has nothing to do with the toc.

span.tocnumber { display: none; } should do it right?

I support removing many of these rarely used user prefs that cause HTML to vary by these prefs. But, independent of that this one is easily controlled by CSS without modifying the HTML.

Also see this:

[subbu@earth:~/work/wmf/parsoid] echo "<table></table>" | php ../core/maintenance/parse.php 
parse.php: warning: reading wikitext from STDIN. Press CTRL+D to parse.

These utf-8 errors are caused by a combination of things: either bad encoding stored in the db, "bad markup", or real bugs in Parsoid. These errors helped us find and fix most of these. The remaining ones are still some lingering ones we haven't investigated because there are far fewer of them now. But, yes I agree that it is probably time to migrate these remaining errors to a different log channel. We'll chat about this in the coming week.

Good sleuthing! Didn't realize envoy and parsoid-async timeouts were involved in the picture. That will let us proceed with T244609 where I had landed at the wrong place assuming the problem was in Parsoid's PEG.

@kostajh, sorry this mail got lost in my post-vacation volume. But, please scope this and if necessary, we can schedule a review meeting to see which pieces need which of us to work on and where the components ought to live.

For the benefit of train engineers, I wanted to note that these UTF-8 errors seen in Parsoid are expected to come up in spurts (lots of errors on the same page OR repeated errors because of retries). So, unless there is a significant spike, this shouldn't be a concern for deployments.

Indeed. Thanks for flagging me. Approved.

In T261181#7137532, @Nikerabbit wrote:

@ssastry T274881: Change translation variable (tvar) syntax is now done pending QA. Does that unblock your work? Is there anything else we should do on our side?

@Volans let me know if you need anything from me as @ihurbain's manager.

Both those scenarios will be alleviated by addressing T211946: Bump parsoid resource limits in production.

I couldn't reproduce readily with a bunch of simplified test cases on the command line. So, this looks like some edge case specific to this page, if at all.

The reason I filed it then (and it is being filed repeatedly again) is that this was a change in behavior - this was not being logged before at all. So, did something change in core around request timeouts? Or did something change in logging?

Anyway, let us please end this thread here. We'll take a look at your specific usecase and see what we can do.

I appreciate and welcome your input but was mostly countering your assertion that your viewpoint and experience is representative of small and medium-sized wikis or that we are ignoring small and medium-sized 3rd party wikis in general. When smaller wikis are behind by several versions, a proposed shift in a future LTS doesn't really count as "an overnight shift" nor as a "slap in the face". Please be careful with your charged statements.

In T236809#7076199, @RobinHood70 wrote:

So, hopefully you can understand that migrating to something that will essentially break all of that isn't just a pain point for us, it's simply not a viable option.

In T236809#7076129, @MGChecker wrote:

If this is a real blocker, then, we'll have to introduce some kind of page property OR some kind of extension config option that will force full sequential reparse for any page that has that flag or extension config set.

To me that sounds like the Parsing Team is open to modifying Parsoid to support your use case.

I would really appreciate clarification on this point, since right above it is stated "However, if extensions introduce state that depends on the page parsed in linear order, yes, we cannot support that.". At the time I read this, this lead me to the conclusion that this is more of a theoretical possibility which has no real chance to be implemented.

But rereading this again, this might actually be a misunderstanding.

None of my comments above and here is to engage with any of the criticism of us making WMF-centric decisions. That is a separate issue and I am not going to wade those waters here. I would like to focus my attention and discussion on the specific issues of wikitext engines in MediaWiki and their future and the migration paths.

As a general principle, we cannot avoid breaking changes or technical upgrades just because there are wikis that use some specific set of features or use them in a particular way.

Awesome! Thanks Arlo for digging. I think we can skip a new security release for Parsoid/JS. If necessary, we can email a security advisory with how to fix this (upgrade package.json and npm update). I cannot wait for the day when Parsoid/JS goes EOL.

Interesting .. I think Parsoid's behavior is the right one here (what if you want to add new content and convert that from an "empty" section to a non-empty one), but that does no one any good since the core notion of sections is what is used by section-edit links.

Good questions.

We are realistically not going to do all this work before RC0. I am going to remove this as a blocker for MW 1.36 release. We should retarget the tasks for 1.37 and be more realistic and about what we intend to really work on for that release.

Yes, I noticed. I expect it is some edge case where the conditions for whitespace reuse were obviously not satisfied. I'll ignore this for now unless we see more of them showing up.

Parsoid doesn't have this bug:

[subbu@earth:~/work/wmf/parsoid] php bin/parse.php --normalize < /tmp/wt

See T272987 - this won't be resolved right away, but yes, @cscott has successfully advocated for that to be one of the first Parsoid read view use cases.

Looks like you are needing a lot of detailed guidance and support in some of this work. I recommend you try to ensure your MediaWiki is set up properly with Parsoid configured properly. And, then, you need to ensure that the XML dump is properly imported (including all templates, modules, and necessary extensions). Then, you can fetch the HTML for it from your MediaWiki install. If you are having trouble doing this setup, please find the appropriate support forum (maybe someone can point you to them). We aren't really set up to provide this level of support. In any case, Phabricator is not the right forum for this level of troubleshooting / MediaWiki support.

In T280246#7002961, @DungLe94 wrote:

My ultimate goal is to parse the Wiki xml dump into HTML. There are many parsers available, but Parsoid is of course the best. The resulted HTML will allow me to create contents adapted to different devices and read on different software (GoldenDict, Google Chrome, Microsoft Edge, Mdict, ...). I'm sure that a lot of people will benefit from this. My short-term goal is to build a dictionary for GoldenDict from Wiktionary dump.

Oh, none of us had reviewed it. +2ed it now. :-)

In T280246#7002927, @Reedy wrote:

reedy@ubuntu64-web-esxi:/var/www/wiki/mediawiki/core/vendor/wikimedia/parsoid$ echo 'Foo' | MW_INSTALL_PATH=/var/www/wiki/mediawiki/core php bin/parse.php --integrated
Unexpected option integrated!

In T279867#6991264, @matmarex wrote:

Possibly related issue: T248039

Parsoid doesn't have this bug, so one less thing to worry about there.

Actually, I am not sure this is right. This time around (compared to Mar 2020 when I seem to have hastily reponded), I actually tested this. See https://www.mediawiki.org/w/index.php?title=User:SSastry_(WMF)/Sandbox&oldid=4525962 ...