In T195486#4229457, @gerritbot wrote:

Change 434961 had a related patch set uploaded (by Subramanya Sastry; owner: Subramanya Sastry):
[mediawiki/services/parsoid@master] Minimize whitespace dirty diffs in existing headings, tables, lists

https://gerrit.wikimedia.org/r/434961

In T194806#4229321, @ssastry wrote:

In T194806#4229268, @Arlolra wrote:

So, we should probably try to fix this.

By "this", I'm going to assume you mean the paragraph wrapping of the abc. The rest of the render looks right to me.

Yes, p-wrapping.

In T194806#4229268, @Arlolra wrote:

So, we should probably try to fix this.

By "this", I'm going to assume you mean the paragraph wrapping of the abc. The rest of the render looks right to me.

In T195263#4227336, @Jdforrester-WMF wrote:

Given that there are by definition no reader-facing pages on Wikidata that have wikitext on them (all the reader-facing pages are structured data pages), converting it over to Remex seems fine. :-)

In T195414#4228628, @ssastry wrote:

In T195414#4227237, @Arlolra wrote:

Btw, just want to draw attention to this comment,

16:43 <+arlolra> why is it that new heading are added with the spacing
16:43 <+arlolra> https://en.wikipedia.org/w/index.php?title=Blinn_College&curid=1246005&diff=842658946&oldid=842657772&diffmode=source
16:44 <+arlolra> it doesn't seem like we should be adding that if we're not going to preserve it

No, these two things are somewhat orthogonal. (a) Whitespace trimming in wikitext -> html is wikitext spec (b) Editors / wikis want whitespace in lists, headings, tables, etc. for readability and so, new content should continue to add them.

The question however is what do we do about headings /list items / table cells that were already present in original wikitext. That is where this gets tricky. There is no good automatic solution. I'm going to file a separate ticket for it.

In T195414#4227237, @Arlolra wrote:

Btw, just want to draw attention to this comment,

16:43 <+arlolra> why is it that new heading are added with the spacing
16:43 <+arlolra> https://en.wikipedia.org/w/index.php?title=Blinn_College&curid=1246005&diff=842658946&oldid=842657772&diffmode=source
16:44 <+arlolra> it doesn't seem like we should be adding that if we're not going to preserve it

In T195414#4227219, @ssastry wrote:

This also explains the mysterious html2wt perf degradation since mid-December that I noticed recently.

https://grafana.wikimedia.org/dashboard/db/parsoid-timing-html2wt?panelId=26&fullscreen&orgId=1&from=now-1y&to=now

There is a noticeable upward bump around mid-December ... and guess what ... html5 ids with fallbacks was deployed on December 12.

This also explains the mysterious html2wt perf degradation since mid-December that I noticed recently.

As expected, it is the fallback ids (!).

All these test cases are fixed with that patch there.

Yes, I think so.

In T187951#4215550, @ssastry wrote:

So, this is a bug introduced by database changes (or a bug elsewhere exposed by those database changes). Looking at https://www.mediawiki.org/wiki/MediaWiki_1.32/wmf.1#Core_changes, there are 6 RDBMS related entries there. So, @aaron or @Krinkle should be able to take it from here.

See https://logstash.wikimedia.org/goto/75432740fdaa6f43376ce203b0956d07 -- that is a search for officewiki events that are not INFO events for the April 1 - May 18 time period. You can see that the event graph starts around April 18th.

In T187951#4212895, @Quiddity wrote:

It's definitely been a problem for more than 6 months. This bug was filed
in Feb, but Sherry and I were experiencing it for longer.

@Deskana, @Esanders maybe you guys can see if perf / platform team folks might have any insight into this at the hackathon. I'll separately poke at instrumenting / tracing parsoid monday.

See timings from about April 19th to May 2nd. You will see the shift happen around Apr 25.

Ok, so I got curious and grepped older logs .. and it seems like this behavior started the week of Apr 24.

So, here is what I found so far over the last hour:

1. When I copied over my officewiki cookie to wtp1025 and use that cookie to run a local commandline parse of officewiki pages, they are consistently parse very quickly
2. When I restrict myself to a single small page (say User:Ssastry), and if I don't actually make an edit, the page opens in VE quickly almost very consistently
3. When I restrict myself to a single small page (say User:Ssastry), and I click Edit Source (and not do anything else), click Read/History, and open the page in VE, it is once again slow (for 2 attempts), and back to being fast again.
4. There are also parse time interactions when multiple pages are open in different tabs.

So, when I hacked my local parsoid to hardcode my cookies into the code and parse the officewiki pages, they all complete in 2s or less. So, something specific to the production setup. Will investigate tomorrow.

In T187951#4212589, @ssastry wrote:

Deskana: all the pages (included the SEO log) page loaded instantaneously, but I think that is because your loads had already cached it.

In T187951#4179512, @Whatamidoing-WMF wrote:

I'm having this problem in Safari. I think that the second edit on a page (in the same session) may be the problem.

I think there is something to this here. To test this, I edited https://office.wikimedia.org/wiki/User:Ssastry and the next load took 20+ seconds ( I could repro. this 2 times) .. the 20+ second is suspiciously long and looks like something timing out before the parse succeeds.

Deskana: all the pages (included the SEO log) page loaded instantaneously, but I think that is because your loads had already cached it.

In T187951#4190361, @Deskana wrote:

Somehow, I finally experienced this problem.

The problem is this query (or its equivalent for whatever page you're on): https://office.wikimedia.org/w/api.php?action=visualeditor&format=json&paction=parse&page=User%3ADeskana_(WMF)%2FSEO_project_activity_log&uselang=en

Pasting the URL into my browser, I can see that sometimes it takes excessively long for it to resolve.

See https://de.wikipedia.org/w/api.php?action=parse&page=Vorlage%3ATabMenu%2FDoku%2FTMT%2F1R&format=json to see a stray </div> in there. It comes from {{TabMenu | Mode=EndTabMenu }}

I will update the queries to remove those that have Tidy removed -- so they can continue to be used.

I decided to eliminate the nowikis and instead a whitespace char in all empty cells which is a much better approach.

In T194658#4206052, @ssastry wrote:

Oh hmm .. bad test case.

Oh hmm .. bad test case.

[subbu@earth:~/work/wmf/mediawiki] echo "<table><tr><td>x</td><td data-parsoid='{&quot;stx&quot;:&quot;row&quot;}'></td><td data-parsoid='{&quot;stx&quot;:&quot;row&quot;}'>y</td></tr></table>" | parse.js --html2wt | parse.js --normalize=parsoid

But, if / where subsets are absolutely required, they should be handled systematically and uniformly everywhere (extensions, templates, messages, UI, etc.).

Related: T191939

In T157418#4205222, @Izno wrote:

In T157418#4205078, @ssastry wrote:

Our round trip testing infrastructure compares roundtripped wikitext with original wikitext. In order to figure out which diffs are syntactic and which are semantic, it does diffing with a bunch of normalizations and other tricks. But, those tricks are proving insufficient with the kind of normalization that is happening with this whitespace trimming. So, we need to update our round-trip diffing strategy. In order to not block other Parsoid deployments (since all deployments have to go through round trip testing and not have any unexplainable semantic diffs), I am reverting this patch till then.

Is there a task for that separate work (add as blocker to this task)?

In T157418#4205064, @gerritbot wrote:

Change 433004 had a related patch set uploaded (by Subramanya Sastry; owner: Subramanya Sastry):
[mediawiki/services/parsoid@master] Revert "RFC T157418: Trim whitespace in wikitext headings, lists, tables"

https://gerrit.wikimedia.org/r/433004

In T193685#4200888, @PeterBowman wrote:

I've been checking linter errors on eswiktionary, it should qualify now.

In T194082#4198263, @Arlolra wrote:

This crasher looks like the result of https://github.com/wikimedia/parsoid/commit/58d49a8588b7831af0e3385315f01c72a60b5936

When the sentinel itself is normalized away (it being a text node and as a result of the call to a.normalize()) it is left dangling and eventually throws while trying to addDiffMarks around it.