This normalisation is done by Parsoid, but seemingly not documented here: https://www.mediawiki.org/wiki/Parsoid/Normalizations nor do I remember it being added.

In T184755#3896218, @Esanders wrote:

This normalisation is done by Parsoid, but seemingly not documented here: https://www.mediawiki.org/wiki/Parsoid/Normalizations nor do I remember it being added.

I am confused now ... I didn't think this was Parsoid related. I still don't have a reproducible example of what the complaint is (from https://www.mediawiki.org/wiki/Topic:Tuhfcu8edr2h8xso )

Reproduction should be very easy:

1. Open the visual editor.
2. Type something. Press return ten times. Type something else.
3. Save page.
4. Discover that your ten blank lines are now just one.

Thanks. Yes, I see that now. So, it is not some edge case behavior.

[subbu@earth tools] echo "<p>foo</p><p></p><p></p><p></p><p>bar</p>" | parse.js --html2wt
foo

bar

I was mistaken about it on the flow thread then. This is a Parsoid bug. Looks like this has been the behavior in Parsoid since the very beginning since there isn't any special normalization we are doing for this.

T175421: Parsoid doesn't insert same spacers when article text starts with two or more newlines. may be related (at least in the sense that it could be fixed at the same time, if not actually due to the same cause).

@Esanders, so, in trying to fix this bug in Parsoid, I was playing around with HTML snippets in the browser and I looked at the following three:

<h1>br tags wrapped in p tags</h1>
<p>foo</p>
<p><br/><br/></p>
<p>boo</p>

<h1>br tags</h1>
<p>foo</p>
<br/><br/>
<p>boo</p>

<h1>Empty p-tags</h1>
<p>foo</p>
<p></p>
<p>boo</p>

When I look at the rendering in the browser, only the first 2 snippets generate the necessary whitespace. The browser suppresses the empty paragraph effectively. Given that, shouldn't VE be generating br tags instead of empty p-tags?

But, one simple solution to tackling this in Parsoid would be to implement a normalization wherein empty p-tags are normalized to <br/> tags. While we can do that relatively easily, is there a reason why VE is generating empty p-tags when editors insert empty lines?

Change 465656 had a related patch set uploaded (by Subramanya Sastry; owner: Subramanya Sastry):
[mediawiki/services/parsoid@master] WIP: Normalize DOM to convert empty <p></p> tags to <p>?<br/></p>?

https://gerrit.wikimedia.org/r/465656

If we need to output <p><br></p> instead of <p></p> in the HTML we send to Parsoid, then I suppose we can do that.

But I'm not sure if VisualEditor or Parsoid is the better place to fix this.

In T184755#3991853, @ssastry wrote:

<h1>Empty p-tags</h1>
<p>foo</p>
<p></p>
<p>boo</p>

When I look at the rendering in the browser, only the first 2 snippets generate the necessary whitespace. The browser suppresses the empty paragraph effectively. Given that, shouldn't VE be generating br tags instead of empty p-tags?

I will note that this applies to the following snippets as well. The empty element has no visible rendering (due to margin collapsing):

<h1>foo</h1>
<h1></h1>
<h1>boo</h1>

<pre>foo</pre>
<pre></pre>
<pre>boo</pre>

<blockquote>foo</blockquote>
<blockquote></blockquote>
<blockquote>boo</blockquote>

But, we probably shouldn't be inserting <br> tags inside empty elements other than <p>, because apparently they are not ignored by Parsoid (and serialize to <br> in wikitext)? This behavior is very difficult for me to understand.

Also… when Parsoid parses a bunch of newlines in wikitext and outputs a bunch of <p><br></p> structures for them, it should mark the <br> node somehow so that we can ignore it in VisualEditor. Currently VisualEditor treats it as any other <br> tag and allows inserting content on either side of it (essentially, rendering it as two empty lines instead of one).

Change 465761 had a related patch set uploaded (by Bartosz Dziewoński; owner: Bartosz Dziewoński):
[VisualEditor/VisualEditor@master] [DNM] Output <br> inside empty <p></p> in Parsoid HTML

https://gerrit.wikimedia.org/r/465761

In T184755#4656385, @matmarex wrote:

But, we probably shouldn't be inserting <br> tags inside empty elements other than <p>, because apparently they are not ignored by Parsoid (and serialize to <br> in wikitext)? This behavior is very difficult for me to understand.

You are right, my original reasoning on relying on how HTML renders those is flawed since none of those empty tags render in HTML either.

Refreshing my memory about the code, the reason paragraphs seem to be special is because p-tags map to newlines and we normalize newline counts to ensure core semantics are preserved. More specifically, newlines in the wt -> html direction map to either <p><br/></p> or <p><br/>some content here </p>. So, in the html -> wt direction, Parsoid expects <br/> to force extra newlines. The absence of <br/> is causing Parsoid to normalize newline counts. I think this behavior is correct since it is a mirror of what the wt -> html generates.

So, the qn. is if Parsoid should accept <p></p> and normalize it to the expected form (my patch) or should VE return HTML that matches Parsoid's expectations. Given that Parsoid normalizes other empty forms to what the core html -> wt code expects, I suppose it is reasonable to have Parsoid normalize it to either <p><br/> (1 newline) or <p><br/></p> (2 newlines), i.e. be more liberal about how it treats variations from canonical HTML. Thoughts?

Relatedly, but not directly related to this patch, this once again highlights missing documentation / spec about canonical HTML Parsoid expects (for canonical wikitext output) and how it handles variations from that. At our recent offsite, we discussed document a html -> wt spec (not just wt -> html spec that we currently publish) that makes explicit and documents this behavior and identifies inconsistencies that can be fixed.

Change 465656 merged by jenkins-bot:
[mediawiki/services/parsoid@master] Scrub DOM to convert <p></p> sequences to NL-emitting normal forms

https://gerrit.wikimedia.org/r/465656

Change 465761 abandoned by Bartosz Dziewoński:
[DNM] Output <br> inside empty <p></p> in Parsoid HTML

Reason:
No longer needed after https://gerrit.wikimedia.org/r/c/mediawiki/services/parsoid/ /465656

https://gerrit.wikimedia.org/r/465761

Mentioned in SAL (#wikimedia-operations) [2018-11-28T21:28:30Z] <arlolra> Updated Parsoid to 18a98af (T209236, T210437, T184755, T187142, T208470, T207286, T206777, T205710, T205546, T204477)