Page MenuHomePhabricator

Edge case bug in AttributeExpander
Closed, ResolvedPublic

Description

See this transcript:

$ cat /tmp/wt
{{1x<invalid>
|{{2x<invalid>y}}|
}}

{{1x<invalid>
|{{2x<invalid>
y}}|
}}

{{1x<invalid>
|{{2x<invalid>
y}}
}}
$ php bin/parse.php < /tmp/tpiwt 
<p data-parsoid='{"dsr":[18,53,0,0]}'>{{1x&lt;invalid>
|{{2x&lt;invalid>y}}|
}}</p>

<p data-parsoid='{"dsr":[55,73,0,0]}'>{{1x&lt;invalid>
|
}}</p>

<p data-parsoid='{"dsr":[75,110,0,0]}'>{{1x&lt;invalid>
|{{2x&lt;invalid>
y}}
}}</p>

So the middle wikitext snippet cause the output to be broken. This is because of the 'splitNlTokens' and associated code in AttributeExpander. The first two snippets lead to the tokenizer creating a <td> with complex attributes ("{{2x<invalid>..."). The last one is a non-attribute td and so doesn't break.

Of the first two snippets, the second snippet expands to a set of tokens that has a newline which triggers the splitTokens code in AttributeExpander which just messes up the <td> token, and when the TSP kicks in later, it drops a bunch of content based on the output of the AttributeExpander.

This effectively is the reason for Parsoid's broken output on https://tpi.wikipedia.org/wiki/Templet:Convert/LoffAonSoff?useparsoid=1

Event Timeline

For the tpiwiki page, an equivalent reduced case would be:

{{1x<noinclude>{{{A}}}</noinclude>
|{{2x<noinclude>{{{B}}}</noinclude>
}}|
}}

Change #1180238 had a related patch set uploaded (by Subramanya Sastry; author: Subramanya Sastry):

[mediawiki/services/parsoid@master] WIP: Don't apply nl-splitting for non-<table> tokens

https://gerrit.wikimedia.org/r/1180238

Change #1180238 merged by jenkins-bot:

[mediawiki/services/parsoid@master] AttributeExpander: Disable nl-splitting for td/th/caption tokens

https://gerrit.wikimedia.org/r/1180238

Change #1181757 had a related patch set uploaded (by C. Scott Ananian; author: C. Scott Ananian):

[mediawiki/vendor@master] Bump wikimedia/parsoid to 0.22.0-a18

https://gerrit.wikimedia.org/r/1181757

Change #1181757 merged by jenkins-bot:

[mediawiki/vendor@master] Bump wikimedia/parsoid to 0.22.0-a18

https://gerrit.wikimedia.org/r/1181757