Page MenuHomePhabricator

Parsoid native template expansion can't handle the SOL newline hack
Open, LowPublic

Description

This should expand as two list items:

{{1x|#foo}}{{1x|#foo}}

This works correctly in Parsoid production when it uses the expandtemplates API call, because expand templates (the legacy preprocessor) returns "\n#foo" in each case and then we tokenize in correct start-of-line context.

But in Parsoid's native template expansion (as used in parserTests, for example), the argument #foo is tokenized in non-SOL context, and templates are always expanded in SOL context. For a template which starts with #foo this is correct, but in this case the template starts with {{{1}}}} and we substitute in the already-tokenized-in-non-SOL-context argument to result in the literal string "#foo".

We'd like to eventually move away from the two-stage preprocess/parse pipeline (because that introduces lots of weird corner cases where the preprocessor and parser interpret markup in different ways, see T172306: Broken wikilinks can be parsed as wikilinks after preprocessing). If/when we do so, we'll need to somehow handle this case natively; probably by either reparsing the argument in SOL context, deferring argument tokenization until the context is known, or replacing the T14974: The newline added to a template, magic word, variable, or parser function that returns line-start wikicode formatting (*#:; {|) causes unexpected parsing hack (introduced to resolve T2529) with a semantically-simpler mechanism that doesn't expose preprocessor internals.

Event Timeline

cscott created this task.Dec 19 2019, 4:19 PM
Restricted Application added subscribers: Liuxinyu970226, Aklapper. · View Herald TranscriptDec 19 2019, 4:19 PM

Change 559542 had a related patch set uploaded (by C. Scott Ananian; owner: C. Scott Ananian):
[mediawiki/services/parsoid@master] Test case for implicit newline insertion when expanding templates

https://gerrit.wikimedia.org/r/559542

Change 559542 merged by jenkins-bot:
[mediawiki/services/parsoid@master] Test case for implicit newline insertion when expanding templates

https://gerrit.wikimedia.org/r/559542

LGoto moved this task from Needs Triage to Backlog on the Parsoid board.Feb 15 2020, 12:05 AM
LGoto triaged this task as Low priority.Feb 21 2020, 5:17 PM
LGoto moved this task from Backlog to Tech Debt / Big changes on the Parsoid board.