Converting {{{foo}}} from wikitext to html to wikitext returns a 500 error
Closed, ResolvedPublic

Description

Can be reproduce here: https://restbase-beta.wmflabs.org/en.wikipedia.beta.wmflabs.org/v1/?doc

POST /transform/wikitext/to/html
{wikitext: '{{{foo}}}', bodyOnly: true}
-> <p about="#mwt1" typeof="mw:Param" id="mwAQ">{{{foo}}}</p>

POST /transform/html/to/wikitext
{html: '<p about="#mwt1" typeof="mw:Param" id="mwAQ">{{{foo}}}</p>'}
-> 500 (unknown error)

Note: these transformations work when going to parsoid directly but not when going through restbase.

SBisson created this task.Sep 18 2015, 2:27 PM
SBisson added projects: RESTBase, Parsoid.
SBisson added subscribers: SBisson, Catrope.
Restricted Application added a subscriber: Aklapper. · View Herald TranscriptSep 18 2015, 2:27 PM
GWicke added a subscriber: GWicke.EditedSep 18 2015, 7:24 PM

We discussed this earlier on IRC. This seems to be a bug in the Parsoid serializer when using the v2 API. It fails if some parsoid-specific attributes are present, but data-parsoid is not.

mobrovac moved this task from Backlog to Blocked / others on the RESTBase board.Sep 20 2015, 9:06 PM

It fails if some parsoid-specific attributes are present, but data-parsoid is not.

This just looks like: https://github.com/wikimedia/parsoid/blob/master/lib/mediawiki.WikitextSerializer.js#L680-L685

Since there's !dp.src, it throws.

ssastry triaged this task as "Normal" priority.Sep 21 2015, 3:34 PM
ssastry set Security to None.

To clarify, this isn't v2 API specific. Parsoid doesn't know to serialize parameters substitued at the top-level without the source present in data-parsoid.

What's more, the current rendering doesn't match the spec and the spec doesn't have a representation for the parameter name.

The above example fails because the docs don't tell you to (and enable) pass in the data-parsoid that was produced in the first transformation.

Arlolra claimed this task.Sep 21 2015, 3:46 PM

Change 240059 had a related patch set uploaded (by Arlolra):
Render default part of parameters at the top level

https://gerrit.wikimedia.org/r/240059

Change 240059 merged by jenkins-bot:
Render default part of parameters at the top level

https://gerrit.wikimedia.org/r/240059

This has been deployed according to the deployment log but I can still reproduce the issue in both beta and production.

In T113044#1658853, I mentioned a few things:

  1. Rendering should match the spec
  2. Spec needs to be altered to represent the parameter name
  3. Serialization needs to take into account the above two, and not rely on src in data-parsoid

Only #1 was fixed in that patch.

This seems part of a more broader issue which is that Parsoid is being asked to serialize Flow HTML without access to data-parsoid ( See https://www.mediawiki.org/w/index.php?title=Topic:Sqiakln6zj2vyqg8&topic_showPostId=sqjazy5nr58n9xdc#flow-post-sqjazy5nr58n9xdc for other scenarios besides this bug report). I am going to file a separate bug for it. I am not sure whether this is a Flow issue or a RESTBase issue, but without data-parsoid, we'll get all these normalization effects in some scenarios (entities, mw:Placeholder spans).

Change 246427 had a related patch set uploaded (by Catrope):
Temporarily disable RESTbase support to avoid data-parsoid issues

https://gerrit.wikimedia.org/r/246427

Change 246427 had a related patch set uploaded (by Catrope):
Temporarily disable RESTbase support to avoid data-parsoid issues

https://gerrit.wikimedia.org/r/246427

This patch works around this bug by not going through RESTbase, but the larger issue remains that Parsoid should not throw 500s when it encounters unexpected input.

Change 246427 merged by jenkins-bot:
Temporarily disable RESTbase support to avoid data-parsoid issues

https://gerrit.wikimedia.org/r/246427

Change 249026 had a related patch set uploaded (by Mattflaschen):
Temporarily disable RESTbase support to avoid data-parsoid issues

https://gerrit.wikimedia.org/r/249026

Change 249026 merged by jenkins-bot:
Temporarily disable RESTbase support to avoid data-parsoid issues

https://gerrit.wikimedia.org/r/249026

https://test.wikipedia.org/wiki/Talk:Sandbox is currently broken as a result of this . Sorry. I was trying to test that the workaround of disabling RESTBase worked around this issue. It happened not to be disabled when I did the test.

I should have just checked that data-parsoid was present, rather than doing this test.

There's a change in behaviour on this one. Now submitting <p about="#mwt1" typeof="mw:Param" id="mwAQ">{{{foo}}}</p> to the html/to/wikitext endpoint produces 400 error, and the following log entry pops up in Parsoid logs:

Message: No source for params
Stack: Bad Request: No source for params.
    at Object.<anonymous> (/srv/deployment/parsoid/deploy-cache/revs/bdde184822a6f76d0777a45a6f552c72fb4bf5dd/src/lib/html2wt/DOMHandlers.js:1408:12)
    at /srv/deployment/parsoid/deploy-cache/revs/bdde184822a6f76d0777a45a6f552c72fb4bf5dd/node_modules/prfun/lib/index.js:576:26

So the error is in Parsoid most likely.

There's a change in behaviour on this one.

That was requested in T135596

So the error is in Parsoid most likely.

Yup, as mentioned above in T113044#1672063, parameter representation is incomplete.

https://www.mediawiki.org/wiki/Specs/HTML/1.3.0#Parameter_Substitution_at_the_top-level

Change 328861 had a related patch set uploaded (by Arlolra):
[WIP] T113044: Complete templatearg representation in spec

https://gerrit.wikimedia.org/r/328861

Change 328861 merged by jenkins-bot:
T113044: Complete templatearg representation in spec

https://gerrit.wikimedia.org/r/328861

Arlolra closed this task as "Resolved".Jan 3 2017, 10:21 PM

Note that this still needs to go through rt-testing nnd deploy. This fix also only helps newer HTML generated after the deploy.

And the spec still needs updating ... yes, I was a little hasty, I suppose.