Page MenuHomePhabricator

Template expansion via expandtemplates API or action=raw fails for templates inside tag extensions (such as <ref>)
Open, NormalPublic

Description

Author: volker.haas

Description:
The problem is that the expandtemplates API call returns with some unexpanded
templates:

Minimal Example:
http://en.wikipedia.org/w/index.php?title=User:Volker.haas/TemplateExpansion&oldid=544039610

API Call (without revision info of article...):
https://en.wikipedia.org/w/api.php?action=expandtemplates&format=jsonfm&text={{User:Volker.haas/TemplateExpansion}}

Result:

{
"expandtemplates": {
"*": "blub<ref>{{Structurae|id=s0015184|title=Padma >Bridge|nolink=1}}, [[Padma Bridge|Report on the official hearing on construction >progress]]<\/ref>\n\n== References ==\n<div class=\"reflist >references->column-count references-column-count-2\" style=\"-moz-column-count: >2; -webkit-column-count: 2; column-count: 2; list-style-type: decimal;\">\n<references group=\"\"><\/references><\/div>"
}
}

This issue is criticial for the PDF rendering software (Extension:Collection and mwlib), since it now uses the API for expanding templates


Version: 1.21.x
Severity: critical
See Also:

Details

Reference
bz46115

Event Timeline

bzimport raised the priority of this task from to High.Nov 22 2014, 1:16 AM
bzimport added a project: MediaWiki-Parser.
bzimport set Reference to bz46115.
bzimport added a subscriber: Unknown Object (MLST).
bzimport created this task.Mar 14 2013, 2:01 PM

With action=parse&prop=text&page=User:Voker.haas/TemplateExpansion

http://en.wikipedia.org/wiki/Special:ApiSandbox#action=parse&format=json&page=User%3AVolker.haas%2FTemplateExpansion&prop=text

So I guess you want to use action=parse&prop=text&page= instead of action=expandtemplate.

ralf_wikimedia wrote:

Does the above call expand templates and give me mediawiki markup? If not, that's not what we want...

Ah no, that render them :-] Sorry.

He7d3r added a comment.EditedMar 16 2013, 12:53 PM

Compare this with the behavior described on T30616: Special:ExpandTemplates - Preview seems to be parsed twice.

The problem is that "expandtemplates" is a bit of a hack: it half-parses the wikitext to get new wikitext. But this half-parsing can't include calling tag hooks, because the tag hook returns HTML rather than wikitext. And it can't necessarily parse the text passed to the tag hook, because the tag hook might do almost anything with it (e.g. <syntaxhighlight> doesn't expand any templates, <ref> does, and it's theoretically possible to create a tag hook that would run inline Lua code or the like. So it just outputs the tag contents as-is.

It sometimes breaks in other ways, too. Consider, for example, what it does to something like "{{#tag:ref|Ref A<ref>Ref B</ref>}}".

Bug 28616 is similar: Special:ExpandTemplates half-parses the wikitext, and then parsed the half-parsed wikitext. This two-step parse can give different results.

ralf_wikimedia wrote:

*** Bug 50090 has been marked as a duplicate of this bug. ***

ralf_wikimedia wrote:

*** Bug 48872 has been marked as a duplicate of this bug. ***

TTO added a comment.Aug 25 2013, 10:39 AM
  • Bug 53260 has been marked as a duplicate of this bug. ***

Is this really still high priority (priority not changed for two years) now that PDF rendering is done by OCG?

Bultro added a comment.Jun 9 2015, 9:27 AM

The merged and blocked tasks have been solved by the introduction of the new PDF generation system last year.

Either this task is solved too, or they have to be unmerged

Aklapper lowered the priority of this task from High to Normal.Jun 9 2015, 10:13 AM
Tgr renamed this task from Template expansion via API fails to Template expansion via expandtemplates API or action=raw fails for templates inside tag extensions (such as <ref>).Aug 5 2016, 9:08 PM
Tgr added a subscriber: Tgr.

Not specific to the API (unsurprisingly, since it is a parser issue), the same problem occurs with action=raw&templates=expand.

Pppery added a subscriber: Pppery.Sep 24 2017, 9:51 PM

The problem is that "expandtemplates" is a bit of a hack: it half-parses the wikitext to get new wikitext. But this half-parsing can't include calling tag hooks, because the tag hook returns HTML rather than wikitext. And it can't necessarily parse the text passed to the tag hook, because the tag hook might do almost anything with it (e.g. <syntaxhighlight> doesn't expand any templates, <ref> does, and it's theoretically possible to create a tag hook that would run inline Lua code or the like. So it just outputs the tag contents as-is.
It sometimes breaks in other ways, too. Consider, for example, what it does to something like "{{#tag:ref|Ref A<ref>Ref B</ref>}}".
Bug 28616 is similar: Special:ExpandTemplates half-parses the wikitext, and then parsed the half-parsed wikitext. This two-step parse can give different results.

It's worse than that. Templates, under normal circumstances, shouldn't be expanded in <nowiki> tags (obviously). But, if those nowiki tags are passed as an argument to [[Module:Preprocess]] on enwiki, or some other module that does something similar, then the templates do need to be expanded.