Pre-save transform skips extensions using wikitext (gallery, references, footnotes, Cite, status indicators, pipe trick, subst, signatures)
Open, HighPublic

Tokens
"Burninate" token, awarded by Liuxinyu970226."Burninate" token, awarded by Pols12."Manufacturing Defect?" token, awarded by Boshomi."The World Burns" token, awarded by SamB."Mountain of Wealth" token, awarded by Ciencia_Al_Poder."Love" token, awarded by He7d3r.
Assigned To
None
Authored By
bzimport, Jul 4 2005

Description

Author: dbenbenn

Description:
See [[User:Dbenbenn/sandbox#Gallery_pipe_trick]], for example. A
link in the caption of a gallery image ending in |]] is broken.


Version: unspecified
Severity: major

Details

Reference
bz2700

Related Objects

There are a very large number of changes, so older changes are hidden. Show Older Changes

(In reply to comment #59)

Can I give another example of what I think is the same issue, the inability to
perform the pipe trick within {{#ref:}}

Example that does not work (at enWS)

{{subst:#tag:ref|[[Author:Charles Darwin|]]}}

That will never work— subst only applies to templates and #tag is a magic word. What you are trying is equivalent to <subst:ref> which will not work.

What will never work? The subst: expansion of the tag? Seems to work perfectly for me and has been for a while.

My issue is that the piping trick[1] stops working when it is inside either the <ref> or {{#tag:ref|...}}

[1] http://en.wikipedia.org/wiki/Help:Pipe_trick

I just noticed that subst isn't working inside of <ref> tags:
https://pt.wikipedia.org/w/index.php?diff=prev&oldid=28035465

I believe this is the same bug.

Pipe trick doesn't work with Translate extension popups, I guess it's this bug again. https://translatewiki.net/w/i.php?title=MediaWiki:Bw-desc-freecol/it&diff=prev&oldid=3550365

(In reply to comment #63)

Pipe trick doesn't work with Translate extension popups, I guess it's this bug
again.
https://translatewiki.net/w/i.php?title=MediaWiki:Bw-desc-freecol/it&diff=prev&oldid=3550365

That does not sounds like this, because it is not wrapping within a extension tag. Edit over GUI works for that title, this sounds like a problem with the edit api. You should fill a new bug for that.

Nx.devnull wrote:

Patch to allow extensions to specify a function that is executed during preSaveTransform

This patch introduces a mechanism similar to the tag hooks, the only difference is that the callback function gets called when the output type is set to OT_WIKI (which normally means during preSaveTransform), and that the enclosing tags with the parameters are not removed.

An extension can register a callback and then do pre-save transformation on its content - this is the best solution in my opinion, since only the extension knows which parts of the content are supposed to be transformed, e.g. in the case of the gallery tag only the description part.

This is a work in progress/proof of concept, in particular the galleryPST function calls parser->preSaveTransform, which might cause problems if it breaks the parser state. This will probably need something re-entrant like recursiveTagParse.

All parser tests passed.

Attached:

Thanks for this patch! We've been in a code slush (http://lists.wikimedia.org/pipermail/wikitech-l/2012-January/057313.html) for a
few weeks so we're just getting around to looking at these.

We're also doing a lot of parser work, so I'm not sure how relevant this is,
but I'll ask them to take a look, but I'll still get someone to look.

wicke wrote:

(In reply to comment #65)

An extension can register a callback and then do pre-save transformation on its
content - this is the best solution in my opinion, since only the extension
knows which parts of the content are supposed to be transformed, e.g. in the
case of the gallery tag only the description part.

I am not too enthusiastic about encouraging extensions to do their own arbitrary pre-save transform processing. In the longer run, general support for extensions working with parsed (and pre-save transformed) wikitext / HTML DOM input could be cleaner and should offer better opportunities for integration with a visual editor. I am working on something like that in the Parsoid parser (http://mediawiki.org/wiki/Parsoid). There is also a simple and undocumented method for something vaguely like this in the PHP parser (the setTransparentTagHook method).

However, I do not have a good solution for partly-parsed and -transformed extensions like gallery. Will give this some thought, and ask others about their ideas.

  • Bug 34999 has been marked as a duplicate of this bug. ***

(In reply to comment #67)

I am not too enthusiastic about encouraging extensions to do their own
arbitrary pre-save transform processing.

lowering priority

(In reply to comment #69)

(In reply to comment #67)
> I am not too enthusiastic about encouraging extensions to do their own
> arbitrary pre-save transform processing.

lowering priority

I am not sure how to equate the "lowering the priority" for a significant issue that doesn't follow normal wiki behaviour? We get numerous responses akin to "I don't like it" however no solutions.

Cite must be one of the most commonly used extensions in wikipedia space, and one would think that it is used multiple times per page. We have a fairly common practice with the renowned pipe trick, and now at comment #69, you are saying that we are "lowering the priority". How about we consider the basic users and solve some of the base underlying issues that bug users every day, or isn't that sexy enough to gather attention? For some of us this is a large annoyance, where we have to create work arounds, by scripting buttons, or doing significant copy and paste.

wicke wrote:

@Billinghurst: Cite can be covered by something like bluehairedlawyer's patch (attachment 5952 / comment 31), as its entire content is parsed as wikitext.

  • Bug 35954 has been marked as a duplicate of this bug. ***

lambdav wrote:

More than 6 years after this bug as been reported, the bug is still there !
How many time should we wait for correction ?
Priority must not be Low !

lambdav wrote:

Priority = High because blocking another High priority bug.

Nx.devnull wrote:

(In reply to comment #67)

(In reply to comment #65)

> An extension can register a callback and then do pre-save transformation on its
> content - this is the best solution in my opinion, since only the extension
> knows which parts of the content are supposed to be transformed, e.g. in the
> case of the gallery tag only the description part.

I am not too enthusiastic about encouraging extensions to do their own
arbitrary pre-save transform processing. In the longer run, general support for
extensions working with parsed (and pre-save transformed) wikitext / HTML DOM
input could be cleaner and should offer better opportunities for integration
with a visual editor. I am working on something like that in the Parsoid parser
(http://mediawiki.org/wiki/Parsoid). There is also a simple and undocumented
method for something vaguely like this in the PHP parser (the
setTransparentTagHook method).

However, I do not have a good solution for partly-parsed and -transformed
extensions like gallery. Will give this some thought, and ask others about
their ideas.

If the problem is that extensions could do arbitrary processing instead of the standard PST, then the only solution I can think of is to allow extension to somehow specify which part of their input should be transformed, and then let the parser do the processing. Implementing that would probably be much harder and more cumbersome.

The only thing that needs to be done on my patch is that I'm not sure the extension calling parser->preSaveTransform is 100% safe. I can give it a go if you want, but I don't want to waste my time if the patch will be rejected because my approach (letting the extension do the PST) is not the correct one.

New parsers are nice and all, but they do not fix the problem that we have right now, and they probably never will if you want to keep the current gallery syntax.

bluehairedlawyer wrote:

Actually David's wrong. This bug is seven not six years old. If its not fixed soon we'll have to figure out who's going to pay for it to go to college...

What would be wrong with running the all the contents of a <gallery> through preSaveTransform? I realise it's not wikitext but then preSaveTransform isn't the full parser? What particular pre-save transforms would corrupt the gallery syntax?

Nx.devnull wrote:

(In reply to comment #76)

Actually David's wrong. This bug is seven not six years old. If its not fixed
soon we'll have to figure out who's going to pay for it to go to college...

What would be wrong with running the all the contents of a <gallery> through
preSaveTransform? I realise it's not wikitext but then preSaveTransform isn't
the full parser? What particular pre-save transforms would corrupt the gallery
syntax?

I don't know off the top of my head what characters are allowed in filenames (in particular, is [, { or ~ allowed), but in theory:

<gallery>
File:Oh noez [[This is not a link.png|]] yeah, I know, this is a contrived example
</gallery>

There's also replacing tildes, substing and the reverse pipe trick. Replacing tildes would be the most realistic problem (if ~ is allowed in filenames)

[ { are both disallowed in any page name (which includes images). Three or more ~'s in a row is also forbidden in page titles. So you don't have to worry about that.

Personally I favour an approach of adding an interface for adding parserHooks that allows specifying options (such as if the parser hook should allow PST).

Actually David's wrong. This bug is seven not six years old. If its not fixed
soon we'll have to figure out who's going to pay for it to go to college...

That's still a good 11 years away. We're cross that bridge when we come to it.

bluehairedlawyer wrote:

Some parser tests

Attached:

bluehairedlawyer wrote:

Allow extensions to specify tag contents can be passed through preSaveTransform without being escaped

This is an alternative to Nikola's patch. It allows extensions and internal setHook functions to specify that there contents can be run through the preSaveTransform function without being escaped.

It passes all the current parser tests and a few others I've managed to throw at it.

Attached:

sumanah wrote:

bluehairedlawyer, could you put this patch in Gerrit?

bluehairedlawyer wrote:

Dan Collins has already done this (which is just as well as I haven't figured out how to do it yet). Here's the url:

https://gerrit.wikimedia.org/r/#/c/8784/

That was a smart way to solve it.

Commented some concerns on gerrit.

patch moved to gerrit, removing patch need review.

ori added a comment.Sep 16 2012, 8:53 PM

Dan Collins has two patches, not one:
https://gerrit.wikimedia.org/r/#/c/8784/ (previously mentioned)
and
https://gerrit.wikimedia.org/r/#/c/8785/

  • Bug 42078 has been marked as a duplicate of this bug. ***

Somebdy should really give those patches some love, as apparently the original committer has disappeared.

Gabriel: As you assigned this report to yourself a year ago, do you plean to rework the two patches?

Gabriel: As you assigned this report to yourself two years ago, do you plan to rework the two patches? Or are they "somehow" obsoleted in the meantime?

Change 8784 abandoned by Siebrand:
(Bug 2700) Pre-save transform inside extension hooks

Reason:
I'm abandoning this patch set as it's been open for a long time without any outlook at the open issues being resolved.

If the submitter or anyone else would like to work on it again, this patch set can be re-activated by clicking "Restore Change". Please only do this if you are actually going to work on it immediately.

https://gerrit.wikimedia.org/r/8784

  • Bug 72079 has been marked as a duplicate of this bug. ***
Nemo_bis set Security to None.
SamB awarded a token.Apr 4 2015, 5:10 AM
SamB added a subscriber: SamB.

Change 8785 had a related patch set uploaded (by Paladox):
(bug T4700) Pre-save transform inside extension hooks

https://gerrit.wikimedia.org/r/8785

Change 8786 had a related patch set uploaded (by MarkTraceur):
Pre-save transform inside extension hooks

https://gerrit.wikimedia.org/r/8786

Restricted Application added a subscriber: Aklapper. · View Herald TranscriptSep 23 2015, 2:25 PM

I was tooling around with this, but...

https://www.mediawiki.org/wiki/Manual:Tag_extensions#How_do_I_render_wikitext_in_my_extension.3F

This section says that tag extensions will never have PSTs applied to them. So maybe this bug is WONTFIX and we should abandon all patches for it?

This bug is not only/mainly about tag extensions: see pipe trick, subst, signatures.

AdamCuerden raised the priority of this task from High to Unbreak Now!.Sep 24 2015, 8:58 AM

this has surely gone on long enough, with maqsses of lost resources.

Aklapper lowered the priority of this task from Unbreak Now! to High.Sep 24 2015, 9:37 AM

This task has not suddenly become more urgent than before, hence reverting the recent Priority change. See https://www.mediawiki.org/wiki/Phabricator/Project_management#Setting_task_priorities for more information. Thanks.

Meno25 removed a subscriber: Meno25.Feb 22 2016, 7:18 PM

Change 272916 had a related patch set uploaded (by Cenarium):
Pre-save transform inside extension tags

https://gerrit.wikimedia.org/r/272916

Elitre added a subscriber: Elitre.Apr 18 2016, 4:35 PM
Restricted Application added a subscriber: Luke081515. · View Herald TranscriptJul 10 2016, 4:00 PM
Pols12 awarded a token.Aug 1 2016, 9:55 AM
Pols12 added a subscriber: Pols12.
He7d3r merged tasks: T10125: Pre-save transform not applied to markup inside <ref> container, T8222: any links inside <gallery> won't autopipe, T8157: template substitution does not work inside <ref> tags, T8814: Pre-save transform not executed for references (signatures, pipe trick, subst don't work), T10929: When a subst: is used inside a <ref> it doesn't get subst'ed, T12927: "pipe trick" links don't work, T14016: subst: don't work inside <ref> tags, T15072: refs fail to parse the piped link trick, T15959: Problem with <ref> and [[xxx (yyy)|]], T16007: Auto-piping does not work in galleries, T15120: New parser breaks signatures inside gallery tags, T16550: Cannot substitute templates withing <ref> tags, T17547: Automatic conversion of the wikitext doesn't work in references, T17467: Pipe trick not working when used as template parameter that is in turn used in a reference citation, T20064: The "pipe trick" does not work in footnotes, T22359: Subst: prefix does not work in image captions inside <gallery></gallery> tags., T15141: Pipetrick doesn't work in gallery image caption, T24258: Five tildes don't produce timestamp in en.wikipedia article namespace, T26131: ~~~~ in <gallery>, T26160: Magic words, templates and parser functions not substituted in <ref> tags, T26446: Automatic signatures (~~~~) do not work in <ref></ref>, T28859: Template substitution doesn't work inside <ref> tags, T30680: Pipe trick doesn't work within ref, T31775: <ref>...</ref> in template, T32135: Substitution not working in gallery tags, T36999: <ref> doesn't expand triple-brace template parameters, T37954: Inter-project links into <ref> do not work when alternative text is empty, T44078: {{subst:foo}} does not work in <ref>, T74079: {{subst: doesn't work inside <ref>...</ref>.Sep 3 2016, 12:53 PM
He7d3r added subscribers: bzimport, Wtmitchell, Krinkle and 14 others.
Izno added a project: Cite.Sep 3 2016, 7:25 PM
Izno moved this task from Unsorted backlog to Defect backlog on the Cite board.Sep 6 2016, 9:25 PM
demon removed a subscriber: demon.Oct 17 2016, 8:11 PM
Liuxinyu970226 awarded a token.
Wargo added a subscriber: Wargo.Apr 19 2017, 5:42 PM
Krinkle removed a subscriber: Krinkle.Apr 19 2017, 7:30 PM