Does TemplateSandbox not address your use case?

TemplateSandbox, as I understand it, still requires the sandboxed templates be saved each time you make a change. What we're doing is injecting values during Show Preview and as the template code gets processed, so you can literally just preview/fix/preview/fix until you're sure the template is working correctly. Once it is, you can save it without having to copy anything anywhere, cuz you're working on the normal template page itself. This is much faster and more convenient than any kind of sandboxed process I've used on other wikis.

The ability to have default values for template arguments specified once at the beginning, not to mention overriding them with sanitized values also wouldn't be addressed by TemplateSandbox. Yes, you can do this via subtemplates and such, but that means that the parser is constantly reprocessing everything to come up with the same result.

I haven't looked into it thoroughly, but I believe as of 1.35, in order to achieve this, we'd have to override the default parser with an entirely new one, just so we could override the preprocessor (as we're already doing), which overrides the preprocessor's newFrame() method, plus, of course, we have to override the default PPFrame itself. I'm not even sure if we have the ability to override the factory's default processor that it uses for pages, though if it's not already there, I assume that ability will be added at some point to give people the choice between the current parser and Parsoid. Our only other alternative, if we can't somehow plug into the low-level parser stuff, is to resort to modifying the base code directly. Either way, that in turn means that our frame is then being used even in places like preloading and signature sanitization, where it's just a needlessly bulkier version of PPFrame. With the code base for the basic parser following roughly the same logic as it does now, what would be nice would be if we could have newFrame() (or whatever factory methods) be told to use our frame for specific things, like direct page views, previews, and automated updates (e.g. API purge or what have you), but nothing else, though that might be asking a bit much, I dunno.

To give you an idea of why we're so concerned about this functionality, here are a few code samples.

This is the basic template, as it would be written on a regular wiki:

Value: {{{phrase|{{{1|Default}}}}}} <!-- Default -->
Lower-case: {{lc:{{{phrase|{{{1|Default}}}}}}}} <!-- default -->
Upper-case: {{uc:{{{phrase|{{{1|Default}}}}}}}} <!-- DEFAULT -->

Our template would be:

{{#define:phrase|{{{1|Default}}}}} <!-- If template is called without a {{{phrase}}} argument, sets it to the value of {{{1}}} or "Default". Otherwise, does nothing. -->
Value: {{{phrase}}} <!-- Default -->
Lower-case: {{lc:{{{phrase}}}}} <!-- default -->
Upper-case: {{uc:{{{phrase}}}}} <!-- DEFAULT -->

Want to preview it with a non-default value?

{{#preview:1|Hey there!}} <!-- Does not affect the template if saved with it...only works during Show Preview -->
{{#define:phrase|{{{1|Default}}}}}
Value: {{{phrase}}} <!-- Hey there! -->
Lower-case: {{lc:{{{phrase}}}}} <!-- hey there! -->
Upper-case: {{uc:{{{phrase}}}}} <!-- HEY THERE! -->

Want to sanitize it so the provided value always starts as ucfirst?

{{#define:phrase|{{{1|default}}}}} <!-- default -->
{{#local:phrase|{{ucfirst:{{{phrase}}}}}}} <!-- Overrides value of {{{phrase}}} so it's now "Default" -->
Value: {{{phrase}}} <!-- Default -->
Lower-case: {{lc:{{{phrase}}}}} <!-- default -->
Upper-case: {{uc:{{{phrase}}}}} <!-- DEFAULT -->

This is just a simple template. You can imagine how much savings in time and energy there are when you're talking about templates with a dozen or more parameters, many of which want sanitization, default values, etc. We have over 1500 templates, most of them built using this extension, and being maintained by maybe three template gurus and a handful of moderately skilled templaters, so you can see why we're so eager to keep the ability to change the underlying frames and continue using our extension.

In T283391#7104503, @RobinHood70 wrote:

TemplateSandbox, as I understand it, still requires the sandboxed templates be saved each time you make a change.

No it doesn't. Did you read https://www.mediawiki.org/wiki/Help:Extension:TemplateSandbox ?

What we're doing is injecting values during Show Preview and as the template code gets processed, so you can literally just preview/fix/preview/fix until you're sure the template is working correctly. Once it is, you can save it without having to copy anything anywhere, cuz you're working on the normal template page itself. This is much faster and more convenient than any kind of sandboxed process I've used on other wikis.

This is literally TemplateSandbox. Except instead of "injecting values", you just preview a real page that uses the template and it shows you what your unsaved template would do to the page. One trick I tend to do is create a page like https://www.mediawiki.org/wiki/Module:Version/doc that I can then "preview" so it runs through most of the test cases to ensure I didn't break the template. (And yes, it works for both normal templates and Lua modules.)

In T283391#7104507, @Legoktm wrote:

No it doesn't. Did you read https://www.mediawiki.org/wiki/Help:Extension:TemplateSandbox ?

No, I read https://www.mediawiki.org/wiki/Extension:TemplateSandbox#Usage which hides this feature down near the bottom in a single, easily missed sentence. It's still a lot more clunky than our system, to my mind. You can do what it's doing with default MediaWiki just by transcluding a test-cases page inside <noinclude> tags at the bottom of your template. It'll then use the live template on those test cases. And that's assuming that you actually have such a page available. Granted, this is likely to be common, but it doesn't help at all during the initial design of a template, where you might not have any pages calling it yet at all, and you're working out very basic kinks, testing permutations that you might not feel the need to put on a test page, etc.

This is literally TemplateSandbox. Except instead of "injecting values", you just preview a real page that uses the template and it shows you what your unsaved template would do to the page. One trick I tend to do is create a page like https://www.mediawiki.org/wiki/Module:Version/doc that I can then "preview" so it runs through most of the test cases to ensure I didn't break the template. (And yes, it works for both normal templates and Lua modules.)

Test case pages are excellent, no argument there, but as I said above, you don't really need TemplateSandbox to preview them, and our extension allows so much more than simple previewing. You can sub in values directly, at will, without having to go edit a different page, just as though you'd called the template with those values. That allows a full range of testing, including using values that may not exist anywhere on the wiki yet, so you can test edge cases, features that are under development, and so forth, all without having to create additional pages. Heck, you can even "create" a template that you don't actually plan to save, just to test out some feature, and go ahead and plug in whatever values you want, all centralized at the top so they take effect throughout the entire template. That's not to mention all the other features our extension introduces, like being able to designate default values or override values, as I showed in my examples. To the best of my knowledge, there's no extension in the public domain that does these things. If there were, we'd be using it.

There's also other stuff based on the same underlying techniques, like data storage and function-like behaviour (i.e., returning values to a parent template), though I'll grant that on the data-storage front, Scribunto is far and away better than our primitive little system. Still, it does what we need.

I have no idea why you'd rather request complex changes to the Parser and/or Preprocessor to accommodate a proprietary extension which could mostly be met by a combo of TemplateSandbox and probably Variables with some tweaks to your templates that a bot could do. I get the feeling you're stuck in the XY problem focusing on the specific technical solution you've already implemented (but don't want to publish code for?) rather than the use cases you want to address (template development and iteration) that are addressed differently these days. For example, Scribunto has a live REPL that most people use to debug templates while creating them, so the extra features you bring up that TemplateSandbox/Variables wouldn't have really aren't that useful.

The current version of the extension was written by a hobby programmer (albeit one who had remarkable insight into how the MW parser works) back between MW 1.10 through 1.15 or so, and has been hacked only enough to get it working as the years have gone by. One of the main modules is actually written entirely in global space! So, probably not something we'd want to release as is, even if we could. More importantly, however, It's her code, so not really ours to decide what to do with. We do have the code publicly available if you want to have a look at it, but not "public" public, as in intended for others to use. https://github.com/uesp/uesp-wikimetatemplate

My code, while a full-fledged redesign, still works on the same underlying principles because I was unable to find anything else, either publicly or by tracing through the MediaWiki code trying to find something more compatible with the upcoming parser changes, that could do what we need. Because you haven't been using this extension for years like we have, it understandably seems like a simple solution to you that we could use TemplateSandbox (which is completely unnecessary, as I said previously), and Variables (which is having its own problems with the latest changes, as noted in the red box at the top of its page). But neither of those comes even close to doing what we've had our wiki doing since MW 1.10. It's so tightly integrated into our templates that doing without it is pretty much inconceivable to us at this point and, as a bot creator, I feel reasonably confident that there's no way a bot could even come close to "tweaking" the templates. All of our templates would need massive rewrites, and I don't believe most of the features even exist in other extensions, apart from Semantic. There's literally code from this extension in almost every template, and when you include all the features it has, they're usually used throughout the better part of each template. In that way, I suppose it's both a blessing and a curse—our templates have all these wonderful features that we use all over the place, but...we use them all over the place!

As for the XY problem, I made a point of starting with the broader concept of simply injecting argument values somehow...it was only when we got into the nitty-gritty that I started talking about what, specifically, we're doing now. Ssastry said in the other thread that he would have someone at least look at our issue, so this is that. That thread went sideways very badly, which I've already admitted was largely down to my initial post, so I gave things a breather before posting this. My thinking once I did was that I should at least post what we're looking for so that it could be discussed and we get either a yay or nay. We're well aware that the answer may be "we're not going to do that", but I figured it's better to ask now, while the changes still are relatively simple to back out (literally a handful of lines to reinstate the ability to specify a custom preprocessor). Sure, it would be great if we could find a way of doing everything that's both supported by the old parser and compatible with Parsoid, but realistically, from my understanding of Parsoid, that's won't be feasible. As it says on Variables' page, non-linear processing of the page is anathema to what both our extensions are doing. You can't set a variable in one piece of code, then process an earlier piece of code where the value shouldn't have been changed yet and hope things will work. That's literally like try to run a program after randomizing the lines of code...it's just not going to work out. We accept that that will almost certainly be an insurmountable problem.

I don't think it's unreasonable, however, that we'd like to keep our current extension working for as long as possible. Until the day comes when the old parser is completely unable to support frame customization and/or is removed from the project entirely, I don't see why a custom preprocessor can't be maintained. It is literally a couple of lines, changed because nobody on codesearch was using them, and when I noticed the change during my code-tracing, I piped up to say "hey, yeah, we still are".

Sorry, I had my extensions mixed up. I've been meaning "Semantic" whenever I said "Scribunto", though both are something we may end up using at some point. I corrected it in my latest message, but wasn't going to go back and edit every message I might've said it in.

I thought it might be helpful to the discussion to provide a link to the template documentation: https://en.uesp.net/wiki/UESPWiki:MetaTemplate

If I'm not forgetting anything, the Variable Definition and Data Sharing functions are the ones that rely on the ability to override the preprocessor. The Variables extension can do something similar to #local/#define, although as I understand it, it works on its own internal data and doesn't affect the template arguments (e.g., {{{1}}}) at all. Still, for the purposes of those two functions, it would probably be equivalent with a few tweaks. That leaves out the remainder of the functions in that section, however, which we use extensively. I believe they would all work fine at frame depths greater than zero, since those are genuine template frames; however, at root level, they would break. That root-level processing allows us to:

• Directly #define/#local variables on a main page in any namespace (which, as you can see if you poke around, we use extensively). With that, we can do things like load or save a {{{description}}} variable and use it directly on the page after that.
• I've already commented on the use of #preview, so I won't repeat myself except to say that we find it an invaluable function when developing, over and above all the usual methods like test pages and the like.
• The #inherit function is used widely, even at root level, to provide what amount to on-the-fly behaviour switches. The same table template can be told to include or exclude columns for every call on the page, or it can be changed half-way down or on a per-call basis (with genuine template parameters overriding the default value set on the page). All by changing one variable as needed, which is typically once at the top of a page.
• A combination of #inherit/#return is used most often at the root level to give someone's sandbox the ability to behave like it's in a different namespace, correctly linking to pages in that namespace and showing correct layout for that namespace, while still retaining awareness that it's actually not, so we don't get sandbox pages appearing in namespace-specific categories.
• #unset is uncommonly used, but helpful for data validation. Data's not valid? Just unset it and thereafter, checks like {{#if:{{{data|}}}|...}} will behave exactly as if the parameter had never been specified in the first place.
• I trust #load and #save are obvious in their function. While they currently have a limited ability to do so, in my rewrite, I'm already working on the ability to fully save wiki markup and have it load and be re-evaluated in the context of a different page. We've wanted this for some time. The drawbacks are probably obvious, in the sense that some users don't think of context changes, but any markup to be saved will have to be marked with a tag, while normal items will simply be parsed before saving. I don't believe Semantic has this ability, though there's a lot of info there and I've only skimmed it.

So, are there some things here that Semantic, Scribunto, and Variables could handle? Absolutely! But not all of it, and as I've said, much of this is used extensively on our wiki. It's not just about templates, it's about enhancing even root page functionality.

Do we expect Parsoid to be able to do all this? Likely not. It works on a completely different concept, which both Variables and our extension will almost certainly be rendered useless by, and we get that. In the meantime, though, until it's no longer possible to maintain, it would be useful to have the ability to do $wgParserConf['preprocessorClass'] = "MetaTemplatePreprocessor"; (or its equivalent, if the mechanism needs to change to work with MWServices), like we've been able to with every version until the last one or two. I expect it would take a developer only a couple of minutes, which is why I don't understand the reluctance to do it. We've literally spent more time debating it than making the change would take. If it comes to that, we'll do it ourselves, since it really is only a couple of lines that need changing, but it would be nice to know that we're using our preprocessor the "right" way, the MediaWiki-prescribed way, until it's simply no longer viable at all.