Page MenuHomePhabricator

Scribunto: increase C stack size above 200
Closed, DeclinedPublicFeature

Description

Feature summary (what you would like to be able to do and where):

The current limit for the C stack in Scribunto seems to be just under 200, which is surprisingly small when compared to the general Lua stack limit of 16,384. It would be helpful to increase the limit.

Use case(s) (list the steps that you performed to discover that problem, and describe the actual underlying problem which you want to solve. Do not describe only a solution):

Since it's not possible to do proper tail recursion with C functions in Lua, the low limit makes it impossible to do any kind of deep recursion involving pcall or xpcall. This resulted in me having to do a major redesign of a Lua wikitext parser that I've been writing, where I was using xpcall in a try-except construct to quickly loop back while doing template/argument parsing, since sets of opening braces are often ambiguous until the following wikitext has been parsed (e.g. "{{{{{" - 5 braces - could mean several different things, depending on what follows).

Benefits (why should this be implemented?):

The alternative has been to propagate the result down the stack via return instead, which is much more fragile (as it's prone to mistakes by me if I miss a link in the chain), and it also seems to be more resource intensive. Although the C stack limit of 200 is not a problem most of the time, it began to cause problems when I tried parsing templates such as this one, which is given as an example of a complex template in the Parsoid preprocessor tests, as it includes parser functions nested to a depth of over 100.

Here is the last version of my wikitext parser which used xpcall, before I remodelled things, so that you can get an idea of the use case. It's still very much in a working state, and has been designed to be as fast as possible (as it may need to be called many hundreds of times on the same page).

Event Timeline

of a Lua wikitext parser that I've been writing

why? https://bash.toolforge.org/quip/AU7VVyPV6snAnmqnK_0N is relevant here.

of a Lua wikitext parser that I've been writing

why? https://bash.toolforge.org/quip/AU7VVyPV6snAnmqnK_0N is relevant here.

It's a last resort measure due to the major problems we've been having at the English Wiktionary with memory limits. We've already implemented special-case versions of this quite successfully, and generalising the approach has resulted in extreme memory reductions (for example, https://en.wiktionary.org/wiki/teacher down to 13.5MB from the current 49.5MB). We're already forced to use Lua-free templates on hundreds of large pages, because of the way Scribunto duplicates things many times in memory.

Plus it makes parsing inputs a lot easier in general, which we need to do a lot anyway.

tstarling subscribed.

It's a last resort measure due to the major problems we've been having at the English Wiktionary with memory limits.

The memory limit has since been doubled on Wiktionary to 100MB, and the GC step size was adjusted (T165935). If there's any remaining problem with memory limits on Wiktionary then we can consider increasing the limit again. The memory limit is easy to change, since it's in MediaWiki's configuration.

Increasing the stack size requires patching and recompiling Lua. This limit of 200 is LUAI_MAXCCALLS in luaconf.h. There needs to be a really good reason for doing that. I'm not seeing a sufficiently strong rationale here.