Page MenuHomePhabricator

Add support for a newer Lua version than Lua 5.1 to luasandbox
Open, Needs TriagePublic

Description

At present, luasandbox is tied to Lua 5.1, which is so out of date that certain features of Lua 5.2 are backported to luasandbox's implementation of Lua.

Perhaps it is time to move on.

To begin with, references to Lua version hardcoded into luasandbox source code could be removed and made configurable in the options for cmake or ./configure (defaulting to Lua 5.1).

The hardcoded Lua version is found in config.cmake and config.m4. The file FindLua51cpp.cmake ought to be, I guess, cloned into FindLua52cpp.cmake and FindLua53cpp.cmake or parametrised somehow.

Details

Related Changes in Gerrit:
SubjectAuthorRepoBranchLines +/-
Arendpietermediawiki/extensions/Scribuntomaster+207 -145
Arendpietermediawiki/extensions/Scribuntomaster+9 -1
Arendpietermediawiki/extensions/Scribuntomaster+117 -8
Arendpietermediawiki/php/luasandboxmaster+76 -0
Arendpietermediawiki/extensions/Scribuntomaster+2 -2
Arendpietermediawiki/coremaster+34 -1
Arendpietermediawiki/extensions/Scribuntomaster+62 -8
Arendpietermediawiki/extensions/Scribuntomaster+1 -1
Arendpietermediawiki/extensions/Scribuntomaster+48 -9
Hasharmediawiki/php/luasandboxmaster+0 -0
Hasharintegration/configmaster+5 -5
Hasharintegration/configmaster+26 -1
Arendpieterintegration/configmaster+16 -4
Arendpietermediawiki/extensions/Scribuntomaster+1 -1
Arendpietermediawiki/php/luasandboxmaster+271 -22
Arendpietermediawiki/php/luasandboxmaster+350 -4
Arendpietermediawiki/php/luasandboxmaster+296 -0
Arendpietermediawiki/php/luasandboxmaster+396 -50
Show related patches Customize query in gerrit

Event Timeline

There are a very large number of changes, so older changes are hidden. Show Older Changes

Lua 5.4 is now available as well. It would be nice to see it use the latest version.

Aklapper renamed this task from Add support for Lua 5.2 or 5.3 to luasandbox to Add support for a newer Lua version than Lua 5.1 to luasandbox.Nov 1 2021, 10:47 AM

To quote @tstarling:

For some background material about why I don't like the idea of moving to Lua 5.2, you could read:

http://lua-users.org/lists/lua-l/2012-02/msg00127.html

It was mentioned there that LuaJIT is based on Lua 5.1 -- that is still the case.

http://lua-users.org/lists/lua-l/2012-02/msg00150.html

"And the second important point is, that it's worthy to sometimes look at each Lua version as a totally unrelated language, "only accidentally bearing a somewhat similar name". It is not considered as something "bad" if you stick with your chosen version of Lua forever in your product (although this surely also has some drawbacks) [1][2]. You can even consider taking it sideways, by adding some patches etc."

So I would mark this as declined, but he also said (in 2014):

Like I said at the start of this project, maybe in a decade or so. Unless there is some actual reason to want to do it?

So marking as stalled for re-evaluation in 2024.

It's now almost 2024 :) 5.4 is now available with new features such as const.

Frostly changed the task status from Stalled to Open.Oct 16 2023, 3:54 AM
Legoktm changed the task status from Open to Stalled.Oct 18 2023, 5:56 AM

It's not 2024 yet.

Aklapper changed the task status from Stalled to Open.Jan 1 2024, 4:16 PM

I am assuming this project is dead (which is super sad). It was extremely nice having something to easily manipulate data from dynamic datasets without hardcoding it into the application.

Which "project" in this case are you assuming is dead?

Do we have any specific list of features that "we" as a community need/want?

Upgrading for the sake of upgrading isn't really a great reason, but if we have some better use cases/similar...

Requoting from above

To quote @tstarling:

For some background material about why I don't like the idea of moving to Lua 5.2, you could read:

http://lua-users.org/lists/lua-l/2012-02/msg00127.html

It was mentioned there that LuaJIT is based on Lua 5.1 -- that is still the case.

http://lua-users.org/lists/lua-l/2012-02/msg00150.html

"And the second important point is, that it's worthy to sometimes look at each Lua version as a totally unrelated language, "only accidentally bearing a somewhat similar name". It is not considered as something "bad" if you stick with your chosen version of Lua forever in your product (although this surely also has some drawbacks) [1][2]. You can even consider taking it sideways, by adding some patches etc."

@tstarling Got any updated opinions about this in 2025?

Lua 5.1 has been dead since 2012. Its not about "new features". Its about using a maintained version. A lot of distros don't even allow installing Lua 5.1 without enabling legacy repos. For example, homebrew has removed it (linked to 5.3 as 5.1 completely 404s now. Debian still has 5.1 and is only getting minor build patches.

I wouldn't even say target 5.2 as thats not supported either. 5.4 is the only supported Lua release right now.

As for the "project", I am talking about the LuaSandbox project. And those cited mailing lists are dead links.

@tstarling Got any updated opinions about this in 2025?

I briefly talked with Tim about this last year, my recollection is that users need to articulate what newer Lua features they want and how they'd use them (correct me if I'm misremembering please!) to justify an upgrade; I'd personally suggest starting a wiki page for people to collaborate on. IIRC some Lua 5.2 features were already backported. Upgrading requires re-auditing the entire security of whichever version of Lua, which is a non-trivial endeavor.

Lua 5.1 has been dead since 2012. Its not about "new features". Its about using a maintained version. A lot of distros don't even allow installing Lua 5.1 without enabling legacy repos. For example, homebrew has removed it (linked to 5.3 as 5.1 completely 404s now. Debian still has 5.1 and is only getting minor build patches.

It's unfortunate that homebrew removed it, but is there actual maintenance that's needed for it? It's basically a finished project at this point. It probably would be useful to create a new unofficial Lua 5.1 upstream that collects the various distro patches that are floating around and provides some binary builds but it's pretty minimal maintenance.

@Legoktm, as for me as Lua developer it's annoying to dig into ancient Lua docs to write the ancient code. But technically I'm completely fine with the set of features that Lua 5.1 provides.

But still. How many years we need to wait more to make the current Lua engine totally incompartible in syntax with the actual Lua version? "Totally incompartible" I mean when to upgrade the engine we will need to completely drop everything written in Lua before and write it again from the scratch.
"Finished project" but why not COBOL then? It's also finished project. I speak COBOL and also fine with it. :)

Let's make the syntax transition between Lua versions as soft as possible. I suggest to start with 5.2 while it's not too late.
These transitions may happen a bit more often than Lua developers makes a new major version. Then finally Wikis will support the current Lua.

I am implementing wiki stuff on a regular base.

  • I do not feel any need to be in line with any current version in the outer world.
  • The codes I know are limited to Wiki page transclusion, by #invoke: in the end, providing better outcome than available via wikitext template syntax.
  • There is no need to apply this outside of a wikitext page.
  • I do not need any code from outside of WMFverse. There might be code examples for a few standard tasks, but they would be ported once into wikitext framework, perhaps with some other solutiuons for unavailable syntax. Much less efforts than upgrading.

There is a doc of 5.1 on my hard disk, and I do know it from cerebral memory.

  • A copy should be available at WMF.

Some interesting new features, related to multibyte strings:

  • Lua 5.2 would let us use hexadecimal byte codes in strings, whereas we currently have to rely on decimal byte codes, which are very uncommon. Example: we could replace the very unusual \194\160 with \xC2\xA0.
  • Lua 5.3 would even let us use Unicode code points in strings. Example: \u{A0}.
    • These code points are converted into their corresponding sequences of UTF-8 bytes (as outlined here). Keep in mind, they remain multibyte strings and come with all the associated caveats when using native string.* functions.

References:

To provide an example, on frwiki we have a Langue.nonLatin() method that works on multibyte strings. Lua 5.3 would let us change this:

local pattern = '[^'
    .. '%z'                        -- U+0000 (UTF-8 00), cas particulier qu'il faut représenter avec %z
    .. '\001-\205\175'             -- U+0001 (UTF-8 01) à U+036F (UTF-8 CDAF)
    .. '\225\180\128-\225\187\191' -- suppléments phonétique, diacritiques et latin : U+1D00 (UTF-8 E1B480) à U+1EFF (UTF-8 E1BBBF)
    .. '\226\128\128-\226\177\191' -- espace, indices, monnaies et symboles divers : U+2000 (UTF-8 E28080) à U+2C7F (UTF-8 E2B1BF)
    .. '\234\156\160-\234\159\191' -- latin étendu D : U+A720 (UTF-8 EA9CA0) à U+A7FF (UTF-8 EA9FBF)
    .. ']'

To this:

local pattern = '[^'
    .. '\u{0}-\u{36F}'     -- U+0000 (UTF-8 00) à U+036F (UTF-8 CDAF)
    .. '\u{1D00}-\u{1EFF}' -- suppléments phonétique, diacritiques et latin : U+1D00 (UTF-8 E1B480) à U+1EFF (UTF-8 E1BBBF)
    .. '\u{2000}-\u{2C7F}' -- espace, indices, monnaies et symboles divers : U+2000 (UTF-8 E28080) à U+2C7F (UTF-8 E2B1BF)
    .. '\u{A720}-\u{A7FF}' -- latin étendu D : U+A720 (UTF-8 EA9CA0) à U+A7FF (UTF-8 EA9FBF)
    .. ']'

I've started https://www.mediawiki.org/wiki/Extension:Scribunto/New_Lua_version. Please add other new Lua features that would be useful there (I've already copied @Od1n's comment above).

@Legoktm, as for me as Lua developer it's annoying to dig into ancient Lua docs to write the ancient code.

There is a doc of 5.1 on my hard disk, and I do know it from cerebral memory.

  • A copy should be available at WMF.

Just to confirm, we're talking about https://www.lua.org/manual/5.1/manual.html right? Should we just import it to mediawiki.org (some of it is already in the Lua reference manual)? Or mirror it on doc.wikimedia.org?

I have reviewed the lists of new features (Lua 5.2, Lua 5.3, Lua 5.4), and apart from the hexadecimal and Unicode escapes I mentioned earlier, I don't see anything particularly compelling. That said, I believe these escapes alone would justify implementing a newer Lua version.

Change #1305098 had a related patch set uploaded (by Arendpieter; author: Arendpieter):

[mediawiki/php/luasandbox@master] Add support for building against Lua 5.4

https://gerrit.wikimedia.org/r/1305098

Lua 5.4 review notes:

  • Removed functions which can be preserved: math.atan2, math.cosh, math.sinh, math.tanh, math.pow, math.frexp, math.ldexp, math.log10, table.getn, table.setn, table.foreach, table.foreachi, table.maxn, unpack.
  • setfenv and getfenv can be reimplemented using lua_setupvalue/debug.setupvalue etc. They were never exposed at WMF, partly due to their removal in 5.2, however Scribunto depends on them internally. In userspace they're replaced by the env parameter to load, but in the C API there is no such parameter and lua_load callers are expected to use lua_setupvalue. This is how load is implemented internally. It remains a bit of a mystery why these functions were removed. In Scribunto, chunks are loaded into the base environment and then later adjusted to be in the isolated environment, which still seems stable enough despite the inelegance of lua_setupvalue.
  • Emergency GC is basically what I tried in T349462. Not configurable. Assume crashy until proven otherwise. Will have to test. Run for a long time under valgrind with an allocation hook that randomly fails. Compile with LUAI_ASSERT.
  • The generational GC mode is new and non-default, presumably lightly tested.
  • string.format("%p") exposes pointer addresses. In LuaSandbox we already have a fork of the whole string library but in standalone this will require a pure-Lua reimplementation of format().
  • Coroutines remain quite scary especially in LuaSandbox which would have to be aware of lua_State forking. Fixed bugs like 1e64c13 and e657a48 show that Roberto finds this area of the code challenging.
  • Arendpieter notes that table.move can run forever and is a DoS vulnerability. A sparse implementation might be possible: sort the keys in the source table and iterate through them instead of iterating through the integers specified by the user. The sparse implementation would be selected when the number of elements to be copied exceeds the size of the table. Lua does not track the size of the table but does store capacity (asize) which gives an upper bound. In the standalone mode, there is no problem, since a pure-Lua implementation is trivial and will call hooks.
  • The GC can call luaE_warnerror() if an error is encountered while running finalizers, thus we need to call lua_setwarnf since the default implementation is not suitable. The documentation doesn't specify whether it is safe for the handler to call back into Lua, potentially re-entering the GC, although it appears to be safe in 5.4-5.5. So LuaSandbox could implement this as a PHP callback, say LuaSandbox::setWarnFunction().
  • match is now protected against C stack exhaustion, but my hook call patch was not accepted upstream and still needs to be applied.

Change #1305098 merged by jenkins-bot:

[mediawiki/php/luasandbox@master] Add support for building against Lua 5.4

https://gerrit.wikimedia.org/r/1305098

Change #1323864 had a related patch set uploaded (by Arendpieter; author: Arendpieter):

[mediawiki/php/luasandbox@master] Restore setfenv/getfenv on Lua 5.4 via the _ENV upvalue

https://gerrit.wikimedia.org/r/1323864

Change #1323937 had a related patch set uploaded (by Arendpieter; author: Arendpieter):

[mediawiki/php/luasandbox@master] Restore Lua 5.1's deprecated math/table functions on Lua 5.4

https://gerrit.wikimedia.org/r/1323937

Change #1323938 had a related patch set uploaded (by Arendpieter; author: Arendpieter):

[mediawiki/php/luasandbox@master] Expose a safe table.move, bounded by the source table's real size

https://gerrit.wikimedia.org/r/1323938

There is a doc of 5.1 on my hard disk, and I do know it from cerebral memory.

  • A copy should be available at WMF.

Just to confirm, we're talking about https://www.lua.org/manual/5.1/manual.html right? Should we just import it to mediawiki.org (some of it is already in the Lua reference manual)? Or mirror it on doc.wikimedia.org?

Yeah, looks like my hard disk copy a dozen years ago, while not comparing every single character.

  • I am not sure about licensing, but a WMF copy would be nice as reference material.

FWIW I have a lua 5.4 parser and transcompiler written in lua 5.1/5.4 which could be used to automatically translate some of the incompatible features if that proved useful. It currently works to run lua 5.4 code inside our existing lua 5.1 runtime. It's described briefly in https://en.wikipedia.org/wiki/User:Cscott/Ideas/Improved_for-loops_for_Lua but I can/should pull that code into a proper repository of its own.

Change #1323864 merged by jenkins-bot:

[mediawiki/php/luasandbox@master] Restore setfenv/getfenv on Lua 5.4 via the _ENV upvalue

https://gerrit.wikimedia.org/r/1323864

Change #1323937 merged by jenkins-bot:

[mediawiki/php/luasandbox@master] Restore Lua 5.1's deprecated math/table functions on Lua 5.4

https://gerrit.wikimedia.org/r/1323937

Change #1329329 had a related patch set uploaded (by Arendpieter; author: Arendpieter):

[mediawiki/php/luasandbox@master] Add a regression test exercising Lua 5.4's emergency-GC path

https://gerrit.wikimedia.org/r/1329329

Change #1329331 had a related patch set uploaded (by Arendpieter; author: Arendpieter):

[mediawiki/extensions/Scribunto@master] Update the allowEnvFuncs doc comment for Lua 5.2+ support

https://gerrit.wikimedia.org/r/1329331

Change #1329332 had a related patch set uploaded (by Arendpieter; author: Arendpieter):

[mediawiki/extensions/Scribunto@master] Fix two getfenv() env-leak bugs for functions with no _ENV upvalue

https://gerrit.wikimedia.org/r/1329332

Change #1323938 merged by jenkins-bot:

[mediawiki/php/luasandbox@master] Expose a safe table.move, bounded by the source table's real size

https://gerrit.wikimedia.org/r/1323938

Change #1329507 had a related patch set uploaded (by Arendpieter; author: Arendpieter):

[mediawiki/extensions/Scribunto@master] Fix setfenv() silently corrupting mw.lua's own shared _ENV

https://gerrit.wikimedia.org/r/1329507

Change #1329331 merged by jenkins-bot:

[mediawiki/extensions/Scribunto@master] Update the allowEnvFuncs doc comment for Lua 5.2+ support

https://gerrit.wikimedia.org/r/1329331

Change #1329332 merged by jenkins-bot:

[mediawiki/extensions/Scribunto@master] Fix two getfenv() env-leak bugs for functions with no _ENV upvalue

https://gerrit.wikimedia.org/r/1329332

Change #1330584 had a related patch set uploaded (by Arendpieter; author: Arendpieter):

[mediawiki/extensions/Scribunto@master] Cast getDurationIntervals()'s per-interval counts to int

https://gerrit.wikimedia.org/r/1330584

Change #1330595 had a related patch set uploaded (by Arendpieter; author: Arendpieter):

[integration/config@master] dockerfiles: Install liblua5.4-dev alongside liblua5.1 in php-compile images

https://gerrit.wikimedia.org/r/1330595

Change #1330595 merged by jenkins-bot:

[integration/config@master] dockerfiles: Install liblua5.4-dev alongside liblua5.1 in php-compile images

https://gerrit.wikimedia.org/r/1330595

Change #1331377 had a related patch set uploaded (by Hashar; author: Hashar):

[integration/config@master] dockerfiles: add liblua5-4dev for php81 + missing changelogs

https://gerrit.wikimedia.org/r/1331377

Change #1331486 had a related patch set uploaded (by Hashar; author: Hashar):

[integration/config@master] jjb: update php83-compile php85-compile

https://gerrit.wikimedia.org/r/1331486

Change #1331377 merged by jenkins-bot:

[integration/config@master] dockerfiles: rebuild for liblua5-4dev

https://gerrit.wikimedia.org/r/1331377

Change #1331486 merged by jenkins-bot:

[integration/config@master] jjb: update php83-compile php85-compile

https://gerrit.wikimedia.org/r/1331486

Change #1331489 had a related patch set uploaded (by Hashar; author: Hashar):

[mediawiki/php/luasandbox@master] (DO NOT SUBMIT) trigger CI builds for liblua5.4

https://gerrit.wikimedia.org/r/1331489

I have sent a dummy change https://gerrit.wikimedia.org/r/c/mediawiki/php/luasandbox/+/1331489

I did a recheck and all jobs passed just fine.

The encountered failure with php84 is a segfault. The image used was docker-registry.wikimedia.org/releng/php84-compile:8.4.24, the Jenkins build console shows:

TEST 18/26 [tests/profiler-sorting.phpt]
FAIL profiler sorting [tests/profiler-sorting.phpt]
...
=====================================================================
TIME END 2026-08-28 09:12:38

=====================================================================
TEST RESULT SUMMARY
---------------------------------------------------------------------
Exts skipped    :     0
Exts tested     :    16
---------------------------------------------------------------------

Number of tests :    26                26
Tests skipped   :     0 (  0.0%) --------
Tests warned    :     0 (  0.0%) (  0.0%)
Tests failed    :     1 (  3.8%) (  3.8%)
Tests passed    :    25 ( 96.2%) ( 96.2%)
---------------------------------------------------------------------
Time taken      : 5.711 seconds
=====================================================================

=====================================================================
FAILED TEST SUMMARY
---------------------------------------------------------------------
profiler sorting [tests/profiler-sorting.phpt]
=====================================================================

The log file is attached to the Jenkins build:

artifact/src/tests/profiler-sorting.log
---- EXPECTED OUTPUT
samples: OK
seconds: OK
percent: OK
---- ACTUAL OUTPUT
samples: OK
seconds: OK
percent: OK
Segmentation fault
---- FAILED

For liblua5.4-dev support, the CI image now have the dev. I have looked at the consoles and the configure script reports:

checking for Lua... lua5.1 5.1.5
checking for lua5.1... yes

T436171 mentioned the usage of legacy vararg function syntax using arg, which is removed in Lua 5.2.

Change #1331489 abandoned by Hashar:

[mediawiki/php/luasandbox@master] (DO NOT SUBMIT) trigger CI builds for liblua5.4

https://gerrit.wikimedia.org/r/1331489

Change #1330584 abandoned by Arendpieter:

[mediawiki/extensions/Scribunto@master] Cast getDurationIntervals()'s per-interval counts to int

Reason:

It's a core bug and should be fixed only in core.

https://gerrit.wikimedia.org/r/1330584

Change #1337816 had a related patch set uploaded (by Arendpieter; author: Arendpieter):

[mediawiki/core@master] Return int values from getDurationIntervals()

https://gerrit.wikimedia.org/r/1337816

Change #1337816 had a related patch set uploaded (by Arendpieter; author: Arendpieter):

[mediawiki/core@master] Return int values from getDurationIntervals()

https://gerrit.wikimedia.org/r/1337816

Change #1329507 merged by jenkins-bot:

[mediawiki/extensions/Scribunto@master] Fix setfenv() silently corrupting mw.lua's own shared _ENV

https://gerrit.wikimedia.org/r/1329507

Change #1337816 merged by jenkins-bot:

[mediawiki/core@master] Return int values from getDurationIntervals()

https://gerrit.wikimedia.org/r/1337816

Change #1339616 had a related patch set uploaded (by Arendpieter; author: Arendpieter):

[mediawiki/extensions/Scribunto@master] Take the invocation frame from frameMap in mw.executeFunction()

https://gerrit.wikimedia.org/r/1339616

Change #1339617 had a related patch set uploaded (by Arendpieter; author: Arendpieter):

[mediawiki/extensions/Scribunto@master] Handle functions without an _ENV upvalue in setfenv() and shared envs

https://gerrit.wikimedia.org/r/1339617

Change #1339618 had a related patch set uploaded (by Arendpieter; author: Arendpieter):

[mediawiki/extensions/Scribunto@master] Fix setfenv() silently corrupting mw.lua's own shared _ENV

https://gerrit.wikimedia.org/r/1339618

Change #1341718 had a related patch set uploaded (by Arendpieter; author: Arendpieter):

[mediawiki/extensions/Scribunto@master] tests: Make the unpack() limit tests exercise the guard

https://gerrit.wikimedia.org/r/1341718

Change #1341718 merged by jenkins-bot:

[mediawiki/extensions/Scribunto@master] tests: Make the unpack() limit tests exercise the guard

https://gerrit.wikimedia.org/r/1341718

Change #1329329 abandoned by Arendpieter:

[mediawiki/php/luasandbox@master] Add a regression test exercising Lua 5.4's emergency-GC path

https://gerrit.wikimedia.org/r/1329329

Change #1339616 merged by jenkins-bot:

[mediawiki/extensions/Scribunto@master] Merge mw.executeModule() and mw.executeFunction()

https://gerrit.wikimedia.org/r/1339616