"Lua error: not enough memory" on certain en.wiktionary pages
Closed, InvalidPublic

Description

ISSUE:
Several users and I see "Lua error: not enough memory" in "water" when all its content is present, as in https://en.wiktionary.org/w/index.php?title=water&oldid=42726961 , and recently also in "man" https://en.wiktionary.org/w/index.php?title=man&oldid=42871191 . Errors persist despite efforts to simplify Lua-using templates in ways that do not reduce functionality.

The temporary workaround has been to split "water" onto two pages, and disable script support and transliteration of translations of "man"; this is undesirable because it reduces functionality. Many entries will one day be as complete as "water", so the issue will effect more pages if the causes are not addressed. (Indeed, in the last week another page developed an error.)

POSSIBLE SOLUTIONS:
We suspect Lua's garbage collection may have an issue: maybe individually not-intensive things, like individual templates, do not free up all their memory when they finish. If the issue is with Lua tasks not releasing memory when done, it would be ideal to improve that.

We also know some of our code is resource-intensive, so perhaps some errors could be solved if you could help us identify which pieces of code used on the above-linked revisions use the most memory, and how they could be streamlined without losing functionality (like automatic transliteration, etc).

Perhaps it would be possible to add "a function to create a 'cache' of language and script objects that can be used by multiple functions"; see explanation and rationale at https://en.wiktionary.org/wiki/Wiktionary:Grease_pit/2017/May#lsobjectcache .

Alternatively, perhaps it would be preferable to just raise the amount of memory that pages can use. (How would that affect mobile users?)

EXTRA DETAILS:
There is some randomness to the errors : I loaded https://en.wiktionary.org/w/index.php?title=water&oldid=42726961 and memory ran out in one place; I hard-refreshed and it ran out in another place. At the same time, there has consistently been no error in https://en.wiktionary.org/wiki/a or https://en.wiktionary.org/wiki/woman , even though "a" is larger than "man"; "a" and "woman" use more templates than "man"; and "woman" has ~2x as many translation templates (which are known to be resource-heavy and to contribute to errors) as "man".

-sche created this task.May 21 2017, 4:28 AM
Restricted Application added a subscriber: Aklapper. · View Herald TranscriptMay 21 2017, 4:28 AM
Urbanecm added a subscriber: Urbanecm.

@Framawiki Are you sure this is Wikimedia Site request? I think this is related to Scribunto more than being a site request.

Anomie closed this task as Invalid.May 21 2017, 9:50 AM
Anomie added a subscriber: Anomie.

We suspect Lua's garbage collection may have an issue

"We suspect" isn't helpful. Provide actual test cases that illustrate that it's actually a problem in Scribunto rather than your module just using too much memory. A 194k page with a multitude of nested template invocations that happens to have errors isn't a test case.

I note if I edit that revision of your 'water' page and change all the uses of {{t}}, {{t+}}, {{t-check}}, {{t-image}}, {{t-needed}}, and {{t-simple}} to use the same language for the first parameter (in my test, I used 'de') without changing anything else at all, it mo longer produces errors. That also causes the page to load 90 fewer modules, mostly "Module:foo-translit". Your "garbage collection issue" might simply be that every one of those modules is cached so they don't have to be reparsed from scratch each time they're used.

Perhaps it would be possible to add "a function to create a 'cache' of language and script objects that can be used by multiple functions"; see explanation and rationale at https://en.wiktionary.org/wiki/Wiktionary:Grease_pit/2017/May#lsobjectcache .

It's not possible to create an mw.loadData with functions, since that would be an instance of T67258: Information can be passed between #invoke's (tracking).

The purpose of mw.loadData isn't to save memory, it's to save the time that would otherwise be required to reparse the data module every time.

Alternatively, perhaps it would be preferable to just raise the amount of memory that pages can use. (How would that affect mobile users?)

While raising the memory limit is possible, at some point in the future you'll probably wind up with an even-huger page that would exceed the new limit.

Alex_brollo added a subscriber: Alex_brollo.EditedFeb 14 2018, 8:55 AM

Another case from it.wikisource: https://it.wikisource.org/wiki/Ricordi_di_Parigi/Uno_sguardo_all%E2%80%99Esposizione

It pops out into a proofread ns0 pages with multiple calls to an it.wikisource module: Modulo:Wl

Using a test page with lots of identical calls to Modulo:Wl, really the "NewPP limit report" into source html tells "Lua memory usage: 50 MB/50 MB", t.i. Lua used really the whole amount of available memory.