Page MenuHomePhabricator

"mw.title.newBatch(...)" is unreasonably memory-hoggy
Open, Needs TriagePublic

Description

Steps to reproduce:

Expected outcome: no error

Observed outcome: "Galat Lua: not enough memory."

"mw.title.newBatch(...)" is supposed to increment the expensive function counter by ONE for every 25 titles queried. Tests can confirm that it works like that. Given that one should be able to afford 500 * 25 =12'500 ifexist requests. However, the test starts failing with "Galat Lua: not enough memory." instead of the feared "Galat Lua: too many expensive function calls." already at ca 7'000 titles, far below 12'500.

Module: https://id.wikipedia.org/wiki/Modul:T410527

Proposed ideas for solution:

  • reduce the memory-hogginess of "mw.title.newBatch(...)" with full current functionality
  • add a parameter "ifexist-only" to reduce the functionality and memory hogginess

Event Timeline

Is there a practical use case where it creates an issue? newBatch() is intended to help avoid hitting parser function limits, but if you are really going to abuse it for 7000+ page lookups, it'll inevitably hit some limit or another. That's not a performance issue, it's the system working as intended to prevent excessive load.

Reportedly LUA can use 50 MiO RAM, quite generous. But if 7'000 pages hog 50 MiO, then every single page consumes ca 7 KiO. Does it really need so much? I could try to call "mw.title.newBatch(...)" in batches of 25 pages, and copy the relevant info from bloated title objects into a LUA table of booleans.

I ran a little experiment: the lua module at https://en.wikipedia.org/w/index.php?title=Module:Sandbox/SD0001/ext&oldid=1323886587 has functions to show the existence status of 500 pages, with and without use of newBatch().

{{#invoke:Sandbox/SD0001/ext|batched}} consistently gives a Lua memory usage of 4.1 MB.
{{#invoke:Sandbox/SD0001/ext|notbatched}} consistently uses 4.2 MB.

So, batching uses the same amount of memory, if not marginally lesser, than creating mw.title objects manually.

Reportedly LUA can use 50 MiO RAM, quite generous. But if 7'000 pages hog 50 MiO, then every single page consumes ca 7 KiO. Does it really need so much? I could try to call "mw.title.newBatch(...)" in batches of 25 pages, and copy the relevant info from bloated title objects into a LUA table of booleans.

I would suggest batches of 500 rather than batches of 25.