Page MenuHomePhabricator

Lua's ipairs() function can no longer iterate over Wikidata references
Closed, ResolvedPublicBUG REPORT

Description

Steps to replicate the issue:

  • In an on-wiki Lua console (below the edit field in module pages), write the following:
    • local refs = mw.wikibase.getEntity( 'Q42' ).claims['P569'][1].references
    • mw.logObject( refs ) (to confirm that it loads correctly)
    • for k,v in pairs( refs ) do mw.log( k ) mw.log( v ) --> normal output
    • for k,v in ipairs( refs ) do mw.log( k ), mw.log( v ) --> no output at all

What happens?:

  • ipairs() isn't able to read the table at all, while previously it could.
    • A consequence of this is that modules that fetch references from Wikidata by using ipairs() have stopped working (see the original bug report below)
  • The immediate problem on-wiki can be fixed (at least in the Norwegian Bokmål Wikipedia's case) by replacing ipairs() with pairs(). However, the challenge is to identify which modules and functions that needs to happen in.

What should have happened instead?:

  • ipairs() should continue to work like it did on reference tables.

I believe this is caused by the changes in I817466a797a37453a95e0a5fc56db3bb3aedb1ff (T401290).

Original report

Steps to replicate the issue:

What happens?:

Norwegian Bokmål Wikipedia relies on content and references from Wikidata in their infoboxes, as do some other projects.

This week, the sources have all gone missing (some pending cache reset), although I can't find any mentioned or visible changes to Wikidata-fetching modules or templates.

What should have happened instead?:

References should show up as normal.

Other information (browser name/version, screenshots, etc.):

Tech News mention a change in how Recent Changes may now display some Wikidata changes. I assume these wouldn't affect this functionality, but who knows.

Event Timeline

jhsoby renamed this task from References no longer fetched from Wikidata through Lua modules to Lua's ipairs() function can no longer iterate over Wikidata references.Oct 18 2025, 9:26 AM
jhsoby updated the task description. (Show Details)
jhsoby updated the task description. (Show Details)

Note: This edit fixes the issue for the most commonly used module that fetches references in the Norwegian Bokmål Wikipedia. However, that's just one module on one wiki – who knows how many other things are impacted by this?

With that fix, the examples from the original bug report no longer showcase the problem. However, you can still see the problem by going here and using the "Preview page with this template" on any of the articles that were mentioned.

All Wikidata references are broken on huwiki.

jhsoby triaged this task as Unbreak Now! priority.Oct 20 2025, 6:47 AM

Boldly setting to UBN! because of the consequences of this bug (missing references on thousands of pages across wikis – for the Norwegian Bokmål Wikipedia's case, it's in the order of hundreds of thousands of pages; no idea about the Hungarian Wikipedia).

Of course, feel free to adjust as needed.

elwiki is among those affected, probably cswiki too.

Thanks for tagging us. We're looking into it. Our Engineers will share more updates asap

After some research, it seems like the problem lies in the masking tables, the schema of the object is different after the patch. This causes different output behavior between pairs and ipairs. We will share more updates asap. Thanks!

Old Object.png (1×885 px, 103 KB)
New Object.png (1×885 px, 129 KB)

Is it possible to revert the broken change? It has seemingly also caused T407744, where affected pages don’t just have broken references, but can’t be rendered at all.

Change #1197274 had a related patch set uploaded (by Lucas Werkmeister (WMDE); author: Neslihan Turan):

[mediawiki/extensions/Wikibase@master] Revert "Implement new usage types for statement with qualifiers and references"

https://gerrit.wikimedia.org/r/1197274

Change #1197276 had a related patch set uploaded (by Neslihan Turan; author: Neslihan Turan):

[mediawiki/extensions/Wikibase@wmf/1.45.0-wmf.23] Revert "Implement new usage types for statement with qualifiers and references"

https://gerrit.wikimedia.org/r/1197276

Change #1197276 merged by jenkins-bot:

[mediawiki/extensions/Wikibase@wmf/1.45.0-wmf.23] Revert "Implement new usage types for statement with qualifiers and references"

https://gerrit.wikimedia.org/r/1197276

Mentioned in SAL (#wikimedia-operations) [2025-10-20T14:27:50Z] <lucaswerkmeister-wmde@deploy2002> Started scap sync-world: Backport for [[gerrit:1197276|Revert "Implement new usage types for statement with qualifiers and references" (T401290 T407684 T407744)]]

Mentioned in SAL (#wikimedia-operations) [2025-10-20T14:31:56Z] <lucaswerkmeister-wmde@deploy2002> lucaswerkmeister-wmde, neslihanturan: Backport for [[gerrit:1197276|Revert "Implement new usage types for statement with qualifiers and references" (T401290 T407684 T407744)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.

Change #1197281 had a related patch set uploaded (by Lucas Werkmeister (WMDE); author: Lucas Werkmeister (WMDE)):

[mediawiki/extensions/Wikibase@wmf/1.45.0-wmf.23] Restore "Implement new usage types for statement with qualifiers and references"

https://gerrit.wikimedia.org/r/1197281

Change #1197281 merged by Lucas Werkmeister (WMDE):

[mediawiki/extensions/Wikibase@wmf/1.45.0-wmf.23] Restore "Implement new usage types for statement with qualifiers and references"

https://gerrit.wikimedia.org/r/1197281

Mentioned in SAL (#wikimedia-operations) [2025-10-20T14:46:50Z] <lucaswerkmeister-wmde@deploy2002> Started scap sync-world: Backport for [[gerrit:1197281|Restore "Implement new usage types for statement with qualifiers and references" (T401290 T407684 T407744)]]

Mentioned in SAL (#wikimedia-operations) [2025-10-20T14:51:05Z] <lucaswerkmeister-wmde@deploy2002> lucaswerkmeister-wmde: Backport for [[gerrit:1197281|Restore "Implement new usage types for statement with qualifiers and references" (T401290 T407684 T407744)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.

Mentioned in SAL (#wikimedia-operations) [2025-10-20T14:55:31Z] <lucaswerkmeister-wmde@deploy2002> Finished scap sync-world: Backport for [[gerrit:1197281|Restore "Implement new usage types for statement with qualifiers and references" (T401290 T407684 T407744)]] (duration: 08m 43s)

Is it possible to revert the broken change? It has seemingly also caused T407744, where affected pages don’t just have broken references, but can’t be rendered at all.

We tried a revert but quickly had to revert that again (i.e. restore the broken change) because of production errors:

InvalidArgumentException: $aspect must use one of the XXX_USAGE constants, "CQR" given!

The volume of these errors was high enough to cause a “MediaWikiHighErrorRate: Elevated rate of MediaWiki errors” alert just from the canary hosts. (I am told this is an unusual feat to achieve.)

I think this also means that the change would’ve been fatal if last week’s train had had to be rolled back for any reason. When adding new usage constants, I think it’s imperative to first add support for parsing / reading the constant in one change, and then add code to emit the new usage kind in a separate change, and to merge them at least one train apart. Then we could’ve also reverted the second change without reverting the first one, and the existing usage in the database wouldn’t have been a problem.

As it stands now, I believe we need a single change which both reverts most of the changes of "Implement new usage types" but also retains or adds support for processing / parsing / loading CQR usages from the database. I’m not sure what it should do when encountering these usages – maybe just pretend they’re C usages for now? (The stack traces of the production errors suggest this would happen somewhere in the vicinity of EntityUsageTable::convertRowsToUsages().)

Change #1197289 had a related patch set uploaded (by Seanleong-wmde; author: Seanleong-wmde):

[mediawiki/extensions/Wikibase@master] Revert "Implement new usage types for statement with qualifiers and references". This patch temporary fixes the CQR aspects not found error as well which will be patched asap.

https://gerrit.wikimedia.org/r/1197289

Change #1197293 had a related patch set uploaded (by Lucas Werkmeister (WMDE); author: Neslihan Turan):

[mediawiki/extensions/Wikibase@master] Revert "Implement new usage types for statement with qualifiers and references"

https://gerrit.wikimedia.org/r/1197293

Change #1197274 abandoned by Lucas Werkmeister (WMDE):

[mediawiki/extensions/Wikibase@master] Revert "Implement new usage types for statement with qualifiers and references"

Reason:

recreated as I3709641e75 (same change, different Change-Id) so we can cherry-pick it to wmf.23 a second time

https://gerrit.wikimedia.org/r/1197274

Change #1197294 had a related patch set uploaded (by Lucas Werkmeister (WMDE); author: Neslihan Turan):

[mediawiki/extensions/Wikibase@wmf/1.45.0-wmf.23] Revert "Implement new usage types for statement with qualifiers and references"

https://gerrit.wikimedia.org/r/1197294

Change #1197294 merged by jenkins-bot:

[mediawiki/extensions/Wikibase@wmf/1.45.0-wmf.23] Revert "Implement new usage types for statement with qualifiers and references"

https://gerrit.wikimedia.org/r/1197294

Mentioned in SAL (#wikimedia-operations) [2025-10-20T16:02:30Z] <lucaswerkmeister-wmde@deploy2002> Started scap sync-world: Backport for [[gerrit:1197294|Revert "Implement new usage types for statement with qualifiers and references" (T401290 T407684 T407744)]]

Mentioned in SAL (#wikimedia-operations) [2025-10-20T16:07:13Z] <lucaswerkmeister-wmde@deploy2002> lucaswerkmeister-wmde: Backport for [[gerrit:1197294|Revert "Implement new usage types for statement with qualifiers and references" (T401290 T407684 T407744)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.

Mentioned in SAL (#wikimedia-operations) [2025-10-20T16:19:14Z] <lucaswerkmeister-wmde@deploy2002> Finished scap sync-world: Backport for [[gerrit:1197294|Revert "Implement new usage types for statement with qualifiers and references" (T401290 T407684 T407744)]] (duration: 16m 43s)

Lucas_Werkmeister_WMDE lowered the priority of this task from Unbreak Now! to Needs Triage.Oct 20 2025, 4:27 PM

Lowering priority as this should hopefully be fixed for now; I’ll leave the verification to the Wikidata Integration in Wikimedia projects team.

Change #1197608 had a related patch set uploaded (by Lucas Werkmeister (WMDE); author: Neslihan Turan):

[mediawiki/extensions/Wikibase@wmf/1.45.0-wmf.24] Revert "Implement new usage types for statement with qualifiers and references"

https://gerrit.wikimedia.org/r/1197608

Alternate steps to reproduce: Go to https://test.wikipedia.org/wiki/Module:Blank?action=edit, input (without saving) the module contents

local p = {}

function p.T407684()
	local refs = mw.wikibase.getEntity( 'Q42' ).claims['P202'][1].references
	mw.logObject(refs)
	mw.log('pairs loop')
	for k,v in pairs( refs ) do
		mw.log( k )
		mw.log( v )
	end
	mw.log('ipairs loop')
	for k,v in ipairs( refs ) do
		mw.log( k )
		mw.log( v )
	end
	mw.log('end T407684')
end

return p

and then call p.T407684() in the debug console and see what it prints. (Right now it’ll print no output between “ipairs loop” and “end T407684” because the bug is still present on wmf.24, hence the above revert.)

Change #1197608 merged by jenkins-bot:

[mediawiki/extensions/Wikibase@wmf/1.45.0-wmf.24] Revert "Implement new usage types for statement with qualifiers and references"

https://gerrit.wikimedia.org/r/1197608

Mentioned in SAL (#wikimedia-operations) [2025-10-21T10:17:13Z] <lucaswerkmeister-wmde@deploy2002> Started scap sync-world: Backport for [[gerrit:1197608|Revert "Implement new usage types for statement with qualifiers and references" (T401290 T407684 T407744)]]

Mentioned in SAL (#wikimedia-operations) [2025-10-21T10:21:29Z] <lucaswerkmeister-wmde@deploy2002> lucaswerkmeister-wmde: Backport for [[gerrit:1197608|Revert "Implement new usage types for statement with qualifiers and references" (T401290 T407684 T407744)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.

Mentioned in SAL (#wikimedia-operations) [2025-10-21T10:26:29Z] <lucaswerkmeister-wmde@deploy2002> Finished scap sync-world: Backport for [[gerrit:1197608|Revert "Implement new usage types for statement with qualifiers and references" (T401290 T407684 T407744)]] (duration: 09m 15s)

Change #1197293 merged by jenkins-bot:

[mediawiki/extensions/Wikibase@master] Revert "Implement new usage types for statement with qualifiers and references"

https://gerrit.wikimedia.org/r/1197293

Change #1197626 had a related patch set uploaded (by Joely Rooke WMDE; author: Joely Rooke WMDE):

[mediawiki/extensions/Wikibase@master] Revert^2 "Implement new usage types for statement with qualifiers and references"

https://gerrit.wikimedia.org/r/1197626

Change #1197626 merged by jenkins-bot:

[mediawiki/extensions/Wikibase@master] Revert^2 "Implement new usage types for statement with qualifiers and references"

https://gerrit.wikimedia.org/r/1197626

Question and comment from Norwegian Wiki:

  • The change that caused problems last time would affect qualifiers in the same way it affected references, but I didn't see any reports about that (missing references are much more noticeable). We do have some modules that use qualifiers, but it's not the same as the module used for references, so I don't know if they would have the same issue wrt. ipairs().

@refers to other user:Haros: Do you have a clue about which modules check qualifiers, and if so: Could you find some example articles that Danny here should check in addition to the three samples he mentioned?

Haros replies:

  • Marie Curie should be an example, but it looks as if it is ok. She got the physics Nobel Prize in 1903. In the infobox, it is listed as

    Utmerkelser 16 oppføringer with "vis" to expand it and there is: Nobelprisen i fysikk (1903; sammen med: Henri Becquerel, Pierre Curie)[21][19]

    That is fetched from Wikidata using for ix,snak in ipairs(qv) do and it works fine as the two names shows. (The references are there [probably?] from the fix Jon Harald Søby referred to above in this discussion.) So, I have an example where ipairs work, but I don't know of any where it does not work for qualifiers. For Fagfelt, further down in the infobox there are references, and it seems to me that these may have been fetched using pairs(), so possibly these references will be lost?

The fix for this now has been deployed, along with the original T401290 code, to all wikis. I have verified the examples kindly provided before are still working, so have closed the ticket. If anyone sees the issue reoccurring, or finds a new bug related to statements, qualifiers or references in lua, please reopen the ticket and ping me : )