Note that null/nil values in tabular data sets are not yet handled correctly by the Lua transformation stage; this is a known and documented limitation.

Lua does not allow storing nil values in tables; it's equivalent to removing the associated key. This produces a sort of "sparse array" which the standard Lua array-like table iteration primitives don't support.

Proper fix would probably include either something very complex (rewriting the null values into a sigil) or something that requires both fixes to Scribunto's data bridge and users' transform code to iterate correctly.

Will see if we can improve that error message though...

Can confirm the non-useful error message on a local copy:

[18f306fe-6c20-4ae5-806e-002074bdfd8d] 2025-07-07 17:39:47: Fatal exception of type "Error"

:D

backtrace locally:

[21fa72edd331cd3ad03e9ed3] /w/index.php/Data:Crude_Oil_Production.chart Error: Call to a member function getContent() on null

Backtrace:

from /var/www/html/w/extensions/Chart/src/ParserFunction.php(302)
#0 /var/www/html/w/extensions/Chart/src/JCChartContentView.php(64): MediaWiki\Extension\Chart\ParserFunction->renderChartForDefinitionContent()
#1 /var/www/html/w/extensions/JsonConfig/includes/JCContentHandler.php(179): MediaWiki\Extension\Chart\JCChartContentView->valueToHtml()
#2 /var/www/html/w/includes/content/ContentHandler.php(1694): JsonConfig\JCContentHandler->fillParserOutput()
#3 /var/www/html/w/includes/content/Renderer/ContentRenderer.php(75): MediaWiki\Content\ContentHandler->getParserOutput()
#4 /var/www/html/w/includes/Revision/RenderedRevision.php(260): MediaWiki\Content\Renderer\ContentRenderer->getParserOutput()
#5 /var/www/html/w/includes/Revision/RenderedRevision.php(233): MediaWiki\Revision\RenderedRevision->getSlotParserOutputUncached()
#6 /var/www/html/w/includes/Revision/RevisionRenderer.php(238): MediaWiki\Revision\RenderedRevision->getSlotParserOutput()
#7 /var/www/html/w/includes/Revision/RevisionRenderer.php(171): MediaWiki\Revision\RevisionRenderer->combineSlotOutput()
#8 /var/www/html/w/includes/Revision/RenderedRevision.php(196): MediaWiki\Revision\RevisionRenderer->MediaWiki\Revision\{closure}()
#9 /var/www/html/w/includes/page/ParserOutputAccess.php(591): MediaWiki\Revision\RenderedRevision->getRevisionParserOutput()
#10 /var/www/html/w/includes/page/ParserOutputAccess.php(677): MediaWiki\Page\ParserOutputAccess->renderRevision()
#11 /var/www/html/w/includes/poolcounter/PoolCounterWorkViaCallback.php(81): MediaWiki\Page\ParserOutputAccess->MediaWiki\Page\{closure}()
#12 /var/www/html/w/includes/poolcounter/PoolCounterWork.php(173): MediaWiki\PoolCounter\PoolCounterWorkViaCallback->doWork()
#13 /var/www/html/w/includes/page/ParserOutputAccess.php(495): MediaWiki\PoolCounter\PoolCounterWork->execute()
#14 /var/www/html/w/includes/page/Article.php(835): MediaWiki\Page\ParserOutputAccess->getParserOutput()
#15 /var/www/html/w/includes/page/Article.php(551): MediaWiki\Page\Article->generateContentOutput()
#16 /var/www/html/w/includes/actions/ViewAction.php(84): MediaWiki\Page\Article->view()
#17 /var/www/html/w/includes/actions/ActionEntryPoint.php(728): MediaWiki\Actions\ViewAction->show()
#18 /var/www/html/w/includes/actions/ActionEntryPoint.php(505): MediaWiki\Actions\ActionEntryPoint->performAction()
#19 /var/www/html/w/includes/actions/ActionEntryPoint.php(143): MediaWiki\Actions\ActionEntryPoint->performRequest()
#20 /var/www/html/w/includes/MediaWikiEntryPoint.php(198): MediaWiki\Actions\ActionEntryPoint->execute()
#21 /var/www/html/w/index.php(58): MediaWiki\MediaWikiEntryPoint->run()
#22 {main}

looks like JCContentLoader->load() is returning a null instead of a ContentWrapper in its Status value, which shouldn't happen if $status->isOk() ...

I think it's this in JCTransformer::execute:

		if ( $status->isOK() ) {
			$content = $status->getValue();
			if ( $content->isValid() ) {
				$status = Status::newGood( JCContentWrapper::newFromContent(
					$title, $content, $parser->getOutput() ) );
			} else {
				$status = $content->getStatus();
			}
		}

If the content isn't valid it goes through the branch that grabs $content->getStatus() which is supposed to indicate errors in the markup however this may be going wrong.

Yeah, this is coming back marked as 'ok' but also with 'errors' on it I guess as warnings. Lemme see where that's coming from...

Ok, looks like to be certain I should be checking for isGood not isOk :D

Fixing that I now get pretty error displays for all the short columns due to the null/nil handling problem, which is at least an improvement:

List "data[0]" has 6 values, but must have 7 values, the same number of values as "schema/fields"
List "data[1]" has 6 values, but must have 7 values, the same number of values as "schema/fields"
List "data[2]" has 6 values, but must have 7 values, the same number of values as "schema/fields"
List "data[3]" has 6 values, but must have 7 values, the same number of values as "schema/fields"
List "data[4]" has 6 values, but must have 7 values, the same number of values as "schema/fields"
List "data[5]" has 6 values, but must have 7 values, the same number of values as "schema/fields"
List "data[6]" has 6 values, but must have 7 values, the same number of values as "schema/fields"
List "data[7]" has 6 values, but must have 7 values, the same number of values as "schema/fields"
List "data[8]" has 6 values, but must have 7 values, the same number of values as "schema/fields"
List "data[9]" has 6 values, but must have 7 values, the same number of values as "schema/fields"
List "data[10]" has 6 values, but must have 7 values, the same number of values as "schema/fields"
List "data[11]" has 6 values, but must have 7 values, the same number of values as "schema/fields"
List "data[12]" has 6 values, but must have 7 values, the same number of values as "schema/fields"
List "data[13]" has 6 values, but must have 7 values, the same number of values as "schema/fields"
List "data[14]" has 6 values, but must have 7 values, the same number of values as "schema/fields"
List "data[15]" has 6 values, but must have 7 values, the same number of values as "schema/fields"
List "data[16]" has 6 values, but must have 7 values, the same number of values as "schema/fields"
List "data[17]" has 6 values, but must have 7 values, the same number of values as "schema/fields"
List "data[18]" has 6 values, but must have 7 values, the same number of values as "schema/fields"

and it doesn't break overall page rendering. I'll merge this as a production fix and then see about better ways to handle the conversion. The TabUtils:select function _should_ work as written on this data, it's just the output from Lua back to PHP that's definitely wrong and I may be able to fix that.

Change #1166888 had a related patch set uploaded (by Bvibber; author: Bvibber):

[mediawiki/extensions/Chart@master] Fix for validation error display in transformed chart data

https://gerrit.wikimedia.org/r/1166888

^ above patch will fix the error display so it doesn't break editing, and shows you all the rows that got broken by the PHP/Lua/PHP conversion. :D

Still need to devise a way to round-trip these cleanly (basically seeing the 'holes' in the returned data and filling them back in with nulls on the PHP side; may or may not have to fix something in Scribunto to transfer them right)

Change #1166888 merged by jenkins-bot:

[mediawiki/extensions/Chart@master] Fix for validation error display in transformed chart data

https://gerrit.wikimedia.org/r/1166888

Change #1166895 had a related patch set uploaded (by Bvibber; author: Bvibber):

[mediawiki/extensions/Chart@wmf/1.45.0-wmf.8] Fix for validation error display in transformed chart data

https://gerrit.wikimedia.org/r/1166895

Change #1166895 merged by jenkins-bot:

[mediawiki/extensions/Chart@wmf/1.45.0-wmf.8] Fix for validation error display in transformed chart data

https://gerrit.wikimedia.org/r/1166895

Mentioned in SAL (#wikimedia-operations) [2025-07-07T18:51:12Z] <bvibber@deploy1003> Started scap sync-world: Backport for [[gerrit:1166895|Fix for validation error display in transformed chart data (T398597)]]

Mentioned in SAL (#wikimedia-operations) [2025-07-07T18:53:18Z] <bvibber@deploy1003> bvibber: Backport for [[gerrit:1166895|Fix for validation error display in transformed chart data (T398597)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.

Mentioned in SAL (#wikimedia-operations) [2025-07-07T18:59:53Z] <bvibber@deploy1003> Finished scap sync-world: Backport for [[gerrit:1166895|Fix for validation error display in transformed chart data (T398597)]] (duration: 08m 40s)

^ ok the error reporting fix is up, so it no longer _completely_ breaks page rendering but shows in-place the list of validation errors of the post-processed data.

I think I can devise a clever fix for the data conversion that won't be too invasive, lemme try some things later today.

Change #1166917 had a related patch set uploaded (by Bvibber; author: Bvibber):

[mediawiki/extensions/JsonConfig@master] Support null values in data columns in transform output

https://gerrit.wikimedia.org/r/1166917

https://commons.wikimedia.org/w/index.php?title=Module:TabUtils&diff=prev&oldid=1053761191

^ Here's a fix for TabUtils which I've confirmed with a local setup where I've applied a server-side patch for the null handling as well. Still have to write tests for the server-side patch

With the patch locally installed I get a successful rendering, with the null segments skipped in the rendering as expected:

Hopefully can push this out today, otherwise try tomorrow.

Test case added, ready for code review.

Change #1166917 merged by jenkins-bot:

[mediawiki/extensions/JsonConfig@master] Support null values in data columns in transform output

https://gerrit.wikimedia.org/r/1166917

Change #1166942 had a related patch set uploaded (by Bvibber; author: Bvibber):

[mediawiki/extensions/JsonConfig@wmf/1.45.0-wmf.8] Support null values in data columns in transform output

https://gerrit.wikimedia.org/r/1166942

It's late in the day, so I'll deploy the fix in the morning Pacific time. :)

Change #1166942 merged by jenkins-bot:

[mediawiki/extensions/JsonConfig@wmf/1.45.0-wmf.8] Support null values in data columns in transform output

https://gerrit.wikimedia.org/r/1166942

Mentioned in SAL (#wikimedia-operations) [2025-07-08T15:22:34Z] <bvibber@deploy1003> Started scap sync-world: Backport for [[gerrit:1166942|Support null values in data columns in transform output (T398597)]]

Mentioned in SAL (#wikimedia-operations) [2025-07-08T15:24:40Z] <bvibber@deploy1003> bvibber: Backport for [[gerrit:1166942|Support null values in data columns in transform output (T398597)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.

Mentioned in SAL (#wikimedia-operations) [2025-07-08T15:31:26Z] <bvibber@deploy1003> Finished scap sync-world: Backport for [[gerrit:1166942|Support null values in data columns in transform output (T398597)]] (duration: 08m 52s)