Page MenuHomePhabricator

PHP Warning: foreach() argument must be of type array|object, null given / PHP Warning: Undefined array key "extensionData"
Closed, ResolvedPublicPRODUCTION ERROR

Description

Error
  • service.version: 1.46.0-wmf.1
  • timestamp: 2025-11-12T11:24:43.315Z
  • labels.phpversion: 8.1.33
  • trace.id: ae7b7223-e886-4a31-ad66-8a41efbe9160
  • Find trace.id in Logstash
labels.normalized_message
[{reqId}] {exception_url}   PHP Warning: foreach() argument must be of type array|object, null given
FrameLocationCall
from/srv/mediawiki/php-1.46.0-wmf.1/vendor/wikimedia/parsoid/src/Core/TOCData.php(309)
#0/srv/mediawiki/php-1.46.0-wmf.1/vendor/wikimedia/parsoid/src/Core/TOCData.php(309)MediaWiki\Exception\MWExceptionHandler::handleError(int, string, string, int)
#1/srv/mediawiki/php-1.46.0-wmf.1/vendor/wikimedia/json-codec/src/JsonStaticClassCodec.php(62)Wikimedia\Parsoid\Core\TOCData::newFromJsonArray(array)
#2/srv/mediawiki/php-1.46.0-wmf.1/vendor/wikimedia/json-codec/src/JsonCodec.php(410)Wikimedia\JsonCodec\JsonStaticClassCodec->newFromJsonArray(string, array)
#3/srv/mediawiki/php-1.46.0-wmf.1/vendor/wikimedia/json-codec/src/JsonCodec.php(401)Wikimedia\JsonCodec\JsonCodec->newFromJsonArray(array, null)
#4/srv/mediawiki/php-1.46.0-wmf.1/includes/json/JsonCodec.php(173)Wikimedia\JsonCodec\JsonCodec->newFromJsonArray(array)
#5/srv/mediawiki/php-1.46.0-wmf.1/includes/Parser/ParserCache.php(673)MediaWiki\Json\JsonCodec->deserialize(array, string)
#6/srv/mediawiki/php-1.46.0-wmf.1/includes/Parser/ParserCache.php(389)MediaWiki\Parser\ParserCache->restoreFromJson(string, string, string)
#7/srv/mediawiki/php-1.46.0-wmf.1/includes/Page/ParserOutputAccess.php(314)MediaWiki\Parser\ParserCache->get(MediaWiki\Page\WikiPage, MediaWiki\Parser\ParserOptions)
#8/srv/mediawiki/php-1.46.0-wmf.1/includes/Page/Article.php(721)MediaWiki\Page\ParserOutputAccess->getCachedParserOutput(MediaWiki\Page\WikiPage, MediaWiki\Parser\ParserOptions, null, array)
#9/srv/mediawiki/php-1.46.0-wmf.1/includes/Page/Article.php(537)MediaWiki\Page\Article->generateContentOutput(MediaWiki\User\User, MediaWiki\Parser\ParserOptions, int, MediaWiki\Output\OutputPage, array)
#10/srv/mediawiki/php-1.46.0-wmf.1/includes/actions/ViewAction.php(71)MediaWiki\Page\Article->view()
#11/srv/mediawiki/php-1.46.0-wmf.1/includes/actions/ActionEntryPoint.php(734)MediaWiki\Actions\ViewAction->show()
#12/srv/mediawiki/php-1.46.0-wmf.1/includes/actions/ActionEntryPoint.php(505)MediaWiki\Actions\ActionEntryPoint->performAction(MediaWiki\Page\Article, MediaWiki\Title\Title)
#13/srv/mediawiki/php-1.46.0-wmf.1/includes/actions/ActionEntryPoint.php(143)MediaWiki\Actions\ActionEntryPoint->performRequest()
#14/srv/mediawiki/php-1.46.0-wmf.1/includes/MediaWikiEntryPoint.php(184)MediaWiki\Actions\ActionEntryPoint->execute()
#15/srv/mediawiki/php-1.46.0-wmf.1/index.php(44)MediaWiki\MediaWikiEntryPoint->run()
#16/srv/mediawiki/w/index.php(3)require(string)
#17{main}
Error
  • mwversion: 1.46.0-wmf.1
  • timestamp: 2025-11-12T11:36:11.491Z
  • phpversion: 8.3.26
  • reqId: 8028de72-f032-4b66-858d-08efc8235e7b
  • Find reqId in Logstash
normalized_message
[{reqId}] {exception_url}   PHP Warning: Undefined array key "extensionData"
FrameLocationCall
from/srv/mediawiki/php-1.46.0-wmf.1/vendor/wikimedia/parsoid/src/Core/TOCData.php(309)
#0/srv/mediawiki/php-1.46.0-wmf.1/vendor/wikimedia/parsoid/src/Core/TOCData.php(309)MediaWiki\Exception\MWExceptionHandler::handleError(int, string, string, int)
#1/srv/mediawiki/php-1.46.0-wmf.1/vendor/wikimedia/json-codec/src/JsonStaticClassCodec.php(62)Wikimedia\Parsoid\Core\TOCData::newFromJsonArray(array)
#2/srv/mediawiki/php-1.46.0-wmf.1/vendor/wikimedia/json-codec/src/JsonCodec.php(410)Wikimedia\JsonCodec\JsonStaticClassCodec->newFromJsonArray(string, array)
#3/srv/mediawiki/php-1.46.0-wmf.1/vendor/wikimedia/json-codec/src/JsonCodec.php(401)Wikimedia\JsonCodec\JsonCodec->newFromJsonArray(array, null)
#4/srv/mediawiki/php-1.46.0-wmf.1/includes/json/JsonCodec.php(173)Wikimedia\JsonCodec\JsonCodec->newFromJsonArray(array)
#5/srv/mediawiki/php-1.46.0-wmf.1/includes/Parser/ParserCache.php(673)MediaWiki\Json\JsonCodec->deserialize(array, string)
#6/srv/mediawiki/php-1.46.0-wmf.1/includes/Parser/ParserCache.php(389)MediaWiki\Parser\ParserCache->restoreFromJson(string, string, string)
#7/srv/mediawiki/php-1.46.0-wmf.1/includes/Page/ParserOutputAccess.php(314)MediaWiki\Parser\ParserCache->get(MediaWiki\Page\WikiFilePage, MediaWiki\Parser\ParserOptions)
#8/srv/mediawiki/php-1.46.0-wmf.1/includes/Page/Article.php(717)MediaWiki\Page\ParserOutputAccess->getCachedParserOutput(MediaWiki\Page\WikiFilePage, MediaWiki\Parser\ParserOptions, null, array)
#9/srv/mediawiki/php-1.46.0-wmf.1/includes/Page/Article.php(537)MediaWiki\Page\Article->generateContentOutput(MediaWiki\User\User, MediaWiki\Parser\ParserOptions, int, MediaWiki\Output\OutputPage, array)
#10/srv/mediawiki/php-1.46.0-wmf.1/includes/Page/ImagePage.php(99)MediaWiki\Page\Article->view()
#11/srv/mediawiki/php-1.46.0-wmf.1/includes/Page/Article.php(1952)MediaWiki\Page\ImagePage->view()
#12/srv/mediawiki/php-1.46.0-wmf.1/includes/actions/RenderAction.php(31)MediaWiki\Page\Article->render()
#13/srv/mediawiki/php-1.46.0-wmf.1/includes/actions/ActionEntryPoint.php(734)MediaWiki\Actions\RenderAction->show()
#14/srv/mediawiki/php-1.46.0-wmf.1/includes/actions/ActionEntryPoint.php(505)MediaWiki\Actions\ActionEntryPoint->performAction(MediaWiki\Page\ImagePage, MediaWiki\Title\Title)
#15/srv/mediawiki/php-1.46.0-wmf.1/includes/actions/ActionEntryPoint.php(143)MediaWiki\Actions\ActionEntryPoint->performRequest()
#16/srv/mediawiki/php-1.46.0-wmf.1/includes/MediaWikiEntryPoint.php(184)MediaWiki\Actions\ActionEntryPoint->execute()
#17/srv/mediawiki/php-1.46.0-wmf.1/index.php(44)MediaWiki\MediaWikiEntryPoint->run()
#18/srv/mediawiki/w/index.php(3)require(string)
#19{main}
Impact

Large at the time of the spike. Per scap output:

11:09:36 Logstash checker Counted 81 error(s) in the last 20 seconds. The threshold is 10.

Notes

Spike on old 1.46.0-wmf.1 while trying to tackle 1.46.0-wmf.2 deployment.
Filing this ticket as I do not see any recent changes in the mentioned code lines and cannot find an existing ticket, so I assume this remains an issue.

Event Timeline

Aklapper triaged this task as High priority.
Aklapper raised the priority of this task from High to Unbreak Now!.

Given the large numbers in this spike and the scap output

11:09:36 Logstash checker Counted 81 error(s) in the last 20 seconds. The threshold is 10.

I am going to UBN this. Feel free to lower the priority.

This looks like old code from before https://gerrit.wikimedia.org/r/1201327 getting hit with new data. I'm not sure what to do about this. This should resolve itself when all code is updated.

Filing this ticket as I do not see any recent changes in the mentioned code lines and cannot find an existing ticket, so I assume this remains an issue.

FWIW, I don’t see any occurrences on wmf.2 in logstash – though given the extreme volume of these warnings I don’t disagree with the UBN! assessment either, even if it turns out to resolve itself as the train continues to roll out.

I'm not sure what we would do to solve this other than rolling out the change in two stages: First change the consumer to be fine with the array key missing, then change the other half of the code.

But it's really only a warning with no observable effect anywhere. Can't we just filter and ignore it for one or two days?

Oh, this should be transient, caused by 314a47dc6db609a7b06b5787e9c0cdba9aedc43b. Once some JSON has been written to ParserOutput etc without extensionData, it would cause warnings when we tried to deserialize it with an older version.

As @thiemowmde says, I should have deployed this in two phases. But I can also fix this by backporting deserialization support to wmf.1, if we want to suppress the noise quicker.

Change #1204630 had a related patch set uploaded (by C. Scott Ananian; author: C. Scott Ananian):

[mediawiki/services/parsoid@master] Don't suppress extensionData in TOCData serialization

https://gerrit.wikimedia.org/r/1204630

Hm, our ParserCache compatibility policy dictates the deserialization compatibility needs to be backported as well. I'm proposing to merge the above patch, which partially reverts the original 314a47dc6db609a7b06b5787e9c0cdba9aedc43b patch but leaves in the deserialization support. I'd have to tag a new version of parsoid and backport that to mediawiki-vendor, then we'd backport the new mediawiki-vendor to wmf.2.

The 1204630 patch should be merged to master regardless and I should backport the deserialization support to all the active release branches before i move forward with removing the 'extensionData' property from the serialized form.

Change #1204631 had a related patch set uploaded (by C. Scott Ananian; author: C. Scott Ananian):

[mediawiki/core@master] DNM: experiment at another solution to T409910

https://gerrit.wikimedia.org/r/1204631

^^^ I thought maybe I could just force all serialized TOCData to have extensionData to avoid the issues on rollback, but that patch ran into issues with CI tests for deserialization fidelity, so ignore 1204631.

Change #1204635 had a related patch set uploaded (by C. Scott Ananian; author: C. Scott Ananian):

[mediawiki/vendor@master] Bump wikimedia/parsoid to 0.23.0-a4

https://gerrit.wikimedia.org/r/1204635

The patches to backport are Bump wikimedia/parsoid to 0.23.0-a4 (1204635) and Bump wikimedia/parsoid to 0.23.0-a4 (1204636), but as described at https://wikitech.wikimedia.org/wiki/Parsoid#If_the_train_branch_has_already_been_cut we need to wait until these patches merge to the master branch before we can backport them to wmf/1.46.0-wmf.2 or else CI will get confused.

Change #1204635 merged by jenkins-bot:

[mediawiki/vendor@master] Bump wikimedia/parsoid to 0.23.0-a4

https://gerrit.wikimedia.org/r/1204635

Change #1204645 had a related patch set uploaded (by C. Scott Ananian; author: C. Scott Ananian):

[mediawiki/vendor@wmf/1.46.0-wmf.2] Bump wikimedia/parsoid to 0.23.0-a4

https://gerrit.wikimedia.org/r/1204645

Cherry-picks to wmf.2 are at Bump wikimedia/parsoid to 0.23.0-a4 (1204646) and Bump wikimedia/parsoid to 0.23.0-a4 (1204645).

Do we want to wait until the "late backport window" or backport this early?

Change #1204645 merged by jenkins-bot:

[mediawiki/vendor@wmf/1.46.0-wmf.2] Bump wikimedia/parsoid to 0.23.0-a4

https://gerrit.wikimedia.org/r/1204645

Mentioned in SAL (#wikimedia-operations) [2025-11-12T19:49:55Z] <cscott@deploy2002> Started scap sync-world: Backport for [[gerrit:1204645|Bump wikimedia/parsoid to 0.23.0-a4 (T409910 T409607)]], [[gerrit:1204646|Bump wikimedia/parsoid to 0.23.0-a4 (T409607)]]

Mentioned in SAL (#wikimedia-operations) [2025-11-12T19:53:45Z] <cscott@deploy2002> cscott: Backport for [[gerrit:1204645|Bump wikimedia/parsoid to 0.23.0-a4 (T409910 T409607)]], [[gerrit:1204646|Bump wikimedia/parsoid to 0.23.0-a4 (T409607)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.

Mentioned in SAL (#wikimedia-operations) [2025-11-12T20:01:30Z] <cscott@deploy2002> Finished scap sync-world: Backport for [[gerrit:1204645|Bump wikimedia/parsoid to 0.23.0-a4 (T409910 T409607)]], [[gerrit:1204646|Bump wikimedia/parsoid to 0.23.0-a4 (T409607)]] (duration: 11m 35s)

cscott claimed this task.

This has now been backported to wmf.2.

wmf.2 now no longer generates ParserCache entries with missing TOC extensiondata. Those ParserCache entries cause issues when rolling back to wmf.1.

We might still have some residual log spam caused by ParserCache entries that were generated during a previous roll-forward, since those are still stuck in the ParserCache. wmf.2 can handle those entries, though (and wmf.1 handles them fine, it just logspams), so the presence of this logspam should entirely cease once we're fully at wmf.2.

Change #1204631 abandoned by C. Scott Ananian:

[mediawiki/core@master] DNM: experiment at another solution to T409910

Reason:

not a good experiment

https://gerrit.wikimedia.org/r/1204631

Change #1204630 merged by jenkins-bot:

[mediawiki/services/parsoid@master] Don't suppress extensionData in TOCData serialization

https://gerrit.wikimedia.org/r/1204630

Change #1206434 had a related patch set uploaded (by Subramanya Sastry; author: Subramanya Sastry):

[mediawiki/vendor@master] Bump wikimedia/parsoid to 0.23.0-a5

https://gerrit.wikimedia.org/r/1206434

Change #1206434 merged by jenkins-bot:

[mediawiki/vendor@master] Bump wikimedia/parsoid to 0.23.0-a5

https://gerrit.wikimedia.org/r/1206434