Page MenuHomePhabricator

WikibaseClient.php: PHP Notice: Undefined index:
Closed, ResolvedPublic

Description

Error

Request ID: XV-w0ApAMDgAACUbsB4AAADY
Request URL: ru.wikipedia.org/w/api.php?action=opensearch&search=...

message
[{exception_id}] {exception_url}   ErrorException from line 1107 of /srv/mediawiki/php-1.34.0-wmf.19/extensions/Wikibase/client/includes/WikibaseClient.php: PHP Notice: Undefined index:
trace
#0 /srv/mediawiki/php-1.34.0-wmf.19/extensions/Wikibase/client/includes/WikibaseClient.php(1107): MWExceptionHandler::handleError(integer, string, string, integer, array)
#1 /srv/mediawiki/php-1.34.0-wmf.19/extensions/Wikibase/client/includes/WikibaseClient.php(1357): Wikibase\Client\WikibaseClient->getRepoItemUriParser()
#2 /srv/mediawiki/php-1.34.0-wmf.19/extensions/Wikibase/client/includes/WikibaseClient.php(1344): Wikibase\Client\WikibaseClient->getDataAccessSnakFormatterFactory()
#3 /srv/mediawiki/php-1.34.0-wmf.19/extensions/Wikibase/client/includes/WikibaseClient.php(1368): Wikibase\Client\WikibaseClient->getStatementGroupRendererFactory()
#4 /srv/mediawiki/php-1.34.0-wmf.19/extensions/Wikibase/client/includes/DataAccess/ParserFunctions/Runner.php(174): Wikibase\Client\WikibaseClient->getPropertyParserFunctionRunner()
#5 /srv/mediawiki/php-1.34.0-wmf.19/extensions/Wikibase/client/includes/Hooks/ParserFunctionRegistrant.php(66): Wikibase\Client\DataAccess\ParserFunctions\Runner::renderEscapedPlainText(Parser, PPTemplateFrame_Hash, array)
#6 /srv/mediawiki/php-1.34.0-wmf.19/includes/parser/Parser.php(3624): Wikibase\Client\Hooks\ParserFunctionRegistrant->Wikibase\Client\Hooks\{closure}(Parser, PPTemplateFrame_Hash, array)
#7 /srv/mediawiki/php-1.34.0-wmf.19/extensions/Scribunto/includes/engines/LuaCommon/LuaCommon.php(780): Parser->callParserFunction(PPTemplateFrame_Hash, string, array)
#8 /srv/mediawiki/php-1.34.0-wmf.19/extensions/Scribunto/includes/engines/LuaSandbox/Engine.php(391): Scribunto_LuaEngine->callParserFunction(string, string, array)
#9 [internal function]: Scribunto_LuaSandboxCallback->__call(string, array)
#10 /srv/mediawiki/php-1.34.0-wmf.19/extensions/Scribunto/includes/engines/LuaSandbox/Engine.php(314): LuaSandboxFunction->call(LuaSandboxFunction)
#11 /srv/mediawiki/php-1.34.0-wmf.19/extensions/Scribunto/includes/engines/LuaCommon/LuaCommon.php(296): Scribunto_LuaSandboxInterpreter->callFunction(LuaSandboxFunction, LuaSandboxFunction)
#12 /srv/mediawiki/php-1.34.0-wmf.19/extensions/Scribunto/includes/engines/LuaCommon/LuaCommon.php(983): Scribunto_LuaEngine->executeFunctionChunk(LuaSandboxFunction, PPTemplateFrame_Hash)
#13 /srv/mediawiki/php-1.34.0-wmf.19/extensions/Scribunto/includes/common/Hooks.php(128): Scribunto_LuaModule->invoke(string, PPTemplateFrame_Hash)
#14 /srv/mediawiki/php-1.34.0-wmf.19/includes/parser/Parser.php(3624): ScribuntoHooks::invokeHook(Parser, PPTemplateFrame_Hash, array)
#15 /srv/mediawiki/php-1.34.0-wmf.19/includes/parser/Parser.php(3329): Parser->callParserFunction(PPTemplateFrame_Hash, string, array)
#16 /srv/mediawiki/php-1.34.0-wmf.19/includes/parser/PPFrame_Hash.php(254): Parser->braceSubstitution(array, PPTemplateFrame_Hash)
#17 /srv/mediawiki/php-1.34.0-wmf.19/includes/parser/Parser.php(3505): PPFrame_Hash->expand(PPNode_Hash_Tree)
#18 /srv/mediawiki/php-1.34.0-wmf.19/includes/parser/PPFrame_Hash.php(254): Parser->braceSubstitution(array, PPTemplateFrame_Hash)
#19 /srv/mediawiki/php-1.34.0-wmf.19/includes/parser/Parser.php(3246): PPFrame_Hash->expand(PPNode_Hash_Tree)
#20 /srv/mediawiki/php-1.34.0-wmf.19/includes/parser/PPFrame_Hash.php(254): Parser->braceSubstitution(array, PPTemplateFrame_Hash)
#21 /srv/mediawiki/php-1.34.0-wmf.19/includes/parser/PPTemplateFrame_Hash.php(167): PPFrame_Hash->expand(PPNode_Hash_Tree, integer)
#22 /srv/mediawiki/php-1.34.0-wmf.19/includes/parser/PPTemplateFrame_Hash.php(179): PPTemplateFrame_Hash->getNamedArgument(string)
#23 /srv/mediawiki/php-1.34.0-wmf.19/includes/parser/PPTemplateFrame_Hash.php(111): PPTemplateFrame_Hash->getArgument(string)
#24 /srv/mediawiki/php-1.34.0-wmf.19/extensions/Scribunto/includes/engines/LuaCommon/LuaCommon.php(693): PPTemplateFrame_Hash->getArguments()
#25 /srv/mediawiki/php-1.34.0-wmf.19/extensions/Scribunto/includes/engines/LuaSandbox/Engine.php(391): Scribunto_LuaEngine->getAllExpandedArguments(string)
#26 [internal function]: Scribunto_LuaSandboxCallback->__call(string, array)
#27 /srv/mediawiki/php-1.34.0-wmf.19/extensions/Scribunto/includes/engines/LuaSandbox/Engine.php(314): LuaSandboxFunction->call(LuaSandboxFunction)
#28 /srv/mediawiki/php-1.34.0-wmf.19/extensions/Scribunto/includes/engines/LuaCommon/LuaCommon.php(296): Scribunto_LuaSandboxInterpreter->callFunction(LuaSandboxFunction, LuaSandboxFunction)
#29 /srv/mediawiki/php-1.34.0-wmf.19/extensions/Scribunto/includes/engines/LuaCommon/LuaCommon.php(983): Scribunto_LuaEngine->executeFunctionChunk(LuaSandboxFunction, PPTemplateFrame_Hash)
#30 /srv/mediawiki/php-1.34.0-wmf.19/extensions/Scribunto/includes/common/Hooks.php(128): Scribunto_LuaModule->invoke(string, PPTemplateFrame_Hash)
#31 /srv/mediawiki/php-1.34.0-wmf.19/includes/parser/Parser.php(3624): ScribuntoHooks::invokeHook(Parser, PPTemplateFrame_Hash, array)
#32 /srv/mediawiki/php-1.34.0-wmf.19/includes/parser/Parser.php(3329): Parser->callParserFunction(PPTemplateFrame_Hash, string, array)
#33 /srv/mediawiki/php-1.34.0-wmf.19/includes/parser/PPFrame_Hash.php(254): Parser->braceSubstitution(array, PPTemplateFrame_Hash)
#34 /srv/mediawiki/php-1.34.0-wmf.19/includes/parser/Parser.php(3505): PPFrame_Hash->expand(PPNode_Hash_Tree)
#35 /srv/mediawiki/php-1.34.0-wmf.19/includes/parser/PPFrame_Hash.php(254): Parser->braceSubstitution(array, PPTemplateFrame_Hash)
#36 /srv/mediawiki/php-1.34.0-wmf.19/includes/parser/Parser.php(3505): PPFrame_Hash->expand(PPNode_Hash_Tree)
#37 /srv/mediawiki/php-1.34.0-wmf.19/includes/parser/PPFrame_Hash.php(254): Parser->braceSubstitution(array, PPFrame_Hash)
#38 /srv/mediawiki/php-1.34.0-wmf.19/includes/parser/Parser.php(3143): PPFrame_Hash->expand(PPNode_Hash_Tree, integer)
#39 /srv/mediawiki/php-1.34.0-wmf.19/includes/parser/Parser.php(1454): Parser->replaceVariables(string)
#40 /srv/mediawiki/php-1.34.0-wmf.19/includes/parser/Parser.php(576): Parser->internalParse(string)
#41 /srv/mediawiki/php-1.34.0-wmf.19/includes/content/WikitextContent.php(365): Parser->parse(string, Title, ParserOptions, boolean, boolean, NULL)
#42 /srv/mediawiki/php-1.34.0-wmf.19/includes/content/AbstractContent.php(555): WikitextContent->fillParserOutput(Title, NULL, ParserOptions, boolean, ParserOutput)
#43 /srv/mediawiki/php-1.34.0-wmf.19/includes/api/ApiParse.php(590): AbstractContent->getParserOutput(Title, NULL, ParserOptions)
#44 /srv/mediawiki/php-1.34.0-wmf.19/includes/api/ApiParse.php(164): ApiParse->getParsedContent(WikiPage, ParserOptions, boolean, NULL, NULL, boolean)
#45 /srv/mediawiki/php-1.34.0-wmf.19/includes/api/ApiMain.php(1589): ApiParse->execute()
#46 /srv/mediawiki/php-1.34.0-wmf.19/includes/api/ApiMain.php(502): ApiMain->executeAction()
#47 /srv/mediawiki/php-1.34.0-wmf.19/extensions/TextExtracts/includes/ApiQueryExtracts.php(235): ApiMain->execute()
#48 /srv/mediawiki/php-1.34.0-wmf.19/extensions/TextExtracts/includes/ApiQueryExtracts.php(159): TextExtracts\ApiQueryExtracts->parse(WikiPage)
#49 /srv/mediawiki/php-1.34.0-wmf.19/extensions/TextExtracts/includes/ApiQueryExtracts.php(96): TextExtracts\ApiQueryExtracts->getExtract(Title)
#50 /srv/mediawiki/php-1.34.0-wmf.19/includes/api/ApiQuery.php(250): TextExtracts\ApiQueryExtracts->execute()
#51 /srv/mediawiki/php-1.34.0-wmf.19/includes/api/ApiMain.php(1589): ApiQuery->execute()
#52 /srv/mediawiki/php-1.34.0-wmf.19/includes/api/ApiMain.php(502): ApiMain->executeAction()
#53 /srv/mediawiki/php-1.34.0-wmf.19/extensions/TextExtracts/includes/Hooks.php(37): ApiMain->execute()
#54 /srv/mediawiki/php-1.34.0-wmf.19/includes/Hooks.php(174): TextExtracts\Hooks::onApiOpenSearchSuggest(array)
#55 /srv/mediawiki/php-1.34.0-wmf.19/includes/Hooks.php(202): Hooks::callHook(string, array, array, NULL)
#56 /srv/mediawiki/php-1.34.0-wmf.19/includes/api/ApiOpenSearch.php(94): Hooks::run(string, array)
#57 /srv/mediawiki/php-1.34.0-wmf.19/includes/api/ApiMain.php(1589): ApiOpenSearch->execute()
#58 /srv/mediawiki/php-1.34.0-wmf.19/includes/api/ApiMain.php(533): ApiMain->executeAction()
#59 /srv/mediawiki/php-1.34.0-wmf.19/includes/api/ApiMain.php(504): ApiMain->executeActionWithErrorHandling()
#60 /srv/mediawiki/php-1.34.0-wmf.19/api.php(86): ApiMain->execute()
#61 /srv/mediawiki/w/api.php(3): require(string)
#62 {main}
Impact

1 hit in the last hour, 9 in the last 24 hours, 60 in the last 7 days.

Notes

Not blocking the train since it's not new.

Event Timeline

Restricted Application added a subscriber: Aklapper. · View Herald TranscriptFri, Aug 23, 3:03 PM

The offending line:

private function getRepoItemUriParser() {
    // B/C compatibility, should be removed soon                                                                                                                         
    // TODO: Move to check repo that has item entity not the default repo                                                                                                
    return new SuffixEntityIdParser(
        $this->getRepositoryDefinitions()->getConceptBaseUris()[''], // <-------
        new ItemIdParser()
    );
}

That's weird:

ladsgroup@mwmaint1002:~$ mwscript eval.php --wiki=ruwiki
> $client = \Wikibase\Client\WikibaseClient::getDefaultInstance();

> var_dump( $client->getRepositoryDefinitions()->getConceptBaseUris() );
array(1) {
  [""]=>
  string(31) "http://www.wikidata.org/entity/"
}

It exists...

My thought is that it can be either a race condition that repositoryDefinitions doesn't get properly initialized or we have another case of opcache corruption. Is it happening on php7 or hhvm?

Yup. The XV-w0ApAMDgAACUbsB4AAADY logstash says: 7.2.16-1+0~20190307202415.17+stretch~1.gbpa7be82+wmf1

All errors are php7: https://logstash.wikimedia.org/goto/b64aca4c135996e93f97b70a2079d7d8

mmodell changed the subtype of this task from "Task" to "Production Error".Wed, Aug 28, 11:05 PM

Smells like T229433. Which is also about '' array index, and PHP 7.2. It's obviously a bug in PHP 7.2, but I've not been able to find evidence that it is due to opcache/T224491.

Joe added a subscriber: Joe.Thu, Sep 12, 8:59 AM

no please do not. This bug has nothing to do with opcache corruption during deployments. The problem happens only episodically, not for all requests that happen to hit that code path, so I doubt that's the cause.

Joe added a comment.Thu, Sep 12, 9:00 AM

Smells like T229433. Which is also about '' array index, and PHP 7.2. It's obviously a bug in PHP 7.2, but I've not been able to find evidence that it is due to opcache/T224491.

Yes, it looks like it's a weird bug with arrays with an empty string as key. But it only seems to happen in specific conditions, and not consistently. Correct?

no please do not. This bug has nothing to do with opcache corruption during deployments. The problem happens only episodically, not for all requests that happen to hit that code path, so I doubt that's the cause.

you're right. Sorry. I meant this is a bug in php7 and there's nothing wikidata devs can do to help :(

hashar added a subscriber: hashar.Thu, Sep 12, 2:45 PM

I have found some other occurrences from WikibaseClient.php, still related to an array keys being an empty string. Both from php 7.2.22-1+0~20190902.26+debian9~1.gbpd64eb7+wmf1

/srv/mediawiki/php-1.34.0-wmf.21/extensions/Wikibase/client/includes/WikibaseClient.php:1591
PHP Notice: Undefined index:

#0 /srv/mediawiki/php-1.34.0-wmf.21/extensions/Wikibase/client/includes/WikibaseClient.php(1591): MWExceptionHandler::handleError(integer, string, string, integer, array)
#1 /srv/mediawiki/php-1.34.0-wmf.21/extensions/Wikibase/client/includes/WikibaseClient.php(1627): Wikibase\Client\WikibaseClient->getItemTermIndex()
#2 /srv/mediawiki/php-1.34.0-wmf.21/extensions/Wikibase/client/WikibaseClient.php(236): Wikibase\Client\WikibaseClient->getDescriptionLookup()
#3 /srv/mediawiki/php-1.34.0-wmf.21/includes/api/ApiModuleManager.php(191): {closure}(ApiQuery, string)
#4 /srv/mediawiki/php-1.34.0-wmf.21/includes/api/ApiModuleManager.php(167): ApiModuleManager->instantiateModule(string, string, Closure)
#5 /srv/mediawiki/php-1.34.0-wmf.21/includes/api/ApiQuery.php(307): ApiModuleManager->getModule(string, string)
#6 /srv/mediawiki/php-1.34.0-wmf.21/includes/api/ApiQuery.php(218): ApiQuery->instantiateModules(array, string)
#7 /srv/mediawiki/php-1.34.0-wmf.21/includes/api/ApiMain.php(1596): ApiQuery->execute()
#8 /srv/mediawiki/php-1.34.0-wmf.21/includes/api/ApiMain.php(534): ApiMain->executeAction()
#9 /srv/mediawiki/php-1.34.0-wmf.21/includes/api/ApiMain.php(505): ApiMain->executeActionWithErrorHandling()
#10 /srv/mediawiki/php-1.34.0-wmf.21/api.php(86): ApiMain->execute()
#11 /srv/mediawiki/w/api.php(3): require(string)
#12 {main}

And:

#0 /srv/mediawiki/php-1.34.0-wmf.21/extensions/Wikibase/client/includes/WikibaseClient.php(1591): MWExceptionHandler::handleError(integer, string, string, integer, array)
#1 /srv/mediawiki/php-1.34.0-wmf.21/extensions/Wikibase/client/includes/WikibaseClient.php(1627): Wikibase\Client\WikibaseClient->getItemTermIndex()
#2 /srv/mediawiki/php-1.34.0-wmf.21/extensions/Wikibase/client/WikibaseClient.php(236): Wikibase\Client\WikibaseClient->getDescriptionLookup()
#3 /srv/mediawiki/php-1.34.0-wmf.21/includes/api/ApiModuleManager.php(191): {closure}(ApiQuery, string)
#4 /srv/mediawiki/php-1.34.0-wmf.21/includes/api/ApiModuleManager.php(167): ApiModuleManager->instantiateModule(string, string, Closure)
#5 /srv/mediawiki/php-1.34.0-wmf.21/includes/api/ApiQuery.php(307): ApiModuleManager->getModule(string, string)
#6 /srv/mediawiki/php-1.34.0-wmf.21/includes/api/ApiQuery.php(218): ApiQuery->instantiateModules(array, string)
#7 /srv/mediawiki/php-1.34.0-wmf.21/includes/api/ApiMain.php(1596): ApiQuery->execute()
#8 /srv/mediawiki/php-1.34.0-wmf.21/includes/api/ApiMain.php(534): ApiMain->executeAction()
#9 /srv/mediawiki/php-1.34.0-wmf.21/includes/api/ApiMain.php(505): ApiMain->executeActionWithErrorHandling()
#10 /srv/mediawiki/php-1.34.0-wmf.21/api.php(86): ApiMain->execute()
#11 /srv/mediawiki/w/api.php(3): require(string)
#12 {main}

Both from

client/includes/WikibaseClient.php
1585     public function getItemTermIndex() {
1586         if ( $this->itemTermIndex === null ) {
1587             $dataAccessSettings = $this->getDataAccessSettings();
1588             $itemSource = $this->getItemSource( $dataAccessSettings );
1589             $itemDatabaseName = $dataAccessSettings->useEntitySourceBasedFederation() ?
1590                 $itemSource->getDatabaseName() :
1591                 $this->getRepositoryDefinitions()->getDatabaseNames()[''];   # array with empty string as a key

It is different than the one found earlier ( $this->getRepositoryDefinitions()->getConceptBaseUris()[''] ), but related since that is an empty string array key :\

So that is probably the exact same bug as T232613 which has the advantage to occur way more frequently.

hashar changed the task status from Open to Stalled.Thu, Sep 12, 9:56 PM

I have made production to generate a core dump, there is already one captured T232613#5489237 . Lets centralize on T232613 for now which is the most frequent occurrence.

The two issues in WikibaseClient when trying to access an empty string key should now have been solved. There was an issue in the memcached PHP module. The fix has been deployed on Sep 16th around noon. More details: T232613#5494695

hashar closed this task as Resolved.Tue, Sep 17, 8:30 AM
hashar claimed this task.

I have checked in logstash, the error is gone since we have deployed the fix for T232613#5494695 (had to upgrade php-memcached).

hashar removed hashar as the assignee of this task.Tue, Sep 17, 8:30 AM