Page MenuHomePhabricator
Create Task
Maniphest T231089

WikibaseClient.php: PHP Notice: Undefined index:
Closed, ResolvedPublicPRODUCTION ERROR
Actions

Description

Error

Request ID: XV-w0ApAMDgAACUbsB4AAADY
Request URL: ru.wikipedia.org/w/api.php?action=opensearch&search=...

message
[{exception_id}] {exception_url}   ErrorException from line 1107 of /srv/mediawiki/php-1.34.0-wmf.19/extensions/Wikibase/client/includes/WikibaseClient.php: PHP Notice: Undefined index:
trace
#0 /srv/mediawiki/php-1.34.0-wmf.19/extensions/Wikibase/client/includes/WikibaseClient.php(1107): MWExceptionHandler::handleError(integer, string, string, integer, array)
#1 /srv/mediawiki/php-1.34.0-wmf.19/extensions/Wikibase/client/includes/WikibaseClient.php(1357): Wikibase\Client\WikibaseClient->getRepoItemUriParser()
#2 /srv/mediawiki/php-1.34.0-wmf.19/extensions/Wikibase/client/includes/WikibaseClient.php(1344): Wikibase\Client\WikibaseClient->getDataAccessSnakFormatterFactory()
#3 /srv/mediawiki/php-1.34.0-wmf.19/extensions/Wikibase/client/includes/WikibaseClient.php(1368): Wikibase\Client\WikibaseClient->getStatementGroupRendererFactory()
#4 /srv/mediawiki/php-1.34.0-wmf.19/extensions/Wikibase/client/includes/DataAccess/ParserFunctions/Runner.php(174): Wikibase\Client\WikibaseClient->getPropertyParserFunctionRunner()
#5 /srv/mediawiki/php-1.34.0-wmf.19/extensions/Wikibase/client/includes/Hooks/ParserFunctionRegistrant.php(66): Wikibase\Client\DataAccess\ParserFunctions\Runner::renderEscapedPlainText(Parser, PPTemplateFrame_Hash, array)
#6 /srv/mediawiki/php-1.34.0-wmf.19/includes/parser/Parser.php(3624): Wikibase\Client\Hooks\ParserFunctionRegistrant->Wikibase\Client\Hooks\{closure}(Parser, PPTemplateFrame_Hash, array)
#7 /srv/mediawiki/php-1.34.0-wmf.19/extensions/Scribunto/includes/engines/LuaCommon/LuaCommon.php(780): Parser->callParserFunction(PPTemplateFrame_Hash, string, array)
#8 /srv/mediawiki/php-1.34.0-wmf.19/extensions/Scribunto/includes/engines/LuaSandbox/Engine.php(391): Scribunto_LuaEngine->callParserFunction(string, string, array)
#9 [internal function]: Scribunto_LuaSandboxCallback->__call(string, array)
#10 /srv/mediawiki/php-1.34.0-wmf.19/extensions/Scribunto/includes/engines/LuaSandbox/Engine.php(314): LuaSandboxFunction->call(LuaSandboxFunction)
#11 /srv/mediawiki/php-1.34.0-wmf.19/extensions/Scribunto/includes/engines/LuaCommon/LuaCommon.php(296): Scribunto_LuaSandboxInterpreter->callFunction(LuaSandboxFunction, LuaSandboxFunction)
#12 /srv/mediawiki/php-1.34.0-wmf.19/extensions/Scribunto/includes/engines/LuaCommon/LuaCommon.php(983): Scribunto_LuaEngine->executeFunctionChunk(LuaSandboxFunction, PPTemplateFrame_Hash)
#13 /srv/mediawiki/php-1.34.0-wmf.19/extensions/Scribunto/includes/common/Hooks.php(128): Scribunto_LuaModule->invoke(string, PPTemplateFrame_Hash)
#14 /srv/mediawiki/php-1.34.0-wmf.19/includes/parser/Parser.php(3624): ScribuntoHooks::invokeHook(Parser, PPTemplateFrame_Hash, array)
#15 /srv/mediawiki/php-1.34.0-wmf.19/includes/parser/Parser.php(3329): Parser->callParserFunction(PPTemplateFrame_Hash, string, array)
#16 /srv/mediawiki/php-1.34.0-wmf.19/includes/parser/PPFrame_Hash.php(254): Parser->braceSubstitution(array, PPTemplateFrame_Hash)
#17 /srv/mediawiki/php-1.34.0-wmf.19/includes/parser/Parser.php(3505): PPFrame_Hash->expand(PPNode_Hash_Tree)
#18 /srv/mediawiki/php-1.34.0-wmf.19/includes/parser/PPFrame_Hash.php(254): Parser->braceSubstitution(array, PPTemplateFrame_Hash)
#19 /srv/mediawiki/php-1.34.0-wmf.19/includes/parser/Parser.php(3246): PPFrame_Hash->expand(PPNode_Hash_Tree)
#20 /srv/mediawiki/php-1.34.0-wmf.19/includes/parser/PPFrame_Hash.php(254): Parser->braceSubstitution(array, PPTemplateFrame_Hash)
#21 /srv/mediawiki/php-1.34.0-wmf.19/includes/parser/PPTemplateFrame_Hash.php(167): PPFrame_Hash->expand(PPNode_Hash_Tree, integer)
#22 /srv/mediawiki/php-1.34.0-wmf.19/includes/parser/PPTemplateFrame_Hash.php(179): PPTemplateFrame_Hash->getNamedArgument(string)
#23 /srv/mediawiki/php-1.34.0-wmf.19/includes/parser/PPTemplateFrame_Hash.php(111): PPTemplateFrame_Hash->getArgument(string)
#24 /srv/mediawiki/php-1.34.0-wmf.19/extensions/Scribunto/includes/engines/LuaCommon/LuaCommon.php(693): PPTemplateFrame_Hash->getArguments()
#25 /srv/mediawiki/php-1.34.0-wmf.19/extensions/Scribunto/includes/engines/LuaSandbox/Engine.php(391): Scribunto_LuaEngine->getAllExpandedArguments(string)
#26 [internal function]: Scribunto_LuaSandboxCallback->__call(string, array)
#27 /srv/mediawiki/php-1.34.0-wmf.19/extensions/Scribunto/includes/engines/LuaSandbox/Engine.php(314): LuaSandboxFunction->call(LuaSandboxFunction)
#28 /srv/mediawiki/php-1.34.0-wmf.19/extensions/Scribunto/includes/engines/LuaCommon/LuaCommon.php(296): Scribunto_LuaSandboxInterpreter->callFunction(LuaSandboxFunction, LuaSandboxFunction)
#29 /srv/mediawiki/php-1.34.0-wmf.19/extensions/Scribunto/includes/engines/LuaCommon/LuaCommon.php(983): Scribunto_LuaEngine->executeFunctionChunk(LuaSandboxFunction, PPTemplateFrame_Hash)
#30 /srv/mediawiki/php-1.34.0-wmf.19/extensions/Scribunto/includes/common/Hooks.php(128): Scribunto_LuaModule->invoke(string, PPTemplateFrame_Hash)
#31 /srv/mediawiki/php-1.34.0-wmf.19/includes/parser/Parser.php(3624): ScribuntoHooks::invokeHook(Parser, PPTemplateFrame_Hash, array)
#32 /srv/mediawiki/php-1.34.0-wmf.19/includes/parser/Parser.php(3329): Parser->callParserFunction(PPTemplateFrame_Hash, string, array)
#33 /srv/mediawiki/php-1.34.0-wmf.19/includes/parser/PPFrame_Hash.php(254): Parser->braceSubstitution(array, PPTemplateFrame_Hash)
#34 /srv/mediawiki/php-1.34.0-wmf.19/includes/parser/Parser.php(3505): PPFrame_Hash->expand(PPNode_Hash_Tree)
#35 /srv/mediawiki/php-1.34.0-wmf.19/includes/parser/PPFrame_Hash.php(254): Parser->braceSubstitution(array, PPTemplateFrame_Hash)
#36 /srv/mediawiki/php-1.34.0-wmf.19/includes/parser/Parser.php(3505): PPFrame_Hash->expand(PPNode_Hash_Tree)
#37 /srv/mediawiki/php-1.34.0-wmf.19/includes/parser/PPFrame_Hash.php(254): Parser->braceSubstitution(array, PPFrame_Hash)
#38 /srv/mediawiki/php-1.34.0-wmf.19/includes/parser/Parser.php(3143): PPFrame_Hash->expand(PPNode_Hash_Tree, integer)
#39 /srv/mediawiki/php-1.34.0-wmf.19/includes/parser/Parser.php(1454): Parser->replaceVariables(string)
#40 /srv/mediawiki/php-1.34.0-wmf.19/includes/parser/Parser.php(576): Parser->internalParse(string)
#41 /srv/mediawiki/php-1.34.0-wmf.19/includes/content/WikitextContent.php(365): Parser->parse(string, Title, ParserOptions, boolean, boolean, NULL)
#42 /srv/mediawiki/php-1.34.0-wmf.19/includes/content/AbstractContent.php(555): WikitextContent->fillParserOutput(Title, NULL, ParserOptions, boolean, ParserOutput)
#43 /srv/mediawiki/php-1.34.0-wmf.19/includes/api/ApiParse.php(590): AbstractContent->getParserOutput(Title, NULL, ParserOptions)
#44 /srv/mediawiki/php-1.34.0-wmf.19/includes/api/ApiParse.php(164): ApiParse->getParsedContent(WikiPage, ParserOptions, boolean, NULL, NULL, boolean)
#45 /srv/mediawiki/php-1.34.0-wmf.19/includes/api/ApiMain.php(1589): ApiParse->execute()
#46 /srv/mediawiki/php-1.34.0-wmf.19/includes/api/ApiMain.php(502): ApiMain->executeAction()
#47 /srv/mediawiki/php-1.34.0-wmf.19/extensions/TextExtracts/includes/ApiQueryExtracts.php(235): ApiMain->execute()
#48 /srv/mediawiki/php-1.34.0-wmf.19/extensions/TextExtracts/includes/ApiQueryExtracts.php(159): TextExtracts\ApiQueryExtracts->parse(WikiPage)
#49 /srv/mediawiki/php-1.34.0-wmf.19/extensions/TextExtracts/includes/ApiQueryExtracts.php(96): TextExtracts\ApiQueryExtracts->getExtract(Title)
#50 /srv/mediawiki/php-1.34.0-wmf.19/includes/api/ApiQuery.php(250): TextExtracts\ApiQueryExtracts->execute()
#51 /srv/mediawiki/php-1.34.0-wmf.19/includes/api/ApiMain.php(1589): ApiQuery->execute()
#52 /srv/mediawiki/php-1.34.0-wmf.19/includes/api/ApiMain.php(502): ApiMain->executeAction()
#53 /srv/mediawiki/php-1.34.0-wmf.19/extensions/TextExtracts/includes/Hooks.php(37): ApiMain->execute()
#54 /srv/mediawiki/php-1.34.0-wmf.19/includes/Hooks.php(174): TextExtracts\Hooks::onApiOpenSearchSuggest(array)
#55 /srv/mediawiki/php-1.34.0-wmf.19/includes/Hooks.php(202): Hooks::callHook(string, array, array, NULL)
#56 /srv/mediawiki/php-1.34.0-wmf.19/includes/api/ApiOpenSearch.php(94): Hooks::run(string, array)
#57 /srv/mediawiki/php-1.34.0-wmf.19/includes/api/ApiMain.php(1589): ApiOpenSearch->execute()
#58 /srv/mediawiki/php-1.34.0-wmf.19/includes/api/ApiMain.php(533): ApiMain->executeAction()
#59 /srv/mediawiki/php-1.34.0-wmf.19/includes/api/ApiMain.php(504): ApiMain->executeActionWithErrorHandling()
#60 /srv/mediawiki/php-1.34.0-wmf.19/api.php(86): ApiMain->execute()
#61 /srv/mediawiki/w/api.php(3): require(string)
#62 {main}
Impact

1 hit in the last hour, 9 in the last 24 hours, 60 in the last 7 days.

Notes

Not blocking the train since it's not new.

Related Objects
Search...

StatusSubtypeAssignedTask
 ResolvedPRODUCTION ERRORNoneT231089 WikibaseClient.php: PHP Notice: Undefined index:
 ResolvedPRODUCTION ERRORNoneT232613 LBFactoryMulti.php: PHP Notice: Undefined index:

Event Timeline

zeljkofilipin created this task.Aug 23 2019, 3:03 PM
Restricted Application added a subscriber: Aklapper. · View Herald TranscriptAug 23 2019, 3:03 PM
Lucas_Werkmeister_WMDE added a project: Wikidata-Campsite (Wikidata-Campsite-Iteration-∞ (On Hold)).Aug 27 2019, 9:01 AM
Lucas_Werkmeister_WMDE added subscribers: WMDE-leszek, Lucas_Werkmeister_WMDE.

The offending line:

private function getRepoItemUriParser() {
    // B/C compatibility, should be removed soon                                                                                                                         
    // TODO: Move to check repo that has item entity not the default repo                                                                                                
    return new SuffixEntityIdParser(
        $this->getRepositoryDefinitions()->getConceptBaseUris()[''], // <-------
        new ItemIdParser()
    );
}
Ladsgroup subscribed.Aug 27 2019, 8:25 PM

That's weird:

ladsgroup@mwmaint1002:~$ mwscript eval.php --wiki=ruwiki
> $client = \Wikibase\Client\WikibaseClient::getDefaultInstance();

> var_dump( $client->getRepositoryDefinitions()->getConceptBaseUris() );
array(1) {
  [""]=>
  string(31) "http://www.wikidata.org/entity/"
}

It exists...

Ladsgroup added a comment.Aug 28 2019, 11:52 AM

My thought is that it can be either a race condition that repositoryDefinitions doesn't get properly initialized or we have another case of opcache corruption. Is it happening on php7 or hhvm?

Ladsgroup added a project: PHP 7.2 support.Aug 28 2019, 12:09 PM

Yup. The XV-w0ApAMDgAACUbsB4AAADY logstash says: 7.2.16-1+0~20190307202415.17+stretch~1.gbpa7be82+wmf1

All errors are php7: https://logstash.wikimedia.org/goto/b64aca4c135996e93f97b70a2079d7d8

image.png (922×211 px, 52 KB)

Ladsgroup added a comment.Aug 28 2019, 12:10 PM

Probably we can just merge this to T224491: PHP 7 corruption during deployment (was: PHP 7 fatals on mw1262)

Ladsgroup moved this task from To Do (prioritised from top to bottom) to Stalled/Waiting on the Wikidata-Campsite (Wikidata-Campsite-Iteration-∞ (On Hold)) board.Aug 28 2019, 12:11 PM
Krinkle moved this task from Untriaged to Aug2019/1.34.wmf.17+ on the Wikimedia-production-error board.Aug 28 2019, 6:09 PM
• mmodell changed the subtype of this task from "Task" to "Production Error".Aug 28 2019, 11:05 PM
Krinkle subscribed.Sep 6 2019, 12:23 AM

Smells like T229433. Which is also about '' array index, and PHP 7.2. It's obviously a bug in PHP 7.2, but I've not been able to find evidence that it is due to opcache/T224491.

Krinkle moved this task from Untriaged to Wikimedia production on the PHP 7.2 support board.Sep 6 2019, 12:23 AM
Krinkle mentioned this in T232613: LBFactoryMulti.php: PHP Notice: Undefined index: .Sep 11 2019, 7:31 PM
Joe subscribed.Sep 12 2019, 8:59 AM
In T231089#5445485, @Ladsgroup wrote:

Probably we can just merge this to T224491: PHP 7 corruption during deployment (was: PHP 7 fatals on mw1262)

no please do not. This bug has nothing to do with opcache corruption during deployments. The problem happens only episodically, not for all requests that happen to hit that code path, so I doubt that's the cause.

Joe added a comment.Sep 12 2019, 9:00 AM
In T231089#5470160, @Krinkle wrote:

Smells like T229433. Which is also about '' array index, and PHP 7.2. It's obviously a bug in PHP 7.2, but I've not been able to find evidence that it is due to opcache/T224491.

Yes, it looks like it's a weird bug with arrays with an empty string as key. But it only seems to happen in specific conditions, and not consistently. Correct?

Ladsgroup added a comment.Sep 12 2019, 10:13 AM
In T231089#5486893, @Joe wrote:
In T231089#5445485, @Ladsgroup wrote:

Probably we can just merge this to T224491: PHP 7 corruption during deployment (was: PHP 7 fatals on mw1262)

no please do not. This bug has nothing to do with opcache corruption during deployments. The problem happens only episodically, not for all requests that happen to hit that code path, so I doubt that's the cause.

you're right. Sorry. I meant this is a bug in php7 and there's nothing wikidata devs can do to help :(

hashar subscribed.Sep 12 2019, 2:45 PM

I have found some other occurrences from WikibaseClient.php, still related to an array keys being an empty string. Both from php 7.2.22-1+0~20190902.26+debian9~1.gbpd64eb7+wmf1

/srv/mediawiki/php-1.34.0-wmf.21/extensions/Wikibase/client/includes/WikibaseClient.php:1591
PHP Notice: Undefined index: 

#0 /srv/mediawiki/php-1.34.0-wmf.21/extensions/Wikibase/client/includes/WikibaseClient.php(1591): MWExceptionHandler::handleError(integer, string, string, integer, array)
#1 /srv/mediawiki/php-1.34.0-wmf.21/extensions/Wikibase/client/includes/WikibaseClient.php(1627): Wikibase\Client\WikibaseClient->getItemTermIndex()
#2 /srv/mediawiki/php-1.34.0-wmf.21/extensions/Wikibase/client/WikibaseClient.php(236): Wikibase\Client\WikibaseClient->getDescriptionLookup()
#3 /srv/mediawiki/php-1.34.0-wmf.21/includes/api/ApiModuleManager.php(191): {closure}(ApiQuery, string)
#4 /srv/mediawiki/php-1.34.0-wmf.21/includes/api/ApiModuleManager.php(167): ApiModuleManager->instantiateModule(string, string, Closure)
#5 /srv/mediawiki/php-1.34.0-wmf.21/includes/api/ApiQuery.php(307): ApiModuleManager->getModule(string, string)
#6 /srv/mediawiki/php-1.34.0-wmf.21/includes/api/ApiQuery.php(218): ApiQuery->instantiateModules(array, string)
#7 /srv/mediawiki/php-1.34.0-wmf.21/includes/api/ApiMain.php(1596): ApiQuery->execute()
#8 /srv/mediawiki/php-1.34.0-wmf.21/includes/api/ApiMain.php(534): ApiMain->executeAction()
#9 /srv/mediawiki/php-1.34.0-wmf.21/includes/api/ApiMain.php(505): ApiMain->executeActionWithErrorHandling()
#10 /srv/mediawiki/php-1.34.0-wmf.21/api.php(86): ApiMain->execute()
#11 /srv/mediawiki/w/api.php(3): require(string)
#12 {main}

And:

#0 /srv/mediawiki/php-1.34.0-wmf.21/extensions/Wikibase/client/includes/WikibaseClient.php(1591): MWExceptionHandler::handleError(integer, string, string, integer, array)
#1 /srv/mediawiki/php-1.34.0-wmf.21/extensions/Wikibase/client/includes/WikibaseClient.php(1627): Wikibase\Client\WikibaseClient->getItemTermIndex()
#2 /srv/mediawiki/php-1.34.0-wmf.21/extensions/Wikibase/client/WikibaseClient.php(236): Wikibase\Client\WikibaseClient->getDescriptionLookup()
#3 /srv/mediawiki/php-1.34.0-wmf.21/includes/api/ApiModuleManager.php(191): {closure}(ApiQuery, string)
#4 /srv/mediawiki/php-1.34.0-wmf.21/includes/api/ApiModuleManager.php(167): ApiModuleManager->instantiateModule(string, string, Closure)
#5 /srv/mediawiki/php-1.34.0-wmf.21/includes/api/ApiQuery.php(307): ApiModuleManager->getModule(string, string)
#6 /srv/mediawiki/php-1.34.0-wmf.21/includes/api/ApiQuery.php(218): ApiQuery->instantiateModules(array, string)
#7 /srv/mediawiki/php-1.34.0-wmf.21/includes/api/ApiMain.php(1596): ApiQuery->execute()
#8 /srv/mediawiki/php-1.34.0-wmf.21/includes/api/ApiMain.php(534): ApiMain->executeAction()
#9 /srv/mediawiki/php-1.34.0-wmf.21/includes/api/ApiMain.php(505): ApiMain->executeActionWithErrorHandling()
#10 /srv/mediawiki/php-1.34.0-wmf.21/api.php(86): ApiMain->execute()
#11 /srv/mediawiki/w/api.php(3): require(string)
#12 {main}

Both from

client/includes/WikibaseClient.php
1585     public function getItemTermIndex() {
1586         if ( $this->itemTermIndex === null ) {
1587             $dataAccessSettings = $this->getDataAccessSettings();
1588             $itemSource = $this->getItemSource( $dataAccessSettings );
1589             $itemDatabaseName = $dataAccessSettings->useEntitySourceBasedFederation() ?
1590                 $itemSource->getDatabaseName() :
1591                 $this->getRepositoryDefinitions()->getDatabaseNames()[''];   # array with empty string as a key

It is different than the one found earlier ( $this->getRepositoryDefinitions()->getConceptBaseUris()[''] ), but related since that is an empty string array key :\

So that is probably the exact same bug as T232613 which has the advantage to occur way more frequently.

hashar changed the task status from Open to Stalled.Sep 12 2019, 9:56 PM

I have made production to generate a core dump, there is already one captured T232613#5489237 . Lets centralize on T232613 for now which is the most frequent occurrence.

hashar added a subtask: T232613: LBFactoryMulti.php: PHP Notice: Undefined index: .Sep 12 2019, 9:57 PM
hashar closed subtask T232613: LBFactoryMulti.php: PHP Notice: Undefined index: as Resolved.Sep 16 2019, 12:51 PM
hashar added a comment.Sep 16 2019, 12:54 PM

The two issues in WikibaseClient when trying to access an empty string key should now have been solved. There was an issue in the memcached PHP module. The fix has been deployed on Sep 16th around noon. More details: T232613#5494695

hashar closed this task as Resolved.Sep 17 2019, 8:30 AM
hashar claimed this task.

I have checked in logstash, the error is gone since we have deployed the fix for T232613#5494695 (had to upgrade php-memcached).

hashar removed hashar as the assignee of this task.Sep 17 2019, 8:30 AM
Maintenance_bot moved this task from incoming to in progress on the Wikidata board.Oct 27 2019, 11:10 PM
Content licensed under Creative Commons Attribution-ShareAlike (CC BY-SA) 4.0 unless otherwise noted; code licensed under GNU General Public License (GPL) 2.0 or later and other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL