Page MenuHomePhabricator

PHP7 corruption reports in 2020 (Call on wrong object, etc.)
Open, MediumPublic

Description

NOTE: See also tracking task for corruptions in 2019 was T224491.

Diagnosing opcache corruption

Please see https://wikitech.wikimedia.org/wiki/User:CDanis/Diagnosing_opcache_corruption

An old error report for the case that began this tracking task
Error: Call to undefined method Monolog\Logger::get()

from /srv/mediawiki/php-1.35.0-wmf.19/includes/libs/objectcache/wancache/WANObjectCache.php:1303
trace
#0 /srv/mediawiki/php-1.35.0-wmf.19/extensions/CentralAuth/includes/CentralAuthUser.php(558): WANObjectCache->getWithSetCallback(string, integer, Closure, array)
#1 /srv/mediawiki/php-1.35.0-wmf.19/extensions/CentralAuth/includes/CentralAuthUser.php(414): CentralAuthUser->loadFromCache()
#2 /srv/mediawiki/php-1.35.0-wmf.19/extensions/CentralAuth/includes/CentralAuthUser.php(589): CentralAuthUser->loadState()
#3 /srv/mediawiki/php-1.35.0-wmf.19/extensions/CentralAuth/includes/CentralAuthUser.php(686): CentralAuthUser->getId()
#4 /srv/mediawiki/php-1.35.0-wmf.19/extensions/CentralAuth/includes/CentralAuthHooks.php(917): CentralAuthUser->exists()
#5 /srv/mediawiki/php-1.35.0-wmf.19/includes/Hooks.php(174): CentralAuthHooks::onGetUserBlock(User, string, NULL)
#6 /srv/mediawiki/php-1.35.0-wmf.19/includes/Hooks.php(202): Hooks::callHook(string, array, array, NULL)
#7 /srv/mediawiki/php-1.35.0-wmf.19/includes/block/BlockManager.php(167): Hooks::run(string, array)
#8 /srv/mediawiki/php-1.35.0-wmf.19/includes/user/User.php(1800): MediaWiki\Block\BlockManager->getUserBlock(User, WebRequest, boolean)
#9 /srv/mediawiki/php-1.35.0-wmf.19/includes/user/User.php(2119): User->getBlockedStatus(boolean)
#10 /srv/mediawiki/php-1.35.0-wmf.19/includes/block/BlockManager.php(470): User->getBlock()
#11 /srv/mediawiki/php-1.35.0-wmf.19/includes/MediaWiki.php(755): MediaWiki\Block\BlockManager->trackBlockWithCookie(User, WebResponse)
#12 /srv/mediawiki/php-1.35.0-wmf.19/includes/api/ApiMain.php(548): MediaWiki::preOutputCommit(DerivativeContext)
#13 /srv/mediawiki/php-1.35.0-wmf.19/includes/api/ApiMain.php(493): ApiMain->executeActionWithErrorHandling()
#14 /srv/mediawiki/php-1.35.0-wmf.19/api.php(84): ApiMain->execute()
#15 /srv/mediawiki/w/api.php(3): require(string)
#16 {main}
  • Request ID: XkWMYwpAIDkAALfuQmcAAACH
  • Host: mw1345
  • Request Method: GET
  • Request URL: commons.wikimedia.org/w/api.php?action=query&generator=allimages&prop=imageinfo&…
  • phpversion: 7.2.26-1+0~20191218.33+debian9~1.gbpb5a340+wmf1
  • timestamp: 2020-02-13T17:51:28+00:00

Logstash record:
https://logstash.wikimedia.org/app/kibana#/doc/logstash-*/logstash-deploy-2020.02.13/mediawiki?id=AXA_rVssh3Uj6x1zToLY

Source code

CentralAuthUser.php@1.35.0-wmf.19

		$cache = MediaWikiServices::getInstance()->getMainWANObjectCache();
		$data = $cache->getWithSetCallback( # line 540
			$this->getCacheKey( $cache ),
			$cache::TTL_DAY,
			function ( $oldValue, &$ttl, array &$setOpts ) {
				$dbr = CentralAuthUtils::getCentralReplicaDB();
				$setOpts += Database::getCacheSetOptions( $dbr );

				$this->loadFromDatabase();
				$this->loadAttached();
				$this->loadGroups();

				$data = [];
				foreach ( self::$mCacheVars as $var ) {
					$data[$var] = $this->$var;
				}

				return $data;
			},
			[ 'pcTTL' => $cache::TTL_PROC_LONG, 'version' => self::VERSION ] # line 558
		); # line 559

WANObjectCache.php@1.35.0-wmf.19

	final public function getWithSetCallback(
		$key, $ttl, $callback, array $opts = [], array $cbParams = []
	) {
		$version = $opts['version'] ?? null;
		$pcTTL = $opts['pcTTL'] ?? self::TTL_UNCACHEABLE;
		$pCache = ( $pcTTL >= 0 )
			? $this->getProcessCache( $opts['pcGroup'] ?? self::PC_PRIMARY )  # line 1296
			: null;

		if ( $pCache && $this->callbackDepth == 0 ) {
			$cached = $pCache->get( $this->getProcessCacheKey( $key, $version ), $pcTTL, false );   # line 1303
			if ( $cached !== false ) {
				$this->logger->debug( "getWithSetCallback($key): process cache hit" );  # line 1305
				return $cached;
			}
		}
Notes

Stuff is very messed up here:

  1. The error Call to undefined method Monolog\Logger::get() is attribued to line WANObjectCache.php:L1303, which is calling $pCache->get() where $pCache is an instance of MapCacheLRU. It is meant to be calling MapCacheLRU->get().
  2. For some reason the engine is thinking about Monolog\Logger, which is strange. For what it's worth, there is an instance of that object "near by" – two lines down on line 1305. Could it be confusing the two??
  3. The stack trace says WANObjectCache->getWithSetCallback was called from CentralAuthUser.php:L558. This is also odd, because that line is neither the start nor the end of the function call. See snippet above. Looks like this may be off by a few as well.

Might be opcache corruption.

Related Objects

Event Timeline

Krinkle created this task.Feb 13 2020, 6:29 PM
Restricted Application added a subscriber: Aklapper. · View Herald TranscriptFeb 13 2020, 6:29 PM
jijiki triaged this task as Medium priority.Feb 13 2020, 6:30 PM
jijiki added a project: serviceops.
jijiki added a subscriber: jijiki.
Krinkle renamed this task from Opcache corruption: Object and method magically missing to PHP7 corruption: Method call proxied to unrelated object (also: Call to undefined method).Feb 13 2020, 6:39 PM
Krinkle renamed this task from PHP7 corruption: Method call proxied to unrelated object (also: Call to undefined method) to PHP7 corruption: Method call executed on unrelated object (also: Call to undefined method).
Krinkle updated the task description. (Show Details)Feb 13 2020, 6:55 PM
Krinkle added a comment.EditedFeb 13 2020, 7:05 PM

Here is another mysterious mis-call which I can't explain:

Logstash single document

  • Host: mw1276
  • timestamp: 2020-02-13T04:10:15+00:00
Call to undefined method Wikibase\Repo\Hooks\Formatters\EntityLinkFormatterFactory::getSerialization()
 from /srv/mediawiki/php-1.35.0-wmf.19/vendor/wikibase/data-model-serialization/src/Serializers/StatementListSerializer.php:72
#0 /srv/mediawiki/php-1.35.0-wmf.19/vendor/wikibase/data-model-serialization/src/Serializers/StatementListSerializer.php(65): Wikibase\DataModel\Serializers\StatementListSerializer->getSerialized(Wikibase\DataModel\Statement\StatementList)
#1 /srv/mediawiki/php-1.35.0-wmf.19/vendor/wikibase/data-model-serialization/src/Serializers/ItemSerializer.php(127): Wikibase\DataModel\Serializers\StatementListSerializer->serialize(Wikibase\DataModel\Statement\StatementList)
#2 /srv/mediawiki/php-1.35.0-wmf.19/vendor/wikibase/data-model-serialization/src/Serializers/ItemSerializer.php(104): Wikibase\DataModel\Serializers\ItemSerializer->addStatementListToSerialization(Wikibase\DataModel\Entity\Item, array)
#3 /srv/mediawiki/php-1.35.0-wmf.19/vendor/wikibase/data-model-serialization/src/Serializers/ItemSerializer.php(94): Wikibase\DataModel\Serializers\ItemSerializer->getSerialized(Wikibase\DataModel\Entity\Item)
#4 /srv/mediawiki/php-1.35.0-wmf.19/vendor/serialization/serialization/src/Serializers/DispatchingSerializer.php(42): Wikibase\DataModel\Serializers\ItemSerializer->serialize(Wikibase\DataModel\Entity\Item)
#5 /srv/mediawiki/php-1.35.0-wmf.19/extensions/Wikibase/client/includes/Serializer/ClientEntitySerializer.php(60): Serializers\DispatchingSerializer->serialize(Wikibase\DataModel\Entity\Item)
#6 /srv/mediawiki/php-1.35.0-wmf.19/extensions/Wikibase/client/includes/DataAccess/Scribunto/EntityAccessor.php(163): Wikibase\Client\Serializer\ClientEntitySerializer->serialize(Wikibase\DataModel\Entity\Item)
#7 /srv/mediawiki/php-1.35.0-wmf.19/extensions/Wikibase/client/includes/DataAccess/Scribunto/Scribunto_LuaWikibaseLibrary.php(414): Wikibase\Client\DataAccess\Scribunto\EntityAccessor->getEntity(string)
#8 /srv/mediawiki/php-1.35.0-wmf.19/extensions/Scribunto/includes/engines/LuaSandbox/LuaSandboxCallback.php(26): Wikibase\Client\DataAccess\Scribunto\Scribunto_LuaWikibaseLibrary->getEntity(string)
#9 [internal function]: Scribunto_LuaSandboxCallback->__call(string, array)
#10 /srv/mediawiki/php-1.35.0-wmf.19/extensions/Scribunto/includes/engines/LuaSandbox/LuaSandboxInterpreter.php(113): LuaSandboxFunction->call(LuaSandboxFunction)
#11 /srv/mediawiki/php-1.35.0-wmf.19/extensions/Scribunto/includes/engines/LuaCommon/LuaEngine.php(291): Scribunto_LuaSandboxInterpreter->callFunction(LuaSandboxFunction, LuaSandboxFunction)
#12 /srv/mediawiki/php-1.35.0-wmf.19/extensions/Scribunto/includes/engines/LuaCommon/LuaModule.php(69): Scribunto_LuaEngine->executeFunctionChunk(LuaSandboxFunction, PPTemplateFrame_Hash)
#13 /srv/mediawiki/php-1.35.0-wmf.19/extensions/Scribunto/includes/common/Hooks.php(128): Scribunto_LuaModule->invoke(string, PPTemplateFrame_Hash)
#14 /srv/mediawiki/php-1.35.0-wmf.19/includes/parser/Parser.php(3599): ScribuntoHooks::invokeHook(Parser, PPTemplateFrame_Hash, array)
#15 /srv/mediawiki/php-1.35.0-wmf.19/includes/parser/Parser.php(3303): Parser->callParserFunction(PPTemplateFrame_Hash, string, array)
#16 /srv/mediawiki/php-1.35.0-wmf.19/includes/parser/PPFrame_Hash.php(253): Parser->braceSubstitution(array, PPTemplateFrame_Hash)
#17 /srv/mediawiki/php-1.35.0-wmf.19/includes/parser/Parser.php(3481): PPFrame_Hash->expand(PPNode_Hash_Tree)
#18 /srv/mediawiki/php-1.35.0-wmf.19/includes/parser/PPFrame_Hash.php(253): Parser->braceSubstitution(array, PPFrame_Hash)
#19 /srv/mediawiki/php-1.35.0-wmf.19/includes/parser/PPTemplateFrame_Hash.php(169): PPFrame_Hash->expand(PPNode_Hash_Tree, integer)
#20 /srv/mediawiki/php-1.35.0-wmf.19/includes/parser/PPTemplateFrame_Hash.php(181): PPTemplateFrame_Hash->getNamedArgument(string)
#21 /srv/mediawiki/php-1.35.0-wmf.19/includes/parser/PPTemplateFrame_Hash.php(113): PPTemplateFrame_Hash->getArgument(string)
#22 /srv/mediawiki/php-1.35.0-wmf.19/extensions/Scribunto/includes/engines/LuaCommon/LuaEngine.php(691): PPTemplateFrame_Hash->getArguments()
#23 /srv/mediawiki/php-1.35.0-wmf.19/extensions/Scribunto/includes/engines/LuaSandbox/LuaSandboxCallback.php(26): Scribunto_LuaEngine->getAllExpandedArguments(string)
#24 [internal function]: Scribunto_LuaSandboxCallback->__call(string, array)
#25 /srv/mediawiki/php-1.35.0-wmf.19/extensions/Scribunto/includes/engines/LuaSandbox/LuaSandboxInterpreter.php(113): LuaSandboxFunction->call(LuaSandboxFunction)
#26 /srv/mediawiki/php-1.35.0-wmf.19/extensions/Scribunto/includes/engines/LuaCommon/LuaEngine.php(291): Scribunto_LuaSandboxInterpreter->callFunction(LuaSandboxFunction, LuaSandboxFunction)
#27 /srv/mediawiki/php-1.35.0-wmf.19/extensions/Scribunto/includes/engines/LuaCommon/LuaModule.php(69): Scribunto_LuaEngine->executeFunctionChunk(LuaSandboxFunction, PPTemplateFrame_Hash)
#28 /srv/mediawiki/php-1.35.0-wmf.19/extensions/Scribunto/includes/common/Hooks.php(128): Scribunto_LuaModule->invoke(string, PPTemplateFrame_Hash)
#29 /srv/mediawiki/php-1.35.0-wmf.19/includes/parser/Parser.php(3599): ScribuntoHooks::invokeHook(Parser, PPTemplateFrame_Hash, array)
#30 /srv/mediawiki/php-1.35.0-wmf.19/includes/parser/Parser.php(3303): Parser->callParserFunction(PPTemplateFrame_Hash, string, array)
#31 /srv/mediawiki/php-1.35.0-wmf.19/includes/parser/PPFrame_Hash.php(253): Parser->braceSubstitution(array, PPTemplateFrame_Hash)
#32 /srv/mediawiki/php-1.35.0-wmf.19/includes/parser/Parser.php(3481): PPFrame_Hash->expand(PPNode_Hash_Tree)
#33 /srv/mediawiki/php-1.35.0-wmf.19/includes/parser/PPFrame_Hash.php(253): Parser->braceSubstitution(array, PPFrame_Hash)
#34 /srv/mediawiki/php-1.35.0-wmf.19/includes/parser/Parser.php(3144): PPFrame_Hash->expand(PPNode_Hash_Tree, integer)
#35 /srv/mediawiki/php-1.35.0-wmf.19/includes/parser/Parser.php(1482): Parser->replaceVariables(string)
#36 /srv/mediawiki/php-1.35.0-wmf.19/includes/parser/Parser.php(598): Parser->internalParse(string)
#37 /srv/mediawiki/php-1.35.0-wmf.19/includes/content/WikitextContent.php(368): Parser->parse(string, Title, ParserOptions, boolean, boolean, integer)
#38 /srv/mediawiki/php-1.35.0-wmf.19/includes/content/AbstractContent.php(565): WikitextContent->fillParserOutput(Title, integer, ParserOptions, boolean, ParserOutput)
#39 /srv/mediawiki/php-1.35.0-wmf.19/includes/Revision/RenderedRevision.php(267): AbstractContent->getParserOutput(Title, integer, ParserOptions, boolean)
#40 /srv/mediawiki/php-1.35.0-wmf.19/includes/Revision/RenderedRevision.php(236): MediaWiki\Revision\RenderedRevision->getSlotParserOutputUncached(WikitextContent, boolean)
#41 /srv/mediawiki/php-1.35.0-wmf.19/includes/Revision/RevisionRenderer.php(215): MediaWiki\Revision\RenderedRevision->getSlotParserOutput(string)
#42 /srv/mediawiki/php-1.35.0-wmf.19/includes/Revision/RevisionRenderer.php(152): MediaWiki\Revision\RevisionRenderer->combineSlotOutput(MediaWiki\Revision\RenderedRevision, array)
#43 [internal function]: MediaWiki\Revision\RevisionRenderer->MediaWiki\Revision\{closure}(MediaWiki\Revision\RenderedRevision, array)
#44 /srv/mediawiki/php-1.35.0-wmf.19/includes/Revision/RenderedRevision.php(198): call_user_func(Closure, MediaWiki\Revision\RenderedRevision, array)
#45 /srv/mediawiki/php-1.35.0-wmf.19/includes/filerepo/file/LocalFile.php(2097): MediaWiki\Revision\RenderedRevision->getRevisionParserOutput()
#46 /srv/mediawiki/php-1.35.0-wmf.19/extensions/CommonsMetadata/src/DataCollector.php(290): LocalFile->getDescriptionText(LanguageEn)
#47 /srv/mediawiki/php-1.35.0-wmf.19/extensions/CommonsMetadata/src/DataCollector.php(98): CommonsMetadata\DataCollector->getDescriptionText(LocalFile, LanguageEn)
#48 /srv/mediawiki/php-1.35.0-wmf.19/extensions/CommonsMetadata/src/HookHandler.php(70): CommonsMetadata\DataCollector->collect(array, LocalFile)
#49 /srv/mediawiki/php-1.35.0-wmf.19/includes/Hooks.php(174): CommonsMetadata\HookHandler::onGetExtendedMetadata(array, LocalFile, RequestContext, boolean, integer)
#50 /srv/mediawiki/php-1.35.0-wmf.19/includes/Hooks.php(202): Hooks::callHook(string, array, array, NULL)
#51 /srv/mediawiki/php-1.35.0-wmf.19/includes/media/FormatMetadata.php(1707): Hooks::run(string, array)
#52 /srv/mediawiki/php-1.35.0-wmf.19/includes/media/FormatMetadata.php(1624): FormatMetadata->getExtendedMetadataFromHook(LocalFile, array, integer)
#53 /srv/mediawiki/php-1.35.0-wmf.19/includes/api/ApiQueryImageInfo.php(575): FormatMetadata->fetchExtendedMetadata(LocalFile)
#54 /srv/mediawiki/php-1.35.0-wmf.19/includes/api/ApiQueryImageInfo.php(183): ApiQueryImageInfo::getInfo(LocalFile, array, ApiResult, NULL, array)
#55 /srv/mediawiki/php-1.35.0-wmf.19/includes/api/ApiQuery.php(255): ApiQueryImageInfo->execute()
#56 /srv/mediawiki/php-1.35.0-wmf.19/includes/api/ApiMain.php(1586): ApiQuery->execute()
#57 /srv/mediawiki/php-1.35.0-wmf.19/includes/api/ApiMain.php(522): ApiMain->executeAction()
#58 /srv/mediawiki/php-1.35.0-wmf.19/includes/api/ApiMain.php(493): ApiMain->executeActionWithErrorHandling()
#59 /srv/mediawiki/php-1.35.0-wmf.19/api.php(84): ApiMain->execute()
#60 /srv/mediawiki/w/api.php(3): require(string)
#61 {main}

Source code: StatementListSerializer.php

	private function getSerialized( StatementList $statementList ) {
		// …

		foreach ( $statementList->toArray() as $statement ) {
			$statement->getPropertyId()->getSerialization(); # line 72

However the class it says it has an instance of (EntityLinkFormatterFactory) doesn't seem to be related to this.

Source code: EntityLinkFormatterFactory.php

namespace Wikibase\Repo\Hooks\Formatters;

/* … */
class EntityLinkFormatterFactory {

Definitely not what getPropertyId() would return an instance of.

I looked at the code base and couldn't find any instance that EntityLinkFormatterFactory object would be set as property id of an statement. Unless some eval magic is involved and we would have bad lua in clients invoking this but I highly highly doubt that.

Krinkle added a comment.EditedMar 17 2020, 1:47 AM

Here is another mysterious mis-call (Logstash single document)

  • timestamp: 2020-02-14T10:04:16
  • reqId: XkZwYwpAICkAAAmWLyEAAAAL
  • host: mw1320
PHP Fatal Error: Uncaught Error: Call to a member function isHandling() on integer

in /srv/mediawiki/php-1.35.0-wmf.19/vendor/monolog/monolog/src/Monolog/Logger.php:304
#0 /srv/mediawiki/php-1.35.0-wmf.19/vendor/monolog/monolog/src/Monolog/Logger.php(628): Monolog\Logger->addRecord(100, 'getWithSetCallb...', Array)
#1 /srv/mediawiki/php-1.35.0-wmf.19/includes/libs/objectcache/wancache/WANObjectCache.php(1305): Monolog\Logger->debug('getWithSetCallb...')

The code in question:

WANObjectCache
setLogger( LoggerInterface $logger ) {
	$this->logger = $logger;
}
getWithSetCallback(  ) {
	
	$this->logger->debug( "getWithSetCallback($key): process cache hit" );
Monolog/Logger
public function debug($message, array $context = array()) {
  return $this->addRecord(static::DEBUG, $message, $context);
}
public function addRecord(int $level, string $message, array $context = array()) {
   $this->handlers = [  ]; // @var HandlerInterface[]
    

    $levelName = static::getLevelName($level); // @var string
    reset($this->handlers);
    while ($handler = current($this->handlers)) {
        if ($handler->isHandling(array('level' => $level))) { # BOOM

In this case it looks like PHP has somehow confused $handler (HandlerInterface) with $level (integer).

Here is another mysterious mis-call (Logstash single document)

  • timestamp: 2020-02-14T10:04:16
  • reqId: XkZwYwpAICkAAAmWLyEAAAAL
  • host: mw1320
PHP Fatal Error: Uncaught Error: Call to a member function isHandling() on integer

in /srv/mediawiki/php-1.35.0-wmf.19/vendor/monolog/monolog/src/Monolog/Logger.php:304
#0 /srv/mediawiki/php-1.35.0-wmf.19/vendor/monolog/monolog/src/Monolog/Logger.php(628): Monolog\Logger->addRecord(100, 'getWithSetCallb...', Array)
#1 /srv/mediawiki/php-1.35.0-wmf.19/includes/libs/objectcache/wancache/WANObjectCache.php(1305): Monolog\Logger->debug('getWithSetCallb...')

It does not appear to align with an automatic restart (the next one was several days later), and also does not align with a near-zero amount of opcache memory left. Odd?

Error message
Fatal error:
Cannot declare class Wikimedia\MWConfig\XWikimediaDebug, because the name is already in use
Notes

A spike of over 7000 fatals in the span of one hour, on the mw1379 server.

It lasted approximately from 2020-06-01 14:00 to 14:50 UTC.

From Logstash mediawiki-errors:

Impact

All wiki web requests on this server were presumably aborted with a system error.

Copying here so that this task remains complete for analysis of the problem, separate from T253673 which is about fixing the root cause. (Controlling the noise)

Today at 2020-07-08T14:00:12, with no deployments happening, mw1346 started generating exceptions at high rate:

/srv/mediawiki/php-1.35.0-wmf.39/includes/config/GlobalVarConfig.php:53 GlobalVarConfig::get: undefined option: 'MinervaOverflowInPageActhons'

#0 /srv/mediawiki/php-1.35.0-wmf.39/skins/MinervaNeue/includes/MinervaHooks.php(131): GlobalVarConfig->get(string)
#1 /srv/mediawiki/php-1.35.0-wmf.39/includes/HookContainer/HookContainer.php(320): MinervaHooks::onMobileFrontendFeaturesRegistration(MobileFrontend\Features\FeaturesManager)
#2 /srv/mediawiki/php-1.35.0-wmf.39/includes/HookContainer/HookContainer.php(131): MediaWiki\HookContainer\HookContainer->callLegacyHook(string, array, array, array)
#3 /srv/mediawiki/php-1.35.0-wmf.39/extensions/MobileFrontend/includes/Features/FeaturesManager.php(43): MediaWiki\HookContainer\HookContainer->run(string, array)
#4 /srv/mediawiki/php-1.35.0-wmf.39/extensions/MobileFrontend/includes/ServiceWiring.php(55): MobileFrontend\Features\FeaturesManager->useHookToRegisterExtensionOrSkinFeatures()
#5 /srv/mediawiki/php-1.35.0-wmf.39/includes/libs/services/ServiceContainer.php(451): Wikimedia\Services\ServiceContainer->{closure}(MediaWiki\MediaWikiServices)
#6 /srv/mediawiki/php-1.35.0-wmf.39/includes/libs/services/ServiceContainer.php(419): Wikimedia\Services\ServiceContainer->createService(string)
#7 /srv/mediawiki/php-1.35.0-wmf.39/extensions/MobileFrontend/includes/MobileFrontendHooks.php(1115): Wikimedia\Services\ServiceContainer->getService(string)
#8 /srv/mediawiki/php-1.35.0-wmf.39/includes/HookContainer/HookContainer.php(320): MobileFrontendHooks::onMakeGlobalVariablesScript(array, OutputPage)
#9 /srv/mediawiki/php-1.35.0-wmf.39/includes/HookContainer/HookContainer.php(131): MediaWiki\HookContainer\HookContainer->callLegacyHook(string, array, array, array)
#10 /srv/mediawiki/php-1.35.0-wmf.39/includes/HookContainer/HookRunner.php(2516): MediaWiki\HookContainer\HookContainer->run(string, array)
#11 /srv/mediawiki/php-1.35.0-wmf.39/includes/OutputPage.php(3386): MediaWiki\HookContainer\HookRunner->onMakeGlobalVariablesScript(array, OutputPage)
#12 /srv/mediawiki/php-1.35.0-wmf.39/includes/OutputPage.php(3035): OutputPage->getJSVars()
#13 /srv/mediawiki/php-1.35.0-wmf.39/includes/OutputPage.php(3056): OutputPage->getRlClient()
#14 /srv/mediawiki/php-1.35.0-wmf.39/includes/skins/SkinMustache.php(82): OutputPage->headElement(SkinApi)
#15 /srv/mediawiki/php-1.35.0-wmf.39/includes/skins/SkinMustache.php(57): SkinMustache->getTemplateData()
#16 /srv/mediawiki/php-1.35.0-wmf.39/includes/skins/SkinTemplate.php(141): SkinMustache->generateHTML()
#17 /srv/mediawiki/php-1.35.0-wmf.39/includes/OutputPage.php(2616): SkinTemplate->outputPage()
#18 /srv/mediawiki/php-1.35.0-wmf.39/includes/api/ApiFormatBase.php(333): OutputPage->output()
#19 /srv/mediawiki/php-1.35.0-wmf.39/includes/api/ApiFormatRaw.php(82): ApiFormatBase->closePrinter()
#20 /srv/mediawiki/php-1.35.0-wmf.39/includes/api/ApiMain.php(1834): ApiFormatRaw->closePrinter()
#21 /srv/mediawiki/php-1.35.0-wmf.39/includes/api/ApiMain.php(608): ApiMain->printResult(integer)
#22 /srv/mediawiki/php-1.35.0-wmf.39/includes/api/ApiMain.php(532): ApiMain->handleException(ConfigException)
#23 /srv/mediawiki/php-1.35.0-wmf.39/includes/api/ApiMain.php(496): ApiMain->executeActionWithErrorHandling()
#24 /srv/mediawiki/php-1.35.0-wmf.39/api.php(89): ApiMain->execute()
#25 /srv/mediawiki/php-1.35.0-wmf.39/api.php(44): wfApiMain()
#26 /srv/mediawiki/w/api.php(3): require(string)
#27 {main}

Note the spelling of MinervaOverflowInPageActhons.

Pybal quickly depooled the host, but probably monitoring kept querying the server and generating the exceptions.

@Joe did php7adm /opcache-free on mw1346 and the exceptions cleared.

Interestingly, according to the opcache metadata, the file where the error was (/srv/mediawiki/php-1.35.0-wmf.39/skins/MinervaNeue/includes/MinervaHooks.php) was in opcache since 2 weeks, which isn't great because it means nothing really caused this issue:

  • no deploy
  • no opcache invalidation by chance
  • no php restarts
mmodell added a subscriber: mmodell.Jul 8 2020, 6:28 PM

Happened twice recently on

host: wtp1040
timestamp: 2020-07-08T18:06:14
reqid: 1e314c17-ae0b-4826-9dad-ff4177672966

[1e314c17-ae0b-4826-9dad-ff4177672966] PHP Fatal error: Uncaught Error: Call to undefined method Monolog\Logger::hasField() in /srv/mediawiki/php-1.35.0-wmf.39/includes/cache/MessageCache.php:629 in /srv/mediawiki/php-1.35.0-wmf.39/includes/cache/MessageCache.php on line 629

exception.trace
#0 /srv/mediawiki/php-1.35.0-wmf.39/includes/cache/MessageCache.php(284): MessageCache->isLanguageLoaded('en')
#1 /srv/mediawiki/php-1.35.0-wmf.39/includes/cache/MessageCache.php(1091): MessageCache->load('en')
#2 /srv/mediawiki/php-1.35.0-wmf.39/includes/cache/MessageCache.php(1018): MessageCache->getMsgFromNamespace('Internalerror', 'en')
#3 /srv/mediawiki/php-1.35.0-wmf.39/includes/cache/MessageCache.php(988): MessageCache->getMessageForLang(Object(LanguageEn), 'internalerror', true, Array)
#4 /srv/mediawiki/php-1.35.0-wmf.39/includes/cache/MessageCache.php(930): MessageCache->getMessageFromFallbackChain(Object(LanguageEn), 'internalerror', true)
#5 /srv/mediawiki/php-1.35.0-wmf.39/includes/language/Message.php(1302): MessageCache->get('internalerror', true, Object(LanguageEn))
#6 /srv/mediawiki/php-1.35.0-wmf.39/includes/language/Message.php(860): Message
CDanis added a subscriber: CDanis.Jul 8 2020, 6:32 PM
jijiki moved this task from Incoming 🐫 to Unsorted on the serviceops board.Aug 17 2020, 11:46 PM
Krinkle renamed this task from PHP7 corruption: Method call executed on unrelated object (also: Call to undefined method) to PHP7 corruptions (Call on wrong object, Call to undefined method, etc.).Aug 29 2020, 6:33 PM

Another corruption:

Uncaught Psr\Log\InvalidArgumentException: Level "400" is not defined, use one of: 0, 1, 2 in /srv/mediawiki/php-1.36.0-wmf.2/vendor/monolog/monolog/src/Monolog/Logger.php:516

#0 /srv/mediawiki/php-1.36.0-wmf.2/vendor/monolog/monolog/src/Monolog/Logger.php(298): Monolog\Logger::getLevelName(400)
#1 /srv/mediawiki/php-1.36.0-wmf.2/vendor/monolog/monolog/src/Monolog/Logger.php(712): Monolog\Logger->addRecord(400, '[{exception_id}...', Array)
#2 /srv/mediawiki/php-1.36.0-wmf.2/includes/exception/MWExceptionHandler.php(346): Monolog\Logger->error('[{exception_id}...', Array)
#3 [internal function]: MWExceptionHandler::handleFatalError()
#4 {main}
  • Request ID: 56ec2797-9aa6-4203-9fd7-9ab1bbc04078
  • Request URL: https://de.wikibooks.org/w/index.php?title=…&diff=…&oldid=0
  • Timestamp: 2020-07-30T11:04
  • Host: mw1275 (appserver)
  • phpversion: 7.2.31-1+0~20200514.41+debian9~1.gbpe2a56b+wmf1

The level 400 very clearly is defined beyond resonable doubt (vendor src).

Krinkle renamed this task from PHP7 corruptions (Call on wrong object, Call to undefined method, etc.) to PHP7 corruption reports in 2020 (Call on wrong object, etc.).Sep 11 2020, 6:19 PM
Krinkle updated the task description. (Show Details)
  • timestamp: 2020-10-01T16:15:49
  • host: mw2290
  • message:
[ab3d143d-d371-41e4-ab74-ae92255e704f] /w/api.php?action=query&prop=revisions&revids=981318678&rvprop=content&rvparse=1&format=xml   Error from line 1369 of /srv/mediawiki/php-1.36.0-wmf.10/includes/libs/objectcache/wancache/WANObjectCache.php: Call to undefined method Monolog\Logger::get()

We've seen this exact one before at the start of this task. The fact that it tends to gravitate toward methods of this name, or this class, or in this file could be something. The file has changed numerous times since, so it's not likely to only be dependent on random chance or the file leading to a specific hash, but rather something with the specific sequence of statements in this file...

As an additional data point, note that in the merged T264241: Class 'LathMathML' not found task from yesterday, it was a different class / file.

Yeah, that's a classic string one-byte flip issue which does seem to be random and rarely seen twice the same way.

Some upstream bugs for the record from the old task, to be copied into the task description:

Joe removed a subscriber: Joe.Oct 5 2020, 6:21 AM
  • timestamp: 2020-10-20T18:10:00 to 2020-10-20T21:15:00
  • host: mw2252
  • message:

[2b171d8b-48ec-480d-b7a4-187dd3af259c] /w/api.php?titles=Image%3ANorrlands_nation_Nya_entr%C3%A9n2.jpg&iiprop=url&iiurlwidth=120&iiurlheight=120&prop=imageinfo&format=json&action=query TypeError from line 395 of /srv/mediawiki/php-1.36.0-wmf.13/extensions/WikibaseCirrusSearch/src/Hooks.php: Return value of Wikibase\Search\Elastic\Hooks::getWBCSConfig() must be an instance of Vikibase\Search\Elastic\WikibaseSearchConfig, instance of Vikibase\Search\Elastic\WikibaseSearchConfig returned

must be an instance of Vikibase\Search\ [...]

Is this some sort of copying glitch?

must be an instance of Vikibase\Search\ [...]

Is this some sort of copying glitch?

No this is the actual corruption.

Pablo-WMDE added a comment.EditedOct 21 2020, 9:06 AM

must be an instance of Vikibase\Search\ [...]

Is this some sort of copying glitch?

No this is the actual corruption.

Thanks. The "string one-byte flip" - which Krinkle mentioned earlier - has become self-aware and is punning us.

Joe added a comment.Oct 21 2020, 10:59 AM
  • timestamp: 2020-10-20T18:10:00 to 2020-10-20T21:15:00
  • host: mw2252

Please note: this was noticed pretty early but the server was not depooled by mistake.

dancy added a subscriber: dancy.Oct 21 2020, 2:49 PM

Noting for the record in all cases where there's a bogus changed character in a string, the bad character is always one less than what it should be (i.e., it's not a bit flip, it's a decrement).