Page MenuHomePhabricator

mw1384 is misbehaving
Closed, DuplicatePublic

Description

The node is depooled now but up to the point of depool was constantly flooding logstash with errors that didn't make much sense:

PHP Warning: trim() expects parameter 1 to be string, object given
 	#0 [internal function]: MWExceptionHandler::handleError(integer, string, string, integer, array)
#1 /srv/mediawiki/php-1.35.0-wmf.36/extensions/Wikibase/repo/includes/Rdf/Values/ObjectUriRdfBuilder.php(47): trim(Wikibase\DataModel\Entity\EntityIdValue)
#2 /srv/mediawiki/php-1.35.0-wmf.36/extensions/Wikibase/repo/includes/Rdf/Values/ObjectUriRdfBuilder.php(36): Wikibase\Repo\Rdf\Values\ObjectUriRdfBuilder->getValueUri(Wikibase\DataModel\Entity\EntityIdValue)
#3 /srv/mediawiki/php-1.35.0-wmf.36/extensions/Wikibase/repo/includes/Rdf/DispatchingValueSnakRdfBuilder.php(54): Wikibase\Repo\Rdf\Values\ObjectUriRdfBuilder->addValue(Wikimedia\Purtle\TurtleRdfWriter, string, string, string, string, Wikibase\DataModel\Snak\PropertyValueSnak)
#4 /srv/mediawiki/php-1.35.0-wmf.36/extensions/Wikibase/repo/includes/Rdf/SnakRdfBuilder.php(138): Wikibase\Rdf\DispatchingValueSnakRdfBuilder->addValue(Wikimedia\Purtle\TurtleRdfWriter, string, string, string, string, Wikibase\DataModel\Snak\PropertyValueSnak)
#5 /srv/mediawiki/php-1.35.0-wmf.36/extensions/Wikibase/repo/includes/Rdf/SnakRdfBuilder.php(86): Wikibase\Rdf\SnakRdfBuilder->addSnakValue(Wikimedia\Purtle\TurtleRdfWriter, string, Wikibase\DataModel\Snak\PropertyValueSnak, string)
#6 /srv/mediawiki/php-1.35.0-wmf.36/extensions/Wikibase/repo/includes/Rdf/FullStatementRdfBuilder.php(251): Wikibase\Rdf\SnakRdfBuilder->addSnak(Wikimedia\Purtle\TurtleRdfWriter, string, Wikibase\DataModel\Snak\PropertyValueSnak, string, string)
#7 /srv/mediawiki/php-1.35.0-wmf.36/extensions/Wikibase/repo/includes/Rdf/FullStatementRdfBuilder.php(142): Wikibase\Rdf\FullStatementRdfBuilder->addMainSnak(Wikibase\DataModel\Entity\ItemId, string, Wikibase\DataModel\Statement\Statement, boolean)
#8 /srv/mediawiki/php-1.35.0-wmf.36/extensions/Wikibase/repo/includes/Rdf/FullStatementRdfBuilder.php(126): Wikibase\Rdf\FullStatementRdfBuilder->addStatement(Wikibase\DataModel\Entity\ItemId, Wikibase\DataModel\Statement\Statement, boolean)
#9 /srv/mediawiki/php-1.35.0-wmf.36/extensions/Wikibase/repo/includes/Rdf/FullStatementRdfBuilder.php(266): Wikibase\Rdf\FullStatementRdfBuilder->addStatements(Wikibase\DataModel\Entity\ItemId, Wikibase\DataModel\Statement\StatementList)
#10 /srv/mediawiki/php-1.35.0-wmf.36/extensions/Wikibase/repo/includes/Rdf/RdfBuilder.php(430): Wikibase\Rdf\FullStatementRdfBuilder->addEntity(Wikibase\DataModel\Entity\Item)
#11 /srv/mediawiki/php-1.35.0-wmf.36/extensions/Wikibase/repo/includes/Rdf/RdfBuilder.php(411): Wikibase\Rdf\RdfBuilder->addSingleEntity(Wikibase\DataModel\Entity\Item)
#12 /srv/mediawiki/php-1.35.0-wmf.36/extensions/Wikibase/repo/includes/LinkedData/EntityDataSerializationService.php(236): Wikibase\Rdf\RdfBuilder->addEntity(Wikibase\DataModel\Entity\Item)
#13 /srv/mediawiki/php-1.35.0-wmf.36/extensions/Wikibase/repo/includes/LinkedData/EntityDataSerializationService.php(182): Wikibase\Repo\LinkedData\EntityDataSerializationService->rdfSerialize(Wikibase\Lib\Store\EntityRevision, NULL, array, Wikibase\Rdf\RdfBuilder, string)
#14 /srv/mediawiki/php-1.35.0-wmf.36/extensions/Wikibase/repo/includes/LinkedData/EntityDataRequestHandler.php(527): Wikibase\Repo\LinkedData\EntityDataSerializationService->getSerializedData(string, Wikibase\Lib\Store\EntityRevision, NULL, array, string)
#15 /srv/mediawiki/php-1.35.0-wmf.36/extensions/Wikibase/repo/includes/LinkedData/EntityDataRequestHandler.php(276): Wikibase\Repo\LinkedData\EntityDataRequestHandler->showData(WebRequest, OutputPage, string, Wikibase\DataModel\Entity\ItemId, integer)
#16 /srv/mediawiki/php-1.35.0-wmf.36/extensions/Wikibase/repo/includes/Specials/SpecialEntityData.php(157): Wikibase\Repo\LinkedData\EntityDataRequestHandler->handleRequest(string, WebRequest, OutputPage)
#17 /srv/mediawiki/php-1.35.0-wmf.36/includes/specialpage/SpecialPage.php(580): Wikibase\Repo\Specials\SpecialEntityData->execute(string)
#18 /srv/mediawiki/php-1.35.0-wmf.36/includes/specialpage/SpecialPageFactory.php(634): SpecialPage->run(string)
#19 /srv/mediawiki/php-1.35.0-wmf.36/includes/MediaWiki.php(307): MediaWiki\SpecialPage\SpecialPageFactory->executePath(Title, RequestContext)
#20 /srv/mediawiki/php-1.35.0-wmf.36/includes/MediaWiki.php(986): MediaWiki->performRequest()
#21 /srv/mediawiki/php-1.35.0-wmf.36/includes/MediaWiki.php(543): MediaWiki->main()
#22 /srv/mediawiki/php-1.35.0-wmf.36/index.php(47): MediaWiki->run()
#23 /srv/mediawiki/w/index.php(3): require(string)
#24 {main}

Basically all of wikidatawiki's errors came from this node (link to logstash: https://logstash.wikimedia.org/goto/d2833a468e804b3a9926c539ee941093):

It was so many errors that caused icinga to scream.

It's depooled for now but needs investiagtion

We tried scap pull and no improvement.

Event Timeline

Restricted Application added a subscriber: Aklapper. · View Herald TranscriptFri, Jun 12, 4:40 PM
Ladsgroup updated the task description. (Show Details)Fri, Jun 12, 4:43 PM

To add some details - I depooled/repooled doing scap pull and it didn't really work. I also depooled, waited hours, repooled and the issue gets back consistently. It may be a single host issue, but not really sure what.

Mentioned in SAL (#wikimedia-operations) [2020-06-12T16:49:54Z] <elukey> restart php-fpm and pool mw1384 - T255282

elukey closed this task as Resolved.Fri, Jun 12, 4:55 PM
elukey claimed this task.

Error seem to be gone after the php-fpm restart, but I forgot to dump the status of APC :( My bet is on APC corruption or something similar, will reopen in case it re-happen.

Is T253673 relevant here?