Page MenuHomePhabricator
Create Task
Maniphest T255282

mw1384 is misbehaving
Closed, DuplicatePublic
Actions

Description

The node is depooled now but up to the point of depool was constantly flooding logstash with errors that didn't make much sense:

PHP Warning: trim() expects parameter 1 to be string, object given
 	#0 [internal function]: MWExceptionHandler::handleError(integer, string, string, integer, array)
#1 /srv/mediawiki/php-1.35.0-wmf.36/extensions/Wikibase/repo/includes/Rdf/Values/ObjectUriRdfBuilder.php(47): trim(Wikibase\DataModel\Entity\EntityIdValue)
#2 /srv/mediawiki/php-1.35.0-wmf.36/extensions/Wikibase/repo/includes/Rdf/Values/ObjectUriRdfBuilder.php(36): Wikibase\Repo\Rdf\Values\ObjectUriRdfBuilder->getValueUri(Wikibase\DataModel\Entity\EntityIdValue)
#3 /srv/mediawiki/php-1.35.0-wmf.36/extensions/Wikibase/repo/includes/Rdf/DispatchingValueSnakRdfBuilder.php(54): Wikibase\Repo\Rdf\Values\ObjectUriRdfBuilder->addValue(Wikimedia\Purtle\TurtleRdfWriter, string, string, string, string, Wikibase\DataModel\Snak\PropertyValueSnak)
#4 /srv/mediawiki/php-1.35.0-wmf.36/extensions/Wikibase/repo/includes/Rdf/SnakRdfBuilder.php(138): Wikibase\Rdf\DispatchingValueSnakRdfBuilder->addValue(Wikimedia\Purtle\TurtleRdfWriter, string, string, string, string, Wikibase\DataModel\Snak\PropertyValueSnak)
#5 /srv/mediawiki/php-1.35.0-wmf.36/extensions/Wikibase/repo/includes/Rdf/SnakRdfBuilder.php(86): Wikibase\Rdf\SnakRdfBuilder->addSnakValue(Wikimedia\Purtle\TurtleRdfWriter, string, Wikibase\DataModel\Snak\PropertyValueSnak, string)
#6 /srv/mediawiki/php-1.35.0-wmf.36/extensions/Wikibase/repo/includes/Rdf/FullStatementRdfBuilder.php(251): Wikibase\Rdf\SnakRdfBuilder->addSnak(Wikimedia\Purtle\TurtleRdfWriter, string, Wikibase\DataModel\Snak\PropertyValueSnak, string, string)
#7 /srv/mediawiki/php-1.35.0-wmf.36/extensions/Wikibase/repo/includes/Rdf/FullStatementRdfBuilder.php(142): Wikibase\Rdf\FullStatementRdfBuilder->addMainSnak(Wikibase\DataModel\Entity\ItemId, string, Wikibase\DataModel\Statement\Statement, boolean)
#8 /srv/mediawiki/php-1.35.0-wmf.36/extensions/Wikibase/repo/includes/Rdf/FullStatementRdfBuilder.php(126): Wikibase\Rdf\FullStatementRdfBuilder->addStatement(Wikibase\DataModel\Entity\ItemId, Wikibase\DataModel\Statement\Statement, boolean)
#9 /srv/mediawiki/php-1.35.0-wmf.36/extensions/Wikibase/repo/includes/Rdf/FullStatementRdfBuilder.php(266): Wikibase\Rdf\FullStatementRdfBuilder->addStatements(Wikibase\DataModel\Entity\ItemId, Wikibase\DataModel\Statement\StatementList)
#10 /srv/mediawiki/php-1.35.0-wmf.36/extensions/Wikibase/repo/includes/Rdf/RdfBuilder.php(430): Wikibase\Rdf\FullStatementRdfBuilder->addEntity(Wikibase\DataModel\Entity\Item)
#11 /srv/mediawiki/php-1.35.0-wmf.36/extensions/Wikibase/repo/includes/Rdf/RdfBuilder.php(411): Wikibase\Rdf\RdfBuilder->addSingleEntity(Wikibase\DataModel\Entity\Item)
#12 /srv/mediawiki/php-1.35.0-wmf.36/extensions/Wikibase/repo/includes/LinkedData/EntityDataSerializationService.php(236): Wikibase\Rdf\RdfBuilder->addEntity(Wikibase\DataModel\Entity\Item)
#13 /srv/mediawiki/php-1.35.0-wmf.36/extensions/Wikibase/repo/includes/LinkedData/EntityDataSerializationService.php(182): Wikibase\Repo\LinkedData\EntityDataSerializationService->rdfSerialize(Wikibase\Lib\Store\EntityRevision, NULL, array, Wikibase\Rdf\RdfBuilder, string)
#14 /srv/mediawiki/php-1.35.0-wmf.36/extensions/Wikibase/repo/includes/LinkedData/EntityDataRequestHandler.php(527): Wikibase\Repo\LinkedData\EntityDataSerializationService->getSerializedData(string, Wikibase\Lib\Store\EntityRevision, NULL, array, string)
#15 /srv/mediawiki/php-1.35.0-wmf.36/extensions/Wikibase/repo/includes/LinkedData/EntityDataRequestHandler.php(276): Wikibase\Repo\LinkedData\EntityDataRequestHandler->showData(WebRequest, OutputPage, string, Wikibase\DataModel\Entity\ItemId, integer)
#16 /srv/mediawiki/php-1.35.0-wmf.36/extensions/Wikibase/repo/includes/Specials/SpecialEntityData.php(157): Wikibase\Repo\LinkedData\EntityDataRequestHandler->handleRequest(string, WebRequest, OutputPage)
#17 /srv/mediawiki/php-1.35.0-wmf.36/includes/specialpage/SpecialPage.php(580): Wikibase\Repo\Specials\SpecialEntityData->execute(string)
#18 /srv/mediawiki/php-1.35.0-wmf.36/includes/specialpage/SpecialPageFactory.php(634): SpecialPage->run(string)
#19 /srv/mediawiki/php-1.35.0-wmf.36/includes/MediaWiki.php(307): MediaWiki\SpecialPage\SpecialPageFactory->executePath(Title, RequestContext)
#20 /srv/mediawiki/php-1.35.0-wmf.36/includes/MediaWiki.php(986): MediaWiki->performRequest()
#21 /srv/mediawiki/php-1.35.0-wmf.36/includes/MediaWiki.php(543): MediaWiki->main()
#22 /srv/mediawiki/php-1.35.0-wmf.36/index.php(47): MediaWiki->run()
#23 /srv/mediawiki/w/index.php(3): require(string)
#24 {main}

Basically all of wikidatawiki's errors came from this node (link to logstash: https://logstash.wikimedia.org/goto/d2833a468e804b3a9926c539ee941093):

image.png (209×789 px, 14 KB)

It was so many errors that caused icinga to scream.

It's depooled for now but needs investiagtion

We tried scap pull and no improvement.

Related Objects

Event Timeline

Ladsgroup created this task.Jun 12 2020, 4:40 PM
Restricted Application added a subscriber: Aklapper. · View Herald TranscriptJun 12 2020, 4:40 PM
Ladsgroup updated the task description. (Show Details)Jun 12 2020, 4:43 PM
elukey added a comment.Jun 12 2020, 4:44 PM

To add some details - I depooled/repooled doing scap pull and it didn't really work. I also depooled, waited hours, repooled and the issue gets back consistently. It may be a single host issue, but not really sure what.

elukey added a project: serviceops.Jun 12 2020, 4:44 PM
Stashbot added a comment.Jun 12 2020, 4:49 PM

Mentioned in SAL (#wikimedia-operations) [2020-06-12T16:49:54Z] <elukey> restart php-fpm and pool mw1384 - T255282

Addshore mentioned this in T255279: Errors in P166 page about wikidata types.Jun 12 2020, 4:52 PM
elukey closed this task as Resolved.Jun 12 2020, 4:55 PM
elukey claimed this task.

Error seem to be gone after the php-fpm restart, but I forgot to dump the status of APC :( My bet is on APC corruption or something similar, will reopen in case it re-happen.

Addshore merged a task: T255279: Errors in P166 page about wikidata types.Jun 12 2020, 4:55 PM
Addshore added subscribers: Manu1400, Adithyak1997.
Addshore added a comment.Jun 15 2020, 9:25 AM

Is T253673 relevant here?

Krinkle closed this task as a duplicate of T245183: PHP7 corruption reports in 2020-2022 (Call on wrong object, etc.).Jun 22 2020, 7:11 PM
• Zbyszko mentioned this in T264042: Query service returns weird Commons links when asking for P50 (author) statement value.Oct 22 2020, 10:04 AM
dcausse mentioned this in T255657: Strange result in Wikidata query (full URLs given instead of identifiers).Oct 22 2020, 1:28 PM
Content licensed under Creative Commons Attribution-ShareAlike (CC BY-SA) 4.0 unless otherwise noted; code licensed under GNU General Public License (GPL) 2.0 or later and other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL