Flow\Exception\InvalidReferenceException in CirrusSearch DataSender::sendData job
Closed, DeclinedPublicPRODUCTION ERROR
Actions

Assigned To

None

Authored By

	Tgr
	May 6 2021, 2:29 PM

Description

Error

mwversion: 1.37.0-wmf.4
reqId: ce75c6aebfd3363649ed46e8
Find reqId in Logstash
Find normalized_message in Logstash

normalized_message

Exception thrown while running DataSender::{method} in cluster {cluster}: {errorMessage}

exception.trace

from /srv/mediawiki/php-1.37.0-wmf.4/extensions/Flow/includes/Model/URLReference.php(27)
#0 /srv/mediawiki/php-1.37.0-wmf.4/extensions/Flow/includes/Model/URLReference.php(64): Flow\Model\URLReference->__construct(Flow\Model\UUID, string, Flow\Model\UUID, Title, string, Flow\Model\UUID, string, string)
#1 /srv/mediawiki/php-1.37.0-wmf.4/extensions/Flow/includes/Data/Mapper/BasicObjectMapper.php(40): Flow\Model\URLReference::fromStorageRow(array, NULL)
#2 /srv/mediawiki/php-1.37.0-wmf.4/extensions/Flow/includes/Data/ObjectLocator.php(315): Flow\Data\Mapper\BasicObjectMapper->fromStorageRow(array)
#3 /srv/mediawiki/php-1.37.0-wmf.4/extensions/Flow/includes/Data/ObjectManager.php(307): Flow\Data\ObjectLocator->load(array)
#4 /srv/mediawiki/php-1.37.0-wmf.4/extensions/Flow/includes/Data/ObjectLocator.php(119): Flow\Data\ObjectManager->load(array)
#5 /srv/mediawiki/php-1.37.0-wmf.4/extensions/Flow/includes/Data/ObjectLocator.php(70): Flow\Data\ObjectLocator->findMulti(array, array)
#6 /srv/mediawiki/php-1.37.0-wmf.4/extensions/Flow/includes/Data/ManagerGroup.php(127): Flow\Data\ObjectLocator->find(array)
#7 /srv/mediawiki/php-1.37.0-wmf.4/extensions/Flow/includes/Data/ManagerGroup.php(139): Flow\Data\ManagerGroup->call(string, array)
#8 /srv/mediawiki/php-1.37.0-wmf.4/extensions/Flow/includes/LinksTableUpdater.php(131): Flow\Data\ManagerGroup->find(string, array)
#9 /srv/mediawiki/php-1.37.0-wmf.4/extensions/Flow/includes/LinksTableUpdater.php(51): Flow\LinksTableUpdater->getReferencesForTitle(Title)
#10 /srv/mediawiki/php-1.37.0-wmf.4/extensions/Flow/includes/Content/BoardContent.php(196): Flow\LinksTableUpdater->mutateParserOutput(Title, ParserOutput)
#11 /srv/mediawiki/php-1.37.0-wmf.4/includes/Revision/RenderedRevision.php(266): Flow\Content\BoardContent->getParserOutput(Title, integer, ParserOptions, boolean)
#12 /srv/mediawiki/php-1.37.0-wmf.4/includes/Revision/RenderedRevision.php(235): MediaWiki\Revision\RenderedRevision->getSlotParserOutputUncached(Flow\Content\BoardContent, boolean)
#13 /srv/mediawiki/php-1.37.0-wmf.4/includes/Revision/RevisionRenderer.php(217): MediaWiki\Revision\RenderedRevision->getSlotParserOutput(string, array)
#14 /srv/mediawiki/php-1.37.0-wmf.4/includes/Revision/RevisionRenderer.php(154): MediaWiki\Revision\RevisionRenderer->combineSlotOutput(MediaWiki\Revision\RenderedRevision, array)
#15 [internal function]: MediaWiki\Revision\RevisionRenderer->MediaWiki\Revision\{closure}(MediaWiki\Revision\RenderedRevision, array)
#16 /srv/mediawiki/php-1.37.0-wmf.4/includes/Revision/RenderedRevision.php(197): call_user_func(Closure, MediaWiki\Revision\RenderedRevision, array)
#17 /srv/mediawiki/php-1.37.0-wmf.4/includes/content/ContentHandler.php(1443): MediaWiki\Revision\RenderedRevision->getRevisionParserOutput(array)
#18 /srv/mediawiki/php-1.37.0-wmf.4/extensions/CirrusSearch/includes/BuildDocument/ParserOutputPageProperties.php(85): ContentHandler->getParserOutputForIndexing(WikiPage, ParserCache)
#19 /srv/mediawiki/php-1.37.0-wmf.4/extensions/CirrusSearch/includes/BuildDocument/ParserOutputPageProperties.php(68): CirrusSearch\BuildDocument\ParserOutputPageProperties->finalizeReal(Elastica\Document, WikiPage, ParserCache, CirrusSearch\CirrusSearch)
#20 /srv/mediawiki/php-1.37.0-wmf.4/extensions/CirrusSearch/includes/BuildDocument/BuildDocument.php(165): CirrusSearch\BuildDocument\ParserOutputPageProperties->finalize(Elastica\Document, Title)
#21 /srv/mediawiki/php-1.37.0-wmf.4/extensions/CirrusSearch/includes/DataSender.php(316): CirrusSearch\BuildDocument\BuildDocument->finalize(Elastica\Document)
#22 /srv/mediawiki/php-1.37.0-wmf.4/extensions/CirrusSearch/includes/Job/ElasticaWrite.php(136): CirrusSearch\DataSender->sendData(string, array)
#23 /srv/mediawiki/php-1.37.0-wmf.4/extensions/CirrusSearch/includes/Job/JobTraits.php(136): CirrusSearch\Job\ElasticaWrite->doJob()
#24 /srv/mediawiki/php-1.37.0-wmf.4/extensions/EventBus/includes/JobExecutor.php(79): CirrusSearch\Job\CirrusGenericJob->run()
#25 /srv/mediawiki/rpc/RunSingleJob.php(76): MediaWiki\Extension\EventBus\JobExecutor->execute(array)
#26 {main}

Impact

15 instances today. Presumably breaks search indexing (but then, does that even work for Flow pages?)

Notes

The URL for which UrlReference triggers this error is http://?????????/mediawiki/load.php... which looks weird. Some kind of internal hack?

Details

Request URL: https://jobrunner.discovery.wmnet/rpc/RunSingleJob.php

Related Objects

Mentioned In: Blog Post: Production Excellence #32: May 2021
Mentioned Here: T76823: Basic search on Flow board (tracking)

Event Timeline

Tgr created this task.May 6 2021, 2:29 PM

Restricted Application added projects: Growth-Team, Discovery-Search. · View Herald TranscriptMay 6 2021, 2:29 PM

Restricted Application added a subscriber: Aklapper. · View Herald Transcript

kostajh moved this task from Inbox to Needs Discussion on the Growth-Team board.May 7 2021, 8:13 AM

MPhamWMF moved this task from needs triage to Current work on the Discovery-Search board.May 10 2021, 3:14 PM

MPhamWMF edited projects, added Discovery-Search (Current work); removed Discovery-Search.

MPhamWMF moved this task from Current work to watching / waiting on the Discovery-Search board.May 10 2021, 3:39 PM

MPhamWMF edited projects, added Discovery-Search; removed Discovery-Search (Current work).

Krinkle moved this task from Untriaged to May 2021 on the Wikimedia-production-error board.May 12 2021, 6:39 PM

Exception thrown while running DataSender is still reported in wmf.6 (count 15).

Krinkle mentioned this in Blog Post: Production Excellence #32: May 2021.Jun 21 2021, 1:31 AM

@DMburugu

Tends to happen in short bursts (e.g. 50/min for a few minutes). Per T76823: Basic search on Flow board (tracking) Flow has no search support so probably fine to ignore.

Lowering the priority of this one given the comments above.

Closing out low/est priority tasks over 6 months old with no activity within last 6 months in order to clean out the backlog of tickets we will not be addressing in the near term. Please feel free to reopen if you think a ticket is important, but bare in mind that given current priorities and resourcing, it is unlikely for the Search team to pick up these tasks for the indefinite future. We hope that the requested changes have either been addressed by or made irrelevant by work the team has done or is doing -- e.g. upgrading Elasticsearch to a newer version will solve various ES-related problems -- or will be subsumed by future work in a more generalized way.

Re-opening tasks and removing from team workboard per IRC feedback given yesterday and discussion with MPham.

Declining this was the right choice IMO. No user impact, doesn't happen a lot, Flow is abandonware so no point in doing unimportant work on it.

Flow\Exception\InvalidReferenceException in CirrusSearch DataSender::sendData jobClosed, DeclinedPublicPRODUCTION ERRORActions