Page MenuHomePhabricator

CirrusSearch generates PHP notices when server is down
Closed, ResolvedPublic

Description

When our ES server went down, these started appearing in our logs.

PHP Fatal error: Call to a member function getTotalHits() on null in /srv/mediawiki/tags/2015-07-15_00:50:47/extensions/CirrusSearch/includes/BuildDocument/RedirectsAndIncomingLinks.php on line 111
PHP Notice: Undefined index: timed_out in /srv/mediawiki/tags/2015-07-15_00:50:47/extensions/CirrusSearch/includes/Searcher.php on line 1199

Event Timeline

Nikerabbit raised the priority of this task from to Needs Triage.
Nikerabbit updated the task description. (Show Details)
Nikerabbit added a project: CirrusSearch.
Nikerabbit added a subscriber: Nikerabbit.
Restricted Application added a project: Discovery. · View Herald TranscriptJul 15 2015, 2:31 PM
Restricted Application added a subscriber: Aklapper. · View Herald Transcript

I'm getting a similar error from runJobs.php:

PHP Fatal error:  Call to a member function getTotalHits() on a non-object
in .../extensions/CirrusSearch/includes/BuildDocument/RedirectsAndIncomingLinks.php on line 111
Deskana triaged this task as Low priority.Dec 31 2015, 12:37 AM
Deskana moved this task from Needs triage to Search on the Discovery board.
Deskana moved this task from Uncategorised to Technical on the CirrusSearch board.
Deskana added a subscriber: Deskana.
Pchelolo added a subscriber: Pchelolo.

Just started happening again recently with a fairly high rate: https://logstash.wikimedia.org/goto/b78e539d56b4286f3096b061d2ebbc51

Restricted Application added a project: Discovery-Search. · View Herald TranscriptAug 2 2018, 10:53 PM
mobrovac raised the priority of this task from Low to High.Aug 3 2018, 10:48 AM
mobrovac added a subscriber: mobrovac.

Raising the priority since there have been more than 20k such messages in the past 24h.

debt added a subscriber: debt.

We'll take a look.

Ping! This is still very noisy in the logs 45K messages in the past week.

Stacktrace if it's helpful:

#0 /srv/mediawiki/php-1.32.0-wmf.16/extensions/CirrusSearch/includes/BuildDocument/RedirectsAndIncomingLinks.php(93): CirrusSearch\BuildDocument\RedirectsAndIncomingLinks->realFinishBatch(array)
#1 /srv/mediawiki/php-1.32.0-wmf.16/includes/Hooks.php(174): CirrusSearch\BuildDocument\RedirectsAndIncomingLinks::finishBatch(array)
#2 /srv/mediawiki/php-1.32.0-wmf.16/includes/Hooks.php(202): Hooks::callHook(string, array, array, NULL)
#3 /srv/mediawiki/php-1.32.0-wmf.16/extensions/CirrusSearch/includes/Updater.php(413): Hooks::run(string, array)
#4 /srv/mediawiki/php-1.32.0-wmf.16/extensions/CirrusSearch/includes/Updater.php(204): CirrusSearch\Updater->buildDocumentsForPages(array, integer)
#5 /srv/mediawiki/php-1.32.0-wmf.16/extensions/CirrusSearch/includes/Updater.php(457): CirrusSearch\Updater->updatePages(array, integer)
#6 /srv/mediawiki/php-1.32.0-wmf.16/extensions/CirrusSearch/includes/Job/IncomingLinkCount.php(36): CirrusSearch\Updater->updateLinkedArticles(array)
#7 /srv/mediawiki/php-1.32.0-wmf.16/extensions/CirrusSearch/includes/Job/Job.php(99): CirrusSearch\Job\IncomingLinkCount->doJob()
#8 /srv/mediawiki/php-1.32.0-wmf.16/extensions/EventBus/includes/JobExecutor.php(67): CirrusSearch\Job\Job->run()
#9 /srv/mediawiki/rpc/RunSingleJob.php(67): JobExecutor->execute(array)
#10 {main}

Change 453539 had a related patch set uploaded (by EBernhardson; owner: EBernhardson):
[mediawiki/extensions/CirrusSearch@master] Handle null result from link count msearch

https://gerrit.wikimedia.org/r/453539

Change 453539 merged by jenkins-bot:
[mediawiki/extensions/CirrusSearch@master] Handle null result from link count msearch

https://gerrit.wikimedia.org/r/453539

debt closed this task as Resolved.Aug 24 2018, 4:07 PM