Checker.php:369 Cannot fetch ids from index
Closed, ResolvedPublicPRODUCTION ERROR
Actions

Assigned To

Authored By

	thcipriani
	May 9 2019, 8:24 PM

Description

AWqeP542m2VjIW06Z09x

trace

#0 /srv/mediawiki/php-1.34.0-wmf.4/extensions/CirrusSearch/includes/Sanity/Checker.php(122): CirrusSearch\Sanity\Checker->loadPagesFromIndex(array)
#1 /srv/mediawiki/php-1.34.0-wmf.4/extensions/CirrusSearch/includes/Job/CheckerJob.php(217): CirrusSearch\Sanity\Checker->check(array)
#2 /srv/mediawiki/php-1.34.0-wmf.4/extensions/CirrusSearch/includes/Job/Job.php(100): CirrusSearch\Job\CheckerJob->doJob()
#3 /srv/mediawiki/php-1.34.0-wmf.4/extensions/EventBus/includes/JobExecutor.php(66): CirrusSearch\Job\Job->run()
#4 /srv/mediawiki/rpc/RunSingleJob.php(77): JobExecutor->execute(array)
#5 {main}

Impact

Over 12,000 failed JobQueue jobs per day, logged as "ERROR" severity in the production exception channel.

Details

	Subject	Repo	Branch	Lines +/-
	Don't spam the logs with errors from saneitizer jobs when elastic is down	mediawiki/extensions/CirrusSearch	master	+23 -3

Customize query in gerrit

Related Objects

Mentioned In: Blog Post: Production Excellence #10: April 2019

Event Timeline

thcipriani created this task.May 9 2019, 8:24 PM

Restricted Application added a project: Discovery-Search. · View Herald TranscriptMay 9 2019, 8:24 PM

Restricted Application added a subscriber: Aklapper. · View Herald Transcript

thcipriani moved this task from Untriaged to Jul2019/1.34.wmf.13+ on the Wikimedia-production-error board.May 9 2019, 8:25 PM

Today I have seen some alarms firing for mediawiki exceptions due to this error :)

Krinkle updated the task description. (Show Details)May 19 2019, 9:38 AM

Krinkle updated the task description. (Show Details)

Seen since 1.34-wmf.1 in the Logstash. Tentatively triaging as High priority due to it being one of the top 5 most frequent production errors, which is making is making it hard to reliably detect new regressions that are less frequent than this one.

If these jobs are not required to succeed (e.g. they just try something and that's it, no further action to be taken), then the Job class should presumably catch and ignore all exceptions and still return true.

If the internal failure rate is of interest to the maintainers, one could consider a Statsd metric or INFO-severity message in its stead.

debt edited projects, added Discovery-Search (Current work); removed Discovery-Search.May 30 2019, 4:12 PM

Change 513585 had a related patch set uploaded (by DCausse; owner: DCausse):
[mediawiki/extensions/CirrusSearch@master] Don't spam the logs with errors from saneitizer jobs when elastic is down

https://gerrit.wikimedia.org/r/513585

gerritbot added a project: Patch-For-Review.May 31 2019, 12:44 PM

dcausse claimed this task.May 31 2019, 12:44 PM

dcausse moved this task from Incoming to Needs review on the Discovery-Search (Current work) board.

Change 513585 merged by jenkins-bot:
[mediawiki/extensions/CirrusSearch@master] Don't spam the logs with errors from saneitizer jobs when elastic is down

https://gerrit.wikimedia.org/r/513585

Maintenance_bot removed a project: Patch-For-Review.May 31 2019, 6:10 PM

ReleaseTaggerBot added a project: MW-1.34-notes (1.34.0-wmf.8; 2019-06-04).May 31 2019, 7:00 PM

Krinkle mentioned this in Blog Post: Production Excellence #10: April 2019.May 31 2019, 7:21 PM

EBernhardson moved this task from Needs review to Needs Reporting on the Discovery-Search (Current work) board.Jun 3 2019, 4:00 PM

Confirmed fixed in prod on 1.34-wmf.8:

• mmodell changed the subtype of this task from "Task" to "Production Error".Aug 28 2019, 11:07 PM

	F29594174: capture.png
	Jun 17 2019, 6:33 PM

extensions/CirrusSearch/includes/Sanity/Checker.php:369 Cannot fetch ids from indexClosed, ResolvedPublicPRODUCTION ERRORActions