Page MenuHomePhabricator
Create Task
Maniphest T306168

Better classification of CirrusSearch errors
Open, Needs TriagePublic5 Estimated Story Points
Actions

Description

As a maintainer of the search infrastructure I want to have more precise metrics regarding errors that occur between CirrusSearch and Elasticsearch so that I can better understand the problems on the cluster.

The CirrusSearch failures are currently categorized into 3 buckets: rejected, failed and unknown. The unknown bucket is currently seeing 1 error/minute so it would be interesting to know what these are, especially if they relate to indexing documents.

Graph: https://grafana.wikimedia.org/d/000000455/elasticsearch-percentiles?orgId=1&refresh=1m&viewPanel=9

AC:

  • the number of unknown errors should be exceptional (close to 0/day)

Details

SubjectRepoBranchLines +/-
Throw proper response exception when response is invalidmediawiki/extensions/CirrusSearchmaster+13 -11
Create new heuristic for config related issuesmediawiki/extensions/CirrusSearchmaster+11 -2
Throw exceptions when multisearch fails due to connectionmediawiki/extensions/CirrusSearchmaster+20 -0
Add logic to validate response data on sendDatamediawiki/extensions/CirrusSearchmaster+27 -2
Fix numBulk sometimes not set in DataSendermediawiki/extensions/CirrusSearchmaster+16 -24
Improve error classification on cirrussearchmediawiki/extensions/CirrusSearchmaster+33 -3
Add a way to identify CirrusSearch error types on logstashmediawiki/extensions/CirrusSearchmaster+2 -0
Customize query in gerrit

Related Objects

Event Timeline

dcausse created this task.Apr 14 2022, 8:59 AM
Restricted Application added a project: Discovery-Search. · View Herald TranscriptApr 14 2022, 8:59 AM
Restricted Application added a subscriber: Aklapper. · View Herald Transcript
EBernhardson edited projects, added Discovery-Search (Current work); removed Discovery-Search.Apr 25 2022, 3:06 PM
EBernhardson set the point value for this task to 5.Apr 25 2022, 3:57 PM
EBernhardson moved this task from Incoming to Ready for Dev -- SWE on the Discovery-Search (Current work) board.
gerritbot added a comment.May 9 2022, 2:51 PM

Change 790323 had a related patch set uploaded (by EJoseph; author: EJoseph):

[mediawiki/extensions/CirrusSearch@master] Add a way to identify CirrusSearch error types on logstash

https://gerrit.wikimedia.org/r/790323

gerritbot added a project: Patch-For-Review.May 9 2022, 2:51 PM
dcausse moved this task from Ready for Dev -- SWE to In Progress on the Discovery-Search (Current work) board.May 9 2022, 3:19 PM
Gehel assigned this task to • EJoseph.May 9 2022, 3:19 PM
gerritbot added a comment.May 9 2022, 10:09 PM

Change 790323 merged by jenkins-bot:

[mediawiki/extensions/CirrusSearch@master] Add a way to identify CirrusSearch error types on logstash

https://gerrit.wikimedia.org/r/790323

Maintenance_bot removed a project: Patch-For-Review.May 9 2022, 10:30 PM
ReleaseTaggerBot added a project: MW-1.39-notes (1.39.0-wmf.12; 2022-05-16).May 10 2022, 2:01 AM
• EJoseph added a comment.EditedMay 30 2022, 10:01 AM

The unknown error message

  • Search backend error during comp_suggest search for '[redacted]' after 252: unknown: Status code 503; upstream connect error or disconnect/reset before headers. reset reason: connection failure same error happens on more_like, comp_suggest queries, it could also be connection termination
  • Search backend error during sending {numBulk} documents to the UNKNOWN index(s) after 253: error message seems to be empty, this happens during send_data_write
  • Search backend error during counting links to 1 pages after 250: unknown: Received null for link count on 1 out of 1 pages on count_links
Search backend error during full_text search for '&&' after 88: parse_exception: parse_exception: Encountered " <AND> "&& "" at line 1, column 0.
Was expecting one of:
    <NOT> ...
    "+" ...
    "-" ...
    <BAREOPER> ...
    "(" ...
    "*" ...
    <QUOTED> ...
    <TERM> ...
    <PREFIXTERM> ...
    <WILDTERM> ...
    <REGEXPTERM> ...
    "[" ...
    "{" ...
    <NUMBER> ...
    <TERM> ...

on full_text

  • Search backend error during regex search for 'insource://>/' after 8: parsing_exception: Required [regex] on regex
  • index_not_found_exception: no such index on full_text
Search backend error during sending 1 documents to the enwiki_general_1617317067 index(s) after 146: bulk: Error in one or more bulk request actions:

update: /enwiki_general_1617317067/_doc/15024081 caused shard is not in primary mode [index: enwiki_general_1617317067]
  - update: /enwiki_general_1617317067/_doc/15024081 caused shard is not in primary mode [index: enwiki_general_1617317067]: retry_on_primary_exception: shard is not in primary mode (CurrentState[STARTED] shard is not in primary mode)

on send_data_write

gerritbot added a comment.Jun 1 2022, 9:59 AM

Change 802082 had a related patch set uploaded (by EJoseph; author: EJoseph):

[mediawiki/extensions/CirrusSearch@master] Improve error classification on cirrussearch

https://gerrit.wikimedia.org/r/802082

gerritbot added a project: Patch-For-Review.Jun 1 2022, 9:59 AM
gerritbot added a comment.Jun 6 2022, 5:05 PM

Change 802082 merged by jenkins-bot:

[mediawiki/extensions/CirrusSearch@master] Improve error classification on cirrussearch

https://gerrit.wikimedia.org/r/802082

Maintenance_bot removed a project: Patch-For-Review.Jun 6 2022, 5:30 PM
ReleaseTaggerBot edited projects, added MW-1.39-notes (1.39.0-wmf.15; 2022-06-06); removed MW-1.39-notes (1.39.0-wmf.12; 2022-05-16).Jun 6 2022, 6:00 PM
gerritbot added a comment.Jun 9 2022, 10:39 AM

Change 804282 had a related patch set uploaded (by EJoseph; author: EJoseph):

[mediawiki/extensions/CirrusSearch@master] Fix numBulk sometimes not set in DataSender

https://gerrit.wikimedia.org/r/804282

gerritbot added a project: Patch-For-Review.Jun 9 2022, 10:39 AM
EBernhardson moved this task from In Progress to Needs review on the Discovery-Search (Current work) board.Jun 9 2022, 2:49 PM
thcipriani mentioned this in T311247: CirrusSearchChangeFailed: Error in one or more bulk request actions.Jun 23 2022, 3:24 PM
gerritbot added a comment.Jun 24 2022, 7:37 AM

Change 804282 merged by jenkins-bot:

[mediawiki/extensions/CirrusSearch@master] Fix numBulk sometimes not set in DataSender

https://gerrit.wikimedia.org/r/804282

ReleaseTaggerBot edited projects, added MW-1.39-notes (1.39.0-wmf.18; 2022-06-27); removed MW-1.39-notes (1.39.0-wmf.15; 2022-06-06).Jun 24 2022, 8:00 AM
gerritbot added a comment.Jul 18 2022, 4:21 PM

Change 814854 had a related patch set uploaded (by EJoseph; author: EJoseph):

[mediawiki/extensions/CirrusSearch@master] Throw proper response exception when response is invalid

https://gerrit.wikimedia.org/r/814854

gerritbot added a comment.Jul 21 2022, 10:30 AM

Change 815920 had a related patch set uploaded (by EJoseph; author: EJoseph):

[mediawiki/extensions/CirrusSearch@master] Add logic to validate response data on sendData

https://gerrit.wikimedia.org/r/815920

gerritbot added a comment.Jul 26 2022, 8:58 AM

Change 815920 merged by jenkins-bot:

[mediawiki/extensions/CirrusSearch@master] Add logic to validate response data on sendData

https://gerrit.wikimedia.org/r/815920

ReleaseTaggerBot edited projects, added MW-1.39-notes (1.39.0-wmf.23; 2022-08-01); removed MW-1.39-notes (1.39.0-wmf.18; 2022-06-27).Jul 26 2022, 9:00 AM
• EJoseph moved this task from Needs review to To Be Deployed on the Discovery-Search (Current work) board.Aug 8 2022, 3:26 PM
EBernhardson moved this task from To Be Deployed to Needs Reporting on the Discovery-Search (Current work) board.Aug 22 2022, 3:16 PM
• EJoseph moved this task from Needs Reporting to In Progress on the Discovery-Search (Current work) board.Aug 24 2022, 9:29 AM
• EJoseph added a comment.Aug 29 2022, 9:43 AM

New unknown error messages

  • Search backend error during sending {numBulk} documents to the UNKNOWN index(s) after 253: error message seems to be empty, this happens during send_data_write
  • index_not_found_exception: no such index and [action.auto_create_index] contains [-*] on send_data_write
  • unknown: Received null for link count on 1 out of 1 pages on count_links
gerritbot added a comment.Aug 31 2022, 10:21 AM

Change 828490 had a related patch set uploaded (by EJoseph; author: EJoseph):

[mediawiki/extensions/CirrusSearch@master] Throw exceptions when multisearch fails due to connection

https://gerrit.wikimedia.org/r/828490

gerritbot added a comment.Sep 1 2022, 10:04 AM

Change 828798 had a related patch set uploaded (by EJoseph; author: EJoseph):

[mediawiki/extensions/CirrusSearch@master] Create new heuristic for config related issues

https://gerrit.wikimedia.org/r/828798

gerritbot added a comment.Sep 6 2022, 3:16 PM

Change 828490 merged by jenkins-bot:

[mediawiki/extensions/CirrusSearch@master] Throw exceptions when multisearch fails due to connection

https://gerrit.wikimedia.org/r/828490

gerritbot added a comment.Sep 6 2022, 3:38 PM

Change 828798 merged by jenkins-bot:

[mediawiki/extensions/CirrusSearch@master] Create new heuristic for config related issues

https://gerrit.wikimedia.org/r/828798

ReleaseTaggerBot added a project: MW-1.40-notes (1.40.0-wmf.1; 2022-09-12).Sep 6 2022, 4:00 PM
gerritbot added a comment.Oct 7 2022, 7:33 AM

Change 814854 abandoned by EJoseph:

[mediawiki/extensions/CirrusSearch@master] Throw proper response exception when response is invalid

Reason:

https://gerrit.wikimedia.org/r/814854

dcausse removed • EJoseph as the assignee of this task.Oct 31 2022, 4:20 PM
dcausse removed projects: MW-1.40-notes (1.40.0-wmf.1; 2022-09-12), MW-1.39-notes (1.39.0-wmf.23; 2022-08-01), Discovery-Search (Current work).
dcausse added a subscriber: • EJoseph.
Content licensed under Creative Commons Attribution-ShareAlike (CC BY-SA) 4.0 unless otherwise noted; code licensed under GNU General Public License (GPL) 2.0 or later and other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL