We lost an Elastic Psi master to hardware failure in T311939 . The only notice we received was an email on Sunday at UTC 0853 . During a subsequent reimage operation, we lost another master and the entire psi cluster in CODFW went offline for a few minutes. (Note that there was no discernable user impact).
Creating these this ticket so Search team can decide on the proper urgency for failed masters and add the appropriate amount/type of alerting.
| Subject | Repo | Branch | Lines +/- | |
|---|---|---|---|---|
| wdqs: add bking as contact for wdqs alerts | operations/puppet | production | +1 -1 |
Action item from today's meeting: reach out to Observability for recommendations on how to handle this.
Per @EBernhardson comment at today's triage meeting, the secondary DC (usually CODFW) will start to receive Elastic read traffic. Had this been the case during the issue mentioned above, it would have had user impact. So this increases the priority of this ticket.
Change 824553 had a related patch set uploaded (by Bking; author: Bking):
[operations/puppet@production] wdqs: add bking as contact for wdqs alerts
Change 824553 merged by Bking:
[operations/puppet@production] wdqs: add bking as contact for wdqs alerts
Tying this in to recent incident , as our monitoring did not catch a user-impacting autocomplete issue.
Suggestion from @EBernhardson : "random guesses at what we need, the search reindex process looks at the old index and the new index, and if the counts are way off it declares failure. Here we build an index with 25k docs instead of 9.5M and the automation said 'that looks fine, promote it'" (Sanity checking during reindex process).