The generic p95 alert has not been very helpful;
it can fire for a number of different reasons
and does not always map directly to user impact.
Replace generic p95 with a more targeted focus
on the 3 query types most relevant to users:
full_text, comp_suggest, and more_like
| Subject | Author | Repo | Branch | Lines +/- | |
|---|---|---|---|---|---|
| elastic: make request latency alerts more granular | Ryan Kemper | operations/puppet | production | +58 -8 |
Change 967281 had a related patch set uploaded (by Ryan Kemper; author: Ryan Kemper):
[operations/puppet@production] elastic: make request latency alerts more granular
Change 967281 merged by Ryan Kemper:
[operations/puppet@production] elastic: make request latency alerts more granular
Alert changes have been merged so sticking in blocked/waiting for now. We should check back in a week and make sure the alerts work as intended.
Here's a graph of some of the alerts firing over time. We do see occasional alerts for these new alerts, but it seems less frequent (and therefore more granular) than with the previous alerting strategy (alerting on generic p95 request time rather than breaking it down per query type)