Reference to ticket #7662 added by dzahn

Reference to ticket #7779 added by dzahn

That seems a bit more sensitive then it ought to be.....

Status changed from 'new' to 'open' by RT_System

What should we change it to?
https://github.com/wikimedia/operations-puppet/blob/production/manifests/role/logging.pp#L117

On Fri Sep 12 17:16:34 2014, aotto wrote:

What should we change it to?

https://github.com/wikimedia/operations-
puppet/blob/production/manifests/role/logging.pp#L117

Hmmm - maybe .01 for warn and .1 for critical. Right now it'll complain if we
have a couple of slow queries a minute.

Change 205603 had a related patch set uploaded (by Filippo Giunchedi):
logging: update CirrusSearch thresholds

https://gerrit.wikimedia.org/r/205603

Change 205603 merged by Filippo Giunchedi:
logging: update CirrusSearch thresholds

https://gerrit.wikimedia.org/r/205603

SOFT states do not send any notifications though. So it would seem ok to me to call it resolved.

If we want to avoid even seeing them in logs/history, there is " log_service_retries=1" but not sure if we can apply that on just this one service. It uses monitoring::ganglia and that doesn't have a parameter for it.

yeah that's true, I think the confusion comes from the fact that notifications are not sent but the alarm still shows when looking for critical/warning/unknown alerts in icinga's default view (both hard and soft states that is)
it'd be better for the alarm not to go into soft state at all, I think the graphite check for example can use "consider only this window of time" so maybe that's another solution too

I've pulled this into the Discovery team's backlog so it ranked against other stuff we need to do. I've moved it pretty close to the top.

Stakeholders: Operations/Cirrus Operators
Benefits: This alert is currently useless - its just ignored. Its supposed to warn us when we get too many slow queries. And maybe it is and we're just ignoring it - or maybe its over cry-wolfy. We need to understand how it works so we can decide.
Estimate: A couple of days - but this is a rough estimate.

Any news?

None. It's still on the list but the team has been concentrating on other
things that have yet to finish. If you want a quick fix I'll +1 disabling
this check and leaving this ticket to discuss reenabling it.

In T84163#1387421, @Manybubbles wrote:

None. It's still on the list but the team has been concentrating on other
things that have yet to finish. If you want a quick fix I'll +1 disabling
this check and leaving this ticket to discuss reenabling it.

I scheduled a downtime for this service of 1 month with a link to this ticket.

That means it won't send notifications (mail, IRC) but is not completely disabled and in one month it will become active again as a reminder.

I note this ticket has been placed "up for grabs" though.

I didn't realize this has had a task already. I talked with @EBernhardson about it a bit and there may be some foreshadowing of issues to come.

ebernhardson
i think it might be a sign of worse things to come. choosing one of the days with more failures (aug 29) ~1% of > queries were logged as slow
ebernhardson
while we are alerting at .004% :)
chasemp
that seems significant then

In T84163#1647247, @Dzahn wrote:

I scheduled a downtime for this service of 1 month with a link to this ticket.

Since that's over it's back as a WARNING for now:

https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=fluorine&service=Slow+CirrusSearch+query+rate

poked erik a bit who will chime in, sounds like it will sit for awhile and is not a fire

I've gone back to review the size of the logs over the last month, it looks to have settled down from some of the worst cases back on aug 29. Figuring out what constitues a problem and what is just normal is very hard to determine, as we intentionally support regex queries that take up to 20s to execute (if that's a good idea or not might be for another ticket).

I'd like to be able to pass on the monitoring portion of this as one of the first tasks for the discovery teams new ops engineer. We haven't made a hire yet but the process is moving along. I think a task like this will be a good way to get them involved and looked around the elasticsearch cluster and understanding what is going on with our systems.

Change 251948 had a related patch set uploaded (by Faidon Liambotis):
Kill CirrusSearch-slow-queries alert

https://gerrit.wikimedia.org/r/251948

Change 251948 merged by Faidon Liambotis:
Kill CirrusSearch-slow-queries alert

https://gerrit.wikimedia.org/r/251948