Page MenuHomePhabricator

icinga strips single quotes from metric names for check_graphite
Closed, ResolvedPublic

Description

newly deployed check in https://gerrit.wikimedia.org/r/251526 results in graphite 500 error:

$ curl 'http://graphite.wikimedia.org/render?format=json&from=-5h&target=summarize%28nonNegativeDerivative%28keepLastValue%28swift.eqiad-prod.containers.mw-media.originals.objects%29%29%2C%201h%29'
<body style="background-color: #666666; color: black;">
<center>
<h2 style='font-family: "Arial"'>
<p>Graphite encountered an unexpected error while handling your request.</p>
<p>Please contact your site administrator if the problem persists.</p>
</h2>
<br/>
<div style="width: 50%; text-align: center; font-family: monospace; background-color: black; font-weight: bold; color: #ff4422;">

</div>

<div style="width: 70%; text-align: left; background-color: black; color: #44ff22; border: thin solid gray;">
<pre>
Traceback (most recent call last):
  File &quot;/usr/lib/python2.7/dist-packages/django/core/handlers/base.py&quot;, line 112, in get_response
    response = wrapped_callback(request, *callback_args, **callback_kwargs)
  File &quot;/usr/lib/python2.7/dist-packages/graphite/render/views.py&quot;, line 122, in renderView
    seriesList = evaluateTarget(requestContext, target)
  File &quot;/usr/lib/python2.7/dist-packages/graphite/render/evaluator.py&quot;, line 10, in evaluateTarget
    result = evaluateTokens(requestContext, tokens)
  File &quot;/usr/lib/python2.7/dist-packages/graphite/render/evaluator.py&quot;, line 21, in evaluateTokens
    return evaluateTokens(requestContext, tokens.expression)
  File &quot;/usr/lib/python2.7/dist-packages/graphite/render/evaluator.py&quot;, line 29, in evaluateTokens
    return func(requestContext, *args)
  File &quot;/usr/lib/python2.7/dist-packages/graphite/render/functions.py&quot;, line 2510, in summarize
    bucketInterval = timestamp - (timestamp % interval)
ZeroDivisionError: integer division or modulo by zero

</pre>
</div>

the definition is

check_command                  check_graphite_threshold!http://graphite.wikimedia.org!10!summarize(nonNegativeDerivative(keepLastValue(swift.eqiad-prod.containers.mw-media.originals.objects)), '1h')!2000!3000!5h!1!--over

it appears to be a quoting problem, when plugging that query into graphite explorer and fetching it via json format it seems to work

$ curl 'http://graphite.wikimedia.org/render?format=json&from=-5h&target=summarize(nonNegativeDerivative(keepLastValue(swift.eqiad-prod.containers.mw-media.originals.objects)),%20%271h%27)'
[{"target": "summarize(nonNegativeDerivative(keepLastValue(swift.eqiad-prod.containers.mw-media.originals.objects)), \"1h\", \"sum\")", "datapoints": [[null, 1447236000], [400.0, 1447239600], [600.0, 1447243200], [600.0, 1447246800], [600.0, 1447250400], [500.0, 1447254000]]}]

Event Timeline

fgiunchedi claimed this task.
fgiunchedi raised the priority of this task from to Medium.
fgiunchedi updated the task description. (Show Details)
fgiunchedi added projects: SRE, Grafana.
fgiunchedi subscribed.

I stand corrected, it seems icinga is silently stripping single quotes when exec-ing the command

[pid 12449] execve("/usr/lib/nagios/plugins/check_graphite", ["/usr/lib/nagios/plugins/check_graphite", "-U", "http://graphite.wikimedia.org", "-T", "10", "check_threshold", "summarize(nonNegativeDerivative(keepLastValue(swift.eqiad-prod.containers.mw-media.originals.objects)), 1h)", "-W", "2000", "-C", "3000", "--from", "5h", "--perc", "1", "--over"], [/* 4 vars */]) = 0

Change 252955 had a related patch set uploaded (by Filippo Giunchedi):
swift: use double quote in graphite query

https://gerrit.wikimedia.org/r/252955

Change 252955 merged by Filippo Giunchedi:
swift: use double quote in graphite query

https://gerrit.wikimedia.org/r/252955

Change 252963 had a related patch set uploaded (by Filippo Giunchedi):
monitoring: fail on graphite metrics using single quotes

https://gerrit.wikimedia.org/r/252963

fgiunchedi renamed this task from http 500 errors from check_graphite on rate of media upload checks to icinga strips single quotes from metric names for check_graphite.Nov 16 2015, 2:30 PM
fgiunchedi set Security to None.

Change 252963 merged by Filippo Giunchedi:
monitoring: fail on graphite metrics using single quotes

https://gerrit.wikimedia.org/r/252963

not really fixed per se, but puppet will barf now