Page MenuHomePhabricator

diamond: certain counters always calculated as 0
Closed, ResolvedPublic

Description

After enabling a bunch of TFO-related metrics in diamond's TCP collector we've noticed that the values sent to graphite are not computed correctly. In particular, diamond seems to always return 0 for counters such as TCPFastOpenPassive. Albeit slowly, the value of TCPFastOpenPassive is indeed increasing on the server (see netstat -s | grep TCPFastOpenPassive) so the derivative should not always be 0.

A similar bug has been reported upstream but the submitter simply worked around the issue by sending the raw metric.

We should find out whether Collector.derivative() is buggy when it comes to "slowly increasing" metrics.

Event Timeline

Change 296380 had a related patch set uploaded (by Ema):
Add 0009-tcp-collector-specify-gauges.patch

https://gerrit.wikimedia.org/r/296380

Change 296380 merged by Ema:
Add 0009-tcp-collector-specify-gauges.patch

https://gerrit.wikimedia.org/r/296380

Mentioned in SAL [2016-06-29T09:20:20Z] <ema> upgrading diamond to 3.5-6 (T138758)

ema triaged this task as Medium priority.Jun 29 2016, 11:25 AM

@ema: afaiu we just patched the TCP collector but we still need to investigate the overall diamond behavior?

@elukey : that's right, we're simply sending gauges instead of counters but the behavior of Collector.derivative() still needs to be investigated.

fgiunchedi claimed this task.

We've upgraded to diamond 4 in T97635: Update diamond to latest upstream version and its TCP collector includes gauges config option, resolving.