| Eevans | |
| Dec 17 2015, 7:49 PM |
| F4711223: xenon_aggregation.png | |
| Nov 9 2016, 11:23 PM |
| F3113127: 9.png | |
| Dec 17 2015, 7:49 PM |
| F3113131: Grafana - RESTBase :: Cassandra :: System 2015-12-17 13-42-17.png | |
| Dec 17 2015, 7:49 PM |
| F3113129: Grafana - RESTBase :: Cassandra :: System 2015-12-17 13-42-03.png | |
| Dec 17 2015, 7:49 PM |
As seen on the Cassandra storage load dashboard:
Viewing the last 7 days, the values seem correct:
Viewing the last 30 days, the values jump by an order of magnitude:
Using the same query string in Graphite directly (aliasByNode(cassandra.restbase1*.org.apache.cassandra.metrics.Storage.Load.count, 1)):
| Subject | Repo | Branch | Lines +/- | |
|---|---|---|---|---|
| graphite: change Cassandra '.count' metrics aggregation | operations/puppet | production | +16 -1 |
I think that has to do on how we aggregated .count metrics
[count] aggregationMethod = sum pattern = \.count$ xFilesFactor = 0.01
and the retention
[default] pattern = .* retentions = 1m:7d,5m:14d,15m:30d,1h:1y
Yes. This would make sense. Do you remember why we aggregated these like this? The \.count$ metrics in Cassandra should be treated as a gauge, I think, and aggregated by-average instead.
I think it comes from statsd recommendation on how to aggregate graphite metrics (https://github.com/etsy/statsd/blob/master/docs/graphite.md). Reading the carbon code it seems the schemas are a list, so we could match the cassandra specific .count metrics first and aggregate on average instead
If this is really a gauge, should the cassandra metric reporter perhaps report it as such?
graphite has no metric type per-se so everything is a gauge, though the .count naming convention to indicate counters is statsd's not graphite's
@fgiunchedi and I discussed this; It would be trivial to do.
The downsides are that it would be the first breaking change to metric name compatibility with the in-built Cassandra reporter (other than the histograms that were unusable anyway). It would also break all of the effected dashboards (easy enough to fix I guess, just tedious). And, it seems that the metric reporter has other users now, so it would similarly break things for them.
I suggested that if we do go this route, that we implement it as a generic name rewriting feature (which would still be pretty trivial to implement). That only addresses the latter of the above concerns though.
Taking this, I'll be changing graphite's storage_aggregation to account for cassandra's count metrics
Change 316810 had a related patch set uploaded (by Filippo Giunchedi):
graphite: change Cassandra '.count' metrics aggregation
Change 316810 merged by Filippo Giunchedi:
graphite: change Cassandra '.count' metrics aggregation
I've updated xenon and cerium aggregation methods for all .count metrics, @Eevans let's compare those in a week or so
Looks like it is working on xenon: https://graphite.wikimedia.org/render/?width=586&height=294&_salt=1478732584.357&from=1477869605&to=1478733609&target=aliasByNode(cassandra.xenon*.org.apache.cassandra.metrics.Storage.Load.count%2C%201)
I'll convert the rest
Mentioned in SAL (#wikimedia-operations) [2016-11-09T23:38:37Z] <godog> update cassandra aggregation scheme for 'count' metrics - T121789