| fgiunchedi | |
| Dec 18 2015, 2:16 PM |
| F3691215: 2016-03-22-143249_688x335_scrot.png | |
| Mar 22 2016, 2:33 PM |
| F3309295: screenshot_KcBLEa.png | |
| Feb 4 2016, 9:41 AM |
for historical reasons diamond sends its samples using statsd every 60s, this matches statsd flush interval. furthermore we're mostly using gauges so none of statsd aggregation capabilities, switching to graphite will relieve some pressure on statsd and possibly reduce diamond metrics lag too.
| Status | Subtype | Assigned | Task | ||
|---|---|---|---|---|---|
| Resolved | fgiunchedi | T105218 check_graphite - "UNKNOWN: More than half of the datapoints are undefined " | |||
| Resolved | fgiunchedi | T101141 UDP rcvbuferrors and inerrors on graphite hosts | |||
| Resolved | fgiunchedi | T121861 switch diamond to use graphite line protocol |
on second thought we should be fine with collectors that send counters too, what happens is that graphite will store the number as is and there shouldn't be a change in semantics since diamond pushes once a minute and statsite also flushes once a minute
I've temporarily switched diamond to graphite on filippo-test-trusty.eqiad.wmflabs to see the net effect it has (if any) on the metrics
the iowait spike drop is puppet being stopped since it would overwrite my diamond changes
Change 268360 had a related patch set uploaded (by Filippo Giunchedi):
diamond: send labs instance metrics via graphite/carbon
Change 268360 merged by Filippo Giunchedi:
diamond: send labs instance metrics via graphite/carbon
merged this now for labs instances, metrics from tcp collector will change type as counters don't get derived metrics like in statsd
309088 Mar 22 14:24 /srv/carbon/whisper/toolserver-legacy/relic/tcp/ListenOverflows.wsp 309088 Mar 22 12:27 /srv/carbon/whisper/toolserver-legacy/relic/tcp/ListenOverflows/lower.wsp 309088 Mar 22 12:29 /srv/carbon/whisper/toolserver-legacy/relic/tcp/ListenOverflows/sum.wsp 309088 Mar 22 12:26 /srv/carbon/whisper/toolserver-legacy/relic/tcp/ListenOverflows/count.wsp 309088 Mar 22 12:30 /srv/carbon/whisper/toolserver-legacy/relic/tcp/ListenOverflows/rate.wsp 309088 Mar 22 12:27 /srv/carbon/whisper/toolserver-legacy/relic/tcp/ListenOverflows/upper.wsp 309088 Mar 22 12:34 /srv/carbon/whisper/toolserver-legacy/relic/tcp/ListenOverflows/mean.wsp
also corresponding drop in udp incoming datagrams on labmon1001
Change 281622 had a related patch set uploaded (by Filippo Giunchedi):
diamond: send production traffic via graphite line protocol
Change 281622 merged by Filippo Giunchedi:
diamond: send production traffic via graphite line protocol
this has been deployed, though the impact on statsd seems to have been minimal in terms of udp packets/drops, though the move also implied moving to TCP
reopening as there seem to be missing ACLs from hosts with public IPs towards graphite1001:2003 tcp (e.g. carbon)
Change 282706 had a related patch set uploaded (by Filippo Giunchedi):
graphite: permit line protocol traffic from ALL_NETWORKS
Change 282706 merged by Filippo Giunchedi:
graphite: permit line protocol traffic from ALL_NETWORKS