Page MenuHomePhabricator

StatsD Exporter drops relayed metrics
Closed, DeclinedPublic

Description

Around half of the metrics are being relayed.

udpreplay direct to socat gets around 26k metrics writing directly.

$ touch direct.txt && socat -t 0 FILE:direct.txt udp-listen:9125,fork
$ ./udpreplay --pps 2000 --host 127.0.0.1 --port 9125 ores1001.pcap
$ cat direct.txt | sed 's/ores.ores1001/\nores.ores1001/g' | sed '/^$/d' | wc -l
26412

The current version of statsd_exporter (0.9.0+ds1-1), when put into the middle, records around 13k metrics.

$ touch forwarded_current.txt && socat -t 0 FILE:forwarded_current.txt udp-listen:9125,fork
$ ./statsd_exporter_0.9.0+ds1-1 --statsd.mapping-config=statsd_exporter.conf --statsd.listen-udp=:8125 --statsd.relay-address=127.0.0.1:9125
$ ./udpreplay --pps 2000 --host localhost --port 8125 ores1001.pcap
$ cat forwarded_current.txt | sed 's/ores.ores1001/\nores.ores1001/g' | sed '/^$/d' | wc -l
13812

Event Timeline

Change 554544 had a related patch set uploaded (by Cwhite; owner: Cwhite):
[operations/debs/prometheus-statsd-exporter@master] when configured to relay statsd traffic, send the raw []byte recieved toward the configured statsd endpoint

https://gerrit.wikimedia.org/r/554544

colewhite renamed this task from StatsD Exporter does not relay dropped metrics to StatsD Exporter drops relayed metrics.Jan 17 2020, 11:43 PM
colewhite updated the task description. (Show Details)

The latest patch appears to help a lot. There is still a discrepancy that I haven't been able to track down.

$ touch forwarded_new.txt && socat -t 0 FILE:forwarded_new.txt udp-listen:9125,fork
$ ./statsd_exporter_gerrit_554544 --statsd.mapping-config=statsd_exporter.conf --statsd.listen-udp=:8125 --statsd.relay-address=127.0.0.1:9125
$ ./udpreplay --pps 2000 --host localhost --port 8125 ores1001.pcap

$ cat forwarded_new.txt | sed 's/ores.ores1001/\nores.ores1001/g' | sed '/^$/d' | wc -l
25862

Still a problem, but probably not big enough to warrant the effort.

Change 554544 abandoned by Cwhite:

[operations/debs/prometheus-statsd-exporter@master] when configured to relay statsd traffic, send the raw []byte recieved toward the configured statsd endpoint

Reason:

https://gerrit.wikimedia.org/r/554544