Page MenuHomePhabricator

StatsD Exporter drops relayed metrics
Closed, DeclinedPublic


Around half of the metrics are being relayed.

udpreplay direct to socat gets around 26k metrics writing directly.

$ touch direct.txt && socat -t 0 FILE:direct.txt udp-listen:9125,fork
$ ./udpreplay --pps 2000 --host --port 9125 ores1001.pcap
$ cat direct.txt | sed 's/ores.ores1001/\nores.ores1001/g' | sed '/^$/d' | wc -l

The current version of statsd_exporter (0.9.0+ds1-1), when put into the middle, records around 13k metrics.

$ touch forwarded_current.txt && socat -t 0 FILE:forwarded_current.txt udp-listen:9125,fork
$ ./statsd_exporter_0.9.0+ds1-1 --statsd.mapping-config=statsd_exporter.conf --statsd.listen-udp=:8125 --statsd.relay-address=
$ ./udpreplay --pps 2000 --host localhost --port 8125 ores1001.pcap
$ cat forwarded_current.txt | sed 's/ores.ores1001/\nores.ores1001/g' | sed '/^$/d' | wc -l

Event Timeline

colewhite created this task.Dec 4 2019, 4:04 PM
Restricted Application added a subscriber: Aklapper. · View Herald TranscriptDec 4 2019, 4:04 PM

Change 554544 had a related patch set uploaded (by Cwhite; owner: Cwhite):
[operations/debs/prometheus-statsd-exporter@master] when configured to relay statsd traffic, send the raw []byte recieved toward the configured statsd endpoint

colewhite triaged this task as Medium priority.Dec 4 2019, 8:17 PM
fgiunchedi moved this task from Inbox to In progress on the observability board.Dec 9 2019, 11:07 AM
colewhite renamed this task from StatsD Exporter does not relay dropped metrics to StatsD Exporter drops relayed metrics.Jan 17 2020, 11:43 PM
colewhite updated the task description. (Show Details)
colewhite updated the task description. (Show Details)Jan 18 2020, 12:11 AM
colewhite added a comment.EditedJan 18 2020, 12:14 AM

The latest patch appears to help a lot. There is still a discrepancy that I haven't been able to track down.

$ touch forwarded_new.txt && socat -t 0 FILE:forwarded_new.txt udp-listen:9125,fork
$ ./statsd_exporter_gerrit_554544 --statsd.mapping-config=statsd_exporter.conf --statsd.listen-udp=:8125 --statsd.relay-address=
$ ./udpreplay --pps 2000 --host localhost --port 8125 ores1001.pcap

$ cat forwarded_new.txt | sed 's/ores.ores1001/\nores.ores1001/g' | sed '/^$/d' | wc -l
colewhite closed this task as Declined.Jun 8 2020, 3:13 PM

Still a problem, but probably not big enough to warrant the effort.