Page MenuHomePhabricator

StatsD Exporter drops relayed metrics
Open, MediumPublic

Description

Around half of the metrics are being relayed.

udpreplay direct to socat gets around 26k metrics writing directly.

$ touch direct.txt && socat -t 0 FILE:direct.txt udp-listen:9125,fork
$ ./udpreplay --pps 2000 --host 127.0.0.1 --port 9125 ores1001.pcap
$ cat direct.txt | sed 's/ores.ores1001/\nores.ores1001/g' | sed '/^$/d' | wc -l
26412

The current version of statsd_exporter (0.9.0+ds1-1), when put into the middle, records around 13k metrics.

$ touch forwarded_current.txt && socat -t 0 FILE:forwarded_current.txt udp-listen:9125,fork
$ ./statsd_exporter_0.9.0+ds1-1 --statsd.mapping-config=statsd_exporter.conf --statsd.listen-udp=:8125 --statsd.relay-address=127.0.0.1:9125
$ ./udpreplay --pps 2000 --host localhost --port 8125 ores1001.pcap
$ cat forwarded_current.txt | sed 's/ores.ores1001/\nores.ores1001/g' | sed '/^$/d' | wc -l
13812

Details

Related Gerrit Patches:

Event Timeline

colewhite created this task.Dec 4 2019, 4:04 PM
Restricted Application added a subscriber: Aklapper. · View Herald TranscriptDec 4 2019, 4:04 PM

Change 554544 had a related patch set uploaded (by Cwhite; owner: Cwhite):
[operations/debs/prometheus-statsd-exporter@master] when configured to relay statsd traffic, send the raw []byte recieved toward the configured statsd endpoint

https://gerrit.wikimedia.org/r/554544

colewhite triaged this task as Medium priority.Dec 4 2019, 8:17 PM
fgiunchedi moved this task from Inbox to In progress on the observability board.Dec 9 2019, 11:07 AM
colewhite renamed this task from StatsD Exporter does not relay dropped metrics to StatsD Exporter drops relayed metrics.Jan 17 2020, 11:43 PM
colewhite updated the task description. (Show Details)
colewhite updated the task description. (Show Details)Jan 18 2020, 12:11 AM
colewhite added a comment.EditedJan 18 2020, 12:14 AM

The latest patch appears to help a lot. There is still a discrepancy that I haven't been able to track down.

$ touch forwarded_new.txt && socat -t 0 FILE:forwarded_new.txt udp-listen:9125,fork
$ ./statsd_exporter_gerrit_554544 --statsd.mapping-config=statsd_exporter.conf --statsd.listen-udp=:8125 --statsd.relay-address=127.0.0.1:9125
$ ./udpreplay --pps 2000 --host localhost --port 8125 ores1001.pcap

$ cat forwarded_new.txt | sed 's/ores.ores1001/\nores.ores1001/g' | sed '/^$/d' | wc -l
25862