Page MenuHomePhabricator

varnishreqstats sends truncated statsd traffic
Closed, ResolvedPublic

Description

Discovered while working on T205870: Fully migrate producers off statsd, looks like varnishreqstats sends truncated statsd traffic:

root@graphite1004:~# ngrep -q -W byline . udp dst port 8125  | grep -v -e '^U ' -e '^$' | grep -v \|
interface: eno1 (10.64.16.0/255.255.252.0)
filter: (ip or ip6) and ( udp dst port 8125 )
match: .
varnish.ulsf
varnish.uls
varnish.ulsfo.upload.frontend.request
varnish.eqiad.upload.frontend.requ
varnish.codfw.upload.frontend.request.backend
varnish.eqsin.upload.frontend.request.b
varnish.eqsin.upload.frontend.request.bac
varnish.esams.upload.frontend.req
varnish.eqiad.upload.frontend.request
varnish.esams.upload.frontend.re
varnish.eqsin.upload.frontend.request.bac
varnish.codfw.upload.frontend.request.client.s
varnish.ulsfo.upload.frontend.request.clie
varnish.esams.upload.frontend.
varnish.e
varnish.eqsin.upload.frontend.request.bac
varni
varnis
varnish.codfw.text.fron
varnish.esams.upload.frontend.re
varnish.esams.upload.frontend.re
varnish.esams.upload.frontend.req
varnish.esams.upload.frontend.requ

From a cursory look at the code it seems to me CacheStatsSender doesn't take into account udp MTU to break up metrics and thus under high traffic partial udp packets are sent out. OTOH varnishstatsd doesn't seem affected as it does take into account MTU.

Event Timeline

Restricted Application added a project: SRE. · View Herald TranscriptDec 19 2018, 4:02 PM
Restricted Application added a subscriber: Aklapper. · View Herald Transcript
ema moved this task from Triage to Caching on the Traffic board.Dec 19 2018, 4:12 PM
herron triaged this task as Medium priority.Jan 2 2019, 9:05 PM
fgiunchedi closed this task as Resolved.Sep 5 2019, 7:35 AM
fgiunchedi claimed this task.

Varnish doesn't send statsd any longer per T220116: Migrate all metrics originated by PoPs from statsd to Prometheus, resolving