Page MenuHomePhabricator

kartotherian sends javascript instead of statsd metric name
Closed, ResolvedPublic

Description

Discovered while working on T205870: Fully migrate producers off statsd, looks like karthoterian itself sends javascript function on statsd.eqiad.wmnet 8125/udp:

root@graphite1004:~# ngrep -q -W byline . udp dst port 8125  | grep -B20 -A20 kartotherian.function
...
U 10.64.0.79:45737 -> 10.64.16.149:8125
kartotherian.gc.minor:11353700|ms
kartotherian.req.osm-intl.15.png:68|ms
kartotherian.req.osm-intl.16.png:44|ms
kartotherian.function (metrics) {
    this.metrics = metrics;
    return this;
}:1|c

Event Timeline

Restricted Application added a subscriber: Aklapper. · View Herald TranscriptDec 19 2018, 2:20 PM
LGoto triaged this task as Low priority.Dec 19 2018, 4:42 PM
LGoto moved this task from Needs triage to Backlog on the Product-Infrastructure-Team-Backlog board.
Mholloway edited projects, added Maps (Kartotherian); removed Maps.
MSantos raised the priority of this task from Low to High.Jan 8 2019, 4:17 PM
MSantos added a comment.EditedMar 29 2019, 9:34 PM

@fgiunchedi is it possible to know the exact time that metric was registered? I don't have access to the graphite cluster so I couldn't perform any tests myself.

@fgiunchedi is it possible to know the exact time that metric was registered? I don't have access to the graphite cluster so I couldn't perform any tests myself.

The metric is sent periodically, e.g. this is from a capture today (the non printable characters at the beginning are the packet headers, can be ignored)

09:04:49.178699 IP maps1002.eqiad.wmnet.38512 > graphite1004.eqiad.wmnet.8125: UDP, length 173
E.....@.@...
@.*
@...p....`.kartotherian.function (metrics) {
    this.metrics = metrics;
    return this;
}:1|c
kartotherian.req.osm-intl.15.png:44|ms
kartotherian.req.osm-intl.11.png.static.1:273|ms
MSantos claimed this task.Apr 12 2019, 4:38 PM
MSantos moved this task from Doing to To Do on the Product-Infrastructure-Team-Backlog (Kanban) board.

Change 525841 had a related patch set uploaded (by MSantos; owner: MSantos):
[mediawiki/services/kartotherian@master] Fix statsd metrics

https://gerrit.wikimedia.org/r/525841

Change 525841 merged by MSantos:
[mediawiki/services/kartotherian@master] Fix statsd metrics

https://gerrit.wikimedia.org/r/525841

Thanks @MSantos ! Definitely seeing less of that type of traffic but still some from karthoterian:

U 10.64.16.42:33395 -> 10.64.16.149:8125
kartotherian.req.osm-intl.18.png:198|ms
kartotherian.function(metrics) {
    this.metrics = metrics;
    return this;
}:1|c


U 10.64.16.42:35554 -> 10.64.16.149:8125
kartotherian.req.osm-intl.18.png:119|ms
kartotherian.function(metrics) {
    this.metrics = metrics;
    return this;
}:1|c
kartotherian.req.osm-intl.12.png:39|ms

@fgiunchedi the final fix was deployed a few minutes ago. Please let me know if you run into any issue.

fgiunchedi closed this task as Resolved.Sep 16 2019, 12:20 PM

Thanks @MSantos I can confirm all metrics are valid now! Thanks for your help!