I want to move this monitoring into graphite via logster. First step, make varnishkafka instance naming consistent:

https://gerrit.wikimedia.org/r/#/c/177546/.

Merging that and then running this:

salt -E 'cp.*|amssq.*' cmd.run '
remove_bad_name_varnishkafka() {
    name=$1
    echo "Removing $name varnishkafka instance."
    service varnishkafka-$name stop &&
    update-rc.d -f varnishkafka-$name remove && 
    rm -fv /etc/ganglia/conf.d/varnishkafka-$name.pyconf /etc/varnishkafka/$name.conf;
}
puppet agent -t &&
test -f /etc/init/varnishkafka-frontend.conf && remove_bad_name_varnishkafka frontend
test -f /etc/init/varnishkafka-$(hostname).conf && remove_bad_name_varnishkafka $(hostname)
'

since this is going to be many metrics and frequently updated we should consider running a local statsd (txstatsd in our case for now) which in turn will consolidate and push to graphite. this for example has been done for swift statsd to reduce the statsd traffic. of course the approach works only if statsd traffic can be aggregated after being processed by statsd (e.g. if the metrics are machine-specific)

I misunderstood "many metrics", what I meant with the previous suggestion applied only if the statsd traffic was going to be reduced by a local txstatsd, this doesn't seem to be the case however.

note however that a local txstatsd is going to be more efficient anyways because it'll batch metrics and push them to graphite via pickle

I've abandoned the previous change in favor of this one:

https://gerrit.wikimedia.org/r/#/c/182072/

I turned on a check_graphite check for this today. I'm going to let this be for a few days, and then hopefully turn off the check_ganglia ones next week.

reopen for https://icinga.wikimedia.org/cgi-bin/icinga/status.cgi?host=all&type=detail&servicestatustypes=8&hoststatustypes=3&serviceprops=2097162&nostatusheader .. or new bug?