Page MenuHomePhabricator

Remove logster from cp* hosts
Closed, ResolvedPublic

Description

Logster runs via cron on cp* hosts and exports varnishkafka metrics via statsd to graphite at varnishkafka.<host>.(webrequest|statsv|eventlogging). This functionality has been superseded by prometheus-varnishkafka-exporter.

The goal of this task is twofold:

  1. Validate the metrics generated by logster are no longer in use
  2. Remove logster from all PoPs.

Event Timeline

The Analytics team cares about two things:

  1. https://grafana.wikimedia.org/d/000000253/varnishkafka?orgId=1
  2. the varnishkafka alerts in puppet

Hi @colewhite. Can you please associate at least one project with this task (via the Add Action...Change Project Tags dropdown). This will allow others to get notified and see this task when looking at the corresponding project workboard. Thanks!

Change 526611 had a related patch set uploaded (by Elukey; owner: Elukey):
[operations/puppet@production] profile::cache::kafka::alerts: move alarms to prometheus

https://gerrit.wikimedia.org/r/526611

https://grafana.wikimedia.org/d/000000253/varnishkafka is now prometheus based, meanwhile the old one is now at https://grafana.wikimedia.org/d/JzhtS4vWz/varnishkafka-graphite. The metrics looks good, the number per hosts are consistent but of course the aggregate it is not (since in graphite I was used to aggregate all the metrics from the pops, meanwhile in the prometheus dashboard I have aggregation per-dc). When the alerts are migrated I'd wait a couple of days to check metrics etc.. and then we'll be able to proceed in removing logster :)

elukey triaged this task as Medium priority.Jul 31 2019, 7:50 AM

Change 526611 merged by Elukey:
[operations/puppet@production] profile::cache::kafka::alerts: move alarms to prometheus

https://gerrit.wikimedia.org/r/526611

Change 528636 had a related patch set uploaded (by Elukey; owner: Elukey):
[operations/puppet@production] profile::cache::kafka::varnish_kafka_delivery_alerts: fix query

https://gerrit.wikimedia.org/r/528636

Change 528636 merged by Elukey:
[operations/puppet@production] profile::cache::kafka::varnish_kafka_delivery_alerts: fix query

https://gerrit.wikimedia.org/r/528636

Change 528638 had a related patch set uploaded (by Elukey; owner: Elukey):
[operations/puppet@production] profile::cache::kafka::varnishkafka_delivery_alerts: fix dashboard link

https://gerrit.wikimedia.org/r/528638

Change 528638 merged by Elukey:
[operations/puppet@production] profile::cache::kafka::varnishkafka_delivery_alerts: fix dashboard link

https://gerrit.wikimedia.org/r/528638

From my point of view logster etc.. on the cp hosts can be removed!

Change 529399 had a related patch set uploaded (by Cwhite; owner: Cwhite):
[operations/puppet@production] logster: add ensure parameter

https://gerrit.wikimedia.org/r/529399

Change 529399 merged by Cwhite:
[operations/puppet@production] logster: add ensure parameter

https://gerrit.wikimedia.org/r/529399

Change 531730 had a related patch set uploaded (by Cwhite; owner: Cwhite):
[operations/puppet@production] profile, varnishkafka: remove logster cron entries from varnishkafka hosts

https://gerrit.wikimedia.org/r/531730

Change 531730 merged by Cwhite:
[operations/puppet@production] profile, varnishkafka: remove logster cron entries from varnishkafka hosts

https://gerrit.wikimedia.org/r/531730