Page MenuHomePhabricator

Deprovision Diamond collectors no longer in use
Closed, ResolvedPublic

Description

With T177196: Port non-deprecated Diamond collectors to Prometheus we've deprecated several Diamond collectors now, we can thus proceed to deprovision / decom the ones that are no longer in use. Time wise this should happen in 10-12 weeks to give Prometheus enough time to collect data.

For hosts running only basic collectors (e.g. system stats already replaced by node-exporter) we should consider removing Diamond altogether.

Collectors to deprecate:

  • memcached modules/memcached/manifests/init.pp
  • nginx: modules/debug_proxy/manifests/init.pp modules/dynamicproxy/manifests/init.pp modules/profile/manifests/docker/registry.pp modules/puppetmaster/manifests/puppetdb.pp
  • ircdstats: modules/mw_rc_irc/manifests/ircserver.pp
  • nutcracker: modules/nutcracker/manifests/monitoring.pp
  • apachestatussimple: modules/phabricator/manifests/monitoring.pp
  • etherpad: modules/profile/manifests/etherpad.pp
  • kubernetes: modules/profile/manifests/kubernetes/master.pp
  • redis: modules/profile/manifests/redis/master.pp modules/profile/manifests/redis/multidc.ppmodules/profile/manifests/redis/slave.pp
  • udpcollector: modules/profile/manifests/statsd.pp
  • pybal: modules/pybal/manifests/monitoring.pp
  • rabbitmq: modules/rabbitmq/manifests/monitor.pp
  • openldap/corp: modules/role/manifests/openldap/corp.pp
  • openldap/labs:modules/role/manifests/openldap/labs.pp
  • ntpd: modules/ntp/manifests/daemon.pp
  • nfsdcollector: modules/labstore/manifests/monitoring/nfsd.pp
  • DirectorySize: modules/role/manifests/labs/nfs/secondary.pp

Diamond in Toolforge is tracked via T210991

Diamond in Cloud VPS is tracked via T210993

Details

Related Changes in Gerrit:
SubjectRepoBranchLines +/-
operations/puppetproduction+11 -15
operations/puppetproduction+4 -17
operations/puppetproduction+2 -0
operations/puppetproduction+26 -2
operations/puppetproduction+32 -0
operations/puppetproduction+2 -0
operations/puppetproduction+0 -140
operations/puppetproduction+1 -0
operations/puppetproduction+3 -0
operations/puppetproduction+0 -3
operations/puppetproduction+1 -3
operations/puppetproduction+0 -3
operations/puppetproduction+2 -0
operations/puppetproduction+1 -3
operations/puppetproduction+2 -0
operations/puppetproduction+0 -4
operations/puppetproduction+1 -1
operations/puppetproduction+1 -193
operations/puppetproduction+2 -8
operations/puppetproduction+1 -1
operations/puppetproduction+3 -0
operations/puppetproduction+1 -1
operations/puppetproduction+1 -56
operations/puppetproduction+1 -56
operations/puppetproduction+14 -0
operations/puppetproduction+1 -4
operations/puppetproduction+1 -0
operations/puppetproduction+13 -1
operations/puppetproduction+2 -0
operations/puppetproduction+1 -0
operations/puppetproduction+1 -1
operations/puppetproduction+0 -67
operations/puppetproduction+0 -69
operations/puppetproduction+0 -4
operations/puppet/nginxmaster+2 -1
operations/puppetproduction+0 -1 K
operations/puppetproduction+4 -0
operations/puppetproduction+1 -4
operations/puppetproduction+2 -5
operations/puppetproduction+8 -0
operations/puppetproduction+1 -0
operations/puppetproduction+3 -1
operations/puppetproduction+8 -32
operations/puppetproduction+118 -42
operations/puppetproduction+2 -0
operations/puppetproduction+2 -0
operations/puppetproduction+4 -8
operations/puppetproduction+1 -0
operations/puppetproduction+5 -1
operations/puppetproduction+7 -1
operations/puppetproduction+1 -0
operations/puppetproduction+1 -1
operations/puppetproduction+1 -0
operations/puppetproduction+0 -75
operations/puppetproduction+1 -54
operations/puppetproduction+1 -5
operations/puppetproduction+2 -5
operations/puppetproduction+1 -0
operations/puppetproduction+1 -0
operations/puppetproduction+2 -1
operations/puppetproduction+5 -15
operations/puppetproduction+1 -0
operations/puppetproduction+1 -0
operations/puppetproduction+1 -0
operations/puppetproduction+6 -0
operations/puppetproduction+8 -0
operations/puppetproduction+0 -7
operations/puppetproduction+1 -0
operations/puppetproduction+4 -0
operations/puppetproduction+1 -0
operations/puppetproduction+5 -1
operations/puppetproduction+15 -5
operations/puppetproduction+0 -5
operations/puppetproduction+8 -0
operations/puppetproduction+1 -0
operations/puppetproduction+9 -5
operations/puppetproduction+2 -1
operations/puppetproduction+3 -0
operations/puppetproduction+1 -0
operations/puppetproduction+0 -132
operations/puppetproduction+0 -102
operations/puppetproduction+5 -0
operations/puppetproduction+0 -264
operations/puppetproduction+1 -2
operations/puppetproduction+1 -8
operations/puppetproduction+4 -0
operations/puppetproduction+2 -2
operations/puppetproduction+0 -2
operations/puppetproduction+2 -21
operations/puppetproduction+0 -5
operations/puppetproduction+1 -0
operations/puppetproduction+33 -16
operations/puppetproduction+3 -1
operations/puppetproduction+7 -0
operations/puppetproduction+3 -2
operations/puppetproduction+3 -2
operations/puppetproduction+1 -0
Show related patches Customize query in gerrit

Related Objects

StatusSubtypeAssignedTask
Resolvedfgiunchedi
Resolvedcolewhite
DeclinedNone
Resolvedfgiunchedi
DuplicateNone
ResolvedMoritzMuehlenhoff
ResolvedMoritzMuehlenhoff
Resolvedtaavi
Resolvedtaavi
Resolved JHedden
Resolved JHedden
Resolved Bstorm
Resolvedbd808
ResolvedAndrew
DeclinedNone
Resolved nskaggs
Resolvedtaavi
Resolvedjbond
Resolvedtaavi
Resolvedtaavi
Resolvedtaavi
Resolvedtaavi
Resolveddcaro
ResolvedAndrew
Resolvedcolewhite

Event Timeline

There are a very large number of changes, so older changes are hidden. Show Older Changes

Change 473302 merged by Cwhite:
[operations/puppet@production] diamond: remove diamond::collector::nginx

https://gerrit.wikimedia.org/r/473302

Change 474309 had a related patch set uploaded (by Cwhite; owner: Cwhite):
[operations/puppet/nginx@master] nginx: remove diamond::collector::nginx reference

https://gerrit.wikimedia.org/r/474309

Change 474321 had a related patch set uploaded (by Cwhite; owner: Cwhite):
[operations/puppet@production] role: add aggregations for TCP Fast Open to prometheus global

https://gerrit.wikimedia.org/r/474321

Change 473295 merged by Cwhite:
[operations/puppet@production] ntp: ensure absent ntpd diamond collector

https://gerrit.wikimedia.org/r/473295

Change 474280 merged by Muehlenhoff:
[operations/puppet@production] Remove Diamond from Swift backends

https://gerrit.wikimedia.org/r/474280

Change 474695 had a related patch set uploaded (by Muehlenhoff; owner: Muehlenhoff):
[operations/puppet@production] Disable Diamond on WDQS hosts

https://gerrit.wikimedia.org/r/474695

Change 474698 had a related patch set uploaded (by Muehlenhoff; owner: Muehlenhoff):
[operations/puppet@production] Absent unused Diamond collector for ldap/corp

https://gerrit.wikimedia.org/r/474698

Change 474699 had a related patch set uploaded (by Muehlenhoff; owner: Muehlenhoff):
[operations/puppet@production] Remove Diamond from openldap/corp servers

https://gerrit.wikimedia.org/r/474699

Change 474321 merged by Cwhite:
[operations/puppet@production] role: add aggregations for TCP Fast Open to prometheus global

https://gerrit.wikimedia.org/r/474321

Change 474698 merged by Muehlenhoff:
[operations/puppet@production] Absent unused Diamond collector for ldap/corp

https://gerrit.wikimedia.org/r/474698

Change 474699 merged by Muehlenhoff:
[operations/puppet@production] Remove Diamond from openldap/corp servers

https://gerrit.wikimedia.org/r/474699

Change 474695 merged by Muehlenhoff:
[operations/puppet@production] Disable Diamond on WDQS hosts

https://gerrit.wikimedia.org/r/474695

Change 474855 had a related patch set uploaded (by Muehlenhoff; owner: Muehlenhoff):
[operations/puppet@production] Remove server-board

https://gerrit.wikimedia.org/r/474855

Change 474864 had a related patch set uploaded (by Muehlenhoff; owner: Muehlenhoff):
[operations/puppet@production] Remove ApacheStatusSimpleCollector

https://gerrit.wikimedia.org/r/474864

Change 474865 had a related patch set uploaded (by Muehlenhoff; owner: Muehlenhoff):
[operations/puppet@production] Remove PyBalStateCollector

https://gerrit.wikimedia.org/r/474865

Change 474887 had a related patch set uploaded (by Muehlenhoff; owner: Muehlenhoff):
[operations/puppet@production] Remove absented Diamond collector for NTP

https://gerrit.wikimedia.org/r/474887

Change 474855 merged by Muehlenhoff:
[operations/puppet@production] Remove server-board

https://gerrit.wikimedia.org/r/474855

Change 474922 had a related patch set uploaded (by Muehlenhoff; owner: Muehlenhoff):
[operations/puppet@production] Disable Diamond on Graphite hosts

https://gerrit.wikimedia.org/r/474922

Change 474930 had a related patch set uploaded (by Muehlenhoff; owner: Muehlenhoff):
[operations/puppet@production] Remove Diamond from restbase servers

https://gerrit.wikimedia.org/r/474930

Change 474309 merged by jenkins-bot:
[operations/puppet/nginx@master] nginx: remove diamond::collector::nginx reference

https://gerrit.wikimedia.org/r/474309

Change 474940 had a related patch set uploaded (by Cwhite; owner: Cwhite):
[operations/puppet@production] nginx: use latest commit

https://gerrit.wikimedia.org/r/474940

Change 474887 merged by Cwhite:
[operations/puppet@production] Remove absented Diamond collector for NTP

https://gerrit.wikimedia.org/r/474887

Change 474865 merged by Muehlenhoff:
[operations/puppet@production] Remove PyBalStateCollector

https://gerrit.wikimedia.org/r/474865

Change 474864 merged by Muehlenhoff:
[operations/puppet@production] Remove ApacheStatusSimpleCollector

https://gerrit.wikimedia.org/r/474864

Change 475009 had a related patch set uploaded (by Cwhite; owner: Cwhite):
[operations/puppet@production] mw_rc_irc: ensure diamond::collector absent

https://gerrit.wikimedia.org/r/475009

Change 475010 had a related patch set uploaded (by Cwhite; owner: Cwhite):
[operations/puppet@production] mw_rc_irc: remove diamond::collector resource

https://gerrit.wikimedia.org/r/475010

Change 474940 merged by Cwhite:
[operations/puppet@production] nginx: use latest commit

https://gerrit.wikimedia.org/r/474940

Change 474922 merged by Muehlenhoff:
[operations/puppet@production] Disable Diamond on Graphite hosts

https://gerrit.wikimedia.org/r/474922

Change 474930 merged by Muehlenhoff:
[operations/puppet@production] Remove Diamond from restbase servers

https://gerrit.wikimedia.org/r/474930

Change 475444 had a related patch set uploaded (by Muehlenhoff; owner: Muehlenhoff):
[operations/puppet@production] Remove Diamond from several analytics roles

https://gerrit.wikimedia.org/r/475444

Change 475444 merged by Muehlenhoff:
[operations/puppet@production] Remove Diamond from several analytics roles

https://gerrit.wikimedia.org/r/475444

Change 475456 had a related patch set uploaded (by Muehlenhoff; owner: Muehlenhoff):
[operations/puppet@production] Remove Diamond from Ganeti hosts

https://gerrit.wikimedia.org/r/475456

Change 475456 merged by Muehlenhoff:
[operations/puppet@production] Remove Diamond from Ganeti hosts

https://gerrit.wikimedia.org/r/475456

Change 475009 merged by Cwhite:
[operations/puppet@production] mw_rc_irc: ensure diamond::collector absent

https://gerrit.wikimedia.org/r/475009

Change 475957 had a related patch set uploaded (by Muehlenhoff; owner: Muehlenhoff):
[operations/puppet@production] Remove IRCDStats collector

https://gerrit.wikimedia.org/r/475957

Change 475967 had a related patch set uploaded (by Muehlenhoff; owner: Muehlenhoff):
[operations/puppet@production] Absent Redis Diamond collector on Redis slaves

https://gerrit.wikimedia.org/r/475967

Change 475969 had a related patch set uploaded (by Muehlenhoff; owner: Muehlenhoff):
[operations/puppet@production] Absent Redis Diamond collector on Redis masters

https://gerrit.wikimedia.org/r/475969

Change 467264 merged by Muehlenhoff:
[operations/puppet@production] Remove Diamond from DB roles

https://gerrit.wikimedia.org/r/467264

Change 475957 merged by Muehlenhoff:
[operations/puppet@production] Remove IRCDStats collector

https://gerrit.wikimedia.org/r/475957

Change 475010 abandoned by Cwhite:
mw_rc_irc: remove diamond::collector resource and collector script

Reason:
Removed in I44ea7b94ab1eab45a923917dd6f0c5b6ae629c5c

https://gerrit.wikimedia.org/r/475010

Change 476001 had a related patch set uploaded (by Muehlenhoff; owner: Muehlenhoff):
[operations/puppet@production] Remove Diamond on additional DB roles

https://gerrit.wikimedia.org/r/476001

Change 475967 merged by Muehlenhoff:
[operations/puppet@production] Absent Redis Diamond collector on Redis slaves

https://gerrit.wikimedia.org/r/475967

Change 476001 merged by Muehlenhoff:
[operations/puppet@production] Remove Diamond on additional DB roles

https://gerrit.wikimedia.org/r/476001

Change 475969 merged by Muehlenhoff:
[operations/puppet@production] Absent Redis Diamond collector on Redis masters

https://gerrit.wikimedia.org/r/475969

Change 476226 had a related patch set uploaded (by Muehlenhoff; owner: Muehlenhoff):
[operations/puppet@production] Remove Diamond from redis::misc systems

https://gerrit.wikimedia.org/r/476226

Change 476226 merged by Muehlenhoff:
[operations/puppet@production] Remove Diamond from redis::misc systems

https://gerrit.wikimedia.org/r/476226

Change 476606 had a related patch set uploaded (by Muehlenhoff; owner: Muehlenhoff):
[operations/puppet@production] Absent NfsdCollector Diamond collector

https://gerrit.wikimedia.org/r/476606

Change 476606 merged by GTirloni:
[operations/puppet@production] Absent NfsdCollector Diamond collector

https://gerrit.wikimedia.org/r/476606

Change 476883 had a related patch set uploaded (by Muehlenhoff; owner: Muehlenhoff):
[operations/puppet@production] Absent Redis Diamond collector on mc* servers

https://gerrit.wikimedia.org/r/476883

Change 476883 merged by Muehlenhoff:
[operations/puppet@production] Absent Redis Diamond collector on mc* servers

https://gerrit.wikimedia.org/r/476883

Change 477231 had a related patch set uploaded (by Muehlenhoff; owner: Muehlenhoff):
[operations/puppet@production] Remove Redis collector

https://gerrit.wikimedia.org/r/477231

Change 477231 merged by Muehlenhoff:
[operations/puppet@production] Remove Redis collector

https://gerrit.wikimedia.org/r/477231

I've deleted the dashboard now. It was created this during an incident investigation and hasn't been used since afaik.

Change 477748 had a related patch set uploaded (by Muehlenhoff; owner: Muehlenhoff):
[operations/puppet@production] Remove Diamond on maps servers

https://gerrit.wikimedia.org/r/477748

Change 477748 merged by Muehlenhoff:
[operations/puppet@production] Remove Diamond on maps servers

https://gerrit.wikimedia.org/r/477748

Change 478016 had a related patch set uploaded (by Muehlenhoff; owner: Muehlenhoff):
[operations/puppet@production] Remove Diamond from DNS roles

https://gerrit.wikimedia.org/r/478016

Change 478368 had a related patch set uploaded (by Cwhite; owner: Cwhite):
[operations/puppet@production] remove directorysize diamond collector

https://gerrit.wikimedia.org/r/478368

Change 478371 had a related patch set uploaded (by Cwhite; owner: Cwhite):
[operations/puppet@production] remove diamond::collector reference from role::labs::nfs::secondary

https://gerrit.wikimedia.org/r/478371

Change 478368 merged by GTirloni:
[operations/puppet@production] remove directorysize diamond collector

https://gerrit.wikimedia.org/r/478368

Change 478016 merged by Muehlenhoff:
[operations/puppet@production] Remove Diamond from DNS roles

https://gerrit.wikimedia.org/r/478016

Change 478371 merged by GTirloni:
[operations/puppet@production] remove diamond::collector reference from role::labs::nfs::secondary

https://gerrit.wikimedia.org/r/478371

Change 469250 merged by Cwhite:
[operations/puppet@production] memcached: remove memcached diamond collector

https://gerrit.wikimedia.org/r/469250

Change 469258 merged by Cwhite:
[operations/puppet@production] memcached: remove diamond::collector resource

https://gerrit.wikimedia.org/r/469258

Change 466907 merged by Cwhite:
[operations/puppet@production] hiera: remove diamond from thumbor role

https://gerrit.wikimedia.org/r/466907

Change 429221 abandoned by Cwhite:
memcached: deprecate Diamond collector

Reason:
Covered by I4506039ff403bce1c5fc5b2093861a937d84e31b

https://gerrit.wikimedia.org/r/429221

Change 479169 had a related patch set uploaded (by Muehlenhoff; owner: Muehlenhoff):
[operations/puppet@production] Remove now obsolete Diamond collector and related conffiles

https://gerrit.wikimedia.org/r/479169

Change 479189 had a related patch set uploaded (by Muehlenhoff; owner: Muehlenhoff):
[operations/puppet@production] Remove Diamond from ORES hosts

https://gerrit.wikimedia.org/r/479189

Change 479169 merged by Muehlenhoff:
[operations/puppet@production] Remove now obsolete Diamond collector and related conffile

https://gerrit.wikimedia.org/r/479169

Change 479189 merged by Muehlenhoff:
[operations/puppet@production] Remove Diamond from ORES hosts

https://gerrit.wikimedia.org/r/479189

Change 479446 had a related patch set uploaded (by Muehlenhoff; owner: Muehlenhoff):
[operations/puppet@production] Remove Diamond from further roles

https://gerrit.wikimedia.org/r/479446

Change 479446 merged by Muehlenhoff:
[operations/puppet@production] Remove Diamond from further roles

https://gerrit.wikimedia.org/r/479446

Change 480032 had a related patch set uploaded (by Muehlenhoff; owner: Muehlenhoff):
[operations/puppet@production] Remove Diamond from further roles

https://gerrit.wikimedia.org/r/480032

Change 480032 merged by Muehlenhoff:
[operations/puppet@production] Remove Diamond from further roles

https://gerrit.wikimedia.org/r/480032

This is done (and the task is getting too big as well), I created T212231 for some followup work.

Change 466908 merged by Cwhite:
[operations/puppet@production] hiera: remove diamond from labweb role

https://gerrit.wikimedia.org/r/466908

Change 464366 abandoned by Cwhite:
memcached, redis: remove diamond

Reason:
no longer needed

https://gerrit.wikimedia.org/r/464366