Uninstall ganglia from the fleet
Closed, ResolvedPublic

Description

Now that fundraising is no longer using ganglia we can uninstall it from the fleet (gmetad / gmond / etc) and remove the relevant puppet bits.

There are a very large number of changes, so older changes are hidden. Show Older Changes

Change 395890 had a related patch set uploaded (by Dzahn; owner: Dzahn):
[operations/puppet@production] ganglia: delete views for kafkatee, hadoop, varnishkafka

https://gerrit.wikimedia.org/r/395890

Change 395896 had a related patch set uploaded (by Dzahn; owner: Dzahn):
[operations/puppet@production] mirrors,poolcounter,tendril,tor,labpuppetmaster,openldap::lab: rm ganglia

https://gerrit.wikimedia.org/r/395896

Change 395896 merged by Dzahn:
[operations/puppet@production] mirrors,poolcounter,tendril,tor,labpuppetmaster,openldap::lab: rm ganglia

https://gerrit.wikimedia.org/r/395896

Change 395900 had a related patch set uploaded (by Dzahn; owner: Dzahn):
[operations/puppet@production] puppetcompiler1001: add role(test)

https://gerrit.wikimedia.org/r/395900

Change 395900 merged by Dzahn:
[operations/puppet@production] puppetcompiler1001: add role(test)

https://gerrit.wikimedia.org/r/395900

Change 395890 merged by Dzahn:
[operations/puppet@production] ganglia: delete views for kafkatee, hadoop, varnishkafka

https://gerrit.wikimedia.org/r/395890

Change 396086 had a related patch set uploaded (by Dzahn; owner: Dzahn):
[operations/puppet@production] logging/kafkatee: remove ganglia monitoring

https://gerrit.wikimedia.org/r/396086

Change 396086 merged by Dzahn:
[operations/puppet@production] logging/kafkatee: remove ganglia monitoring

https://gerrit.wikimedia.org/r/396086

Change 396088 had a related patch set uploaded (by Dzahn; owner: Dzahn):
[operations/puppet/kafkatee@master] kafkatee: remove Ganglia monitoring class and script

https://gerrit.wikimedia.org/r/396088

Change 396101 had a related patch set uploaded (by Dzahn; owner: Dzahn):
[operations/puppet@production] site: add ores200[19] as spare systems

https://gerrit.wikimedia.org/r/396101

Change 396101 had a related patch set uploaded (by Dzahn; owner: Dzahn):
[operations/puppet@production] site: add ores200[19] as spare systems

https://gerrit.wikimedia.org/r/396101

Change 396101 merged by Dzahn:
[operations/puppet@production] site: add ores200[19] as spare systems

https://gerrit.wikimedia.org/r/396101

Change 396104 had a related patch set uploaded (by Dzahn; owner: Dzahn):
[operations/puppet@production] site: add eventlog2001 as spare::system

https://gerrit.wikimedia.org/r/396104

Change 396104 merged by Dzahn:
[operations/puppet@production] site: add eventlog2001 as spare::system

https://gerrit.wikimedia.org/r/396104

Change 396106 had a related patch set uploaded (by Dzahn; owner: Dzahn):
[operations/puppet@production] bastionhost, mw_rc_irc,backup::offsite,pybaltest: rm ganglia

https://gerrit.wikimedia.org/r/396106

Change 396106 merged by Dzahn:
[operations/puppet@production] bastionhost, mw_rc_irc,backup::offsite,pybaltest: rm ganglia

https://gerrit.wikimedia.org/r/396106

Change 396129 had a related patch set uploaded (by Dzahn; owner: Dzahn):
[operations/puppet@production] lvs::balancer: remove ganglia

https://gerrit.wikimedia.org/r/396129

Change 396129 merged by Dzahn:
[operations/puppet@production] lvs::balancer: remove ganglia

https://gerrit.wikimedia.org/r/396129

Change 396290 had a related patch set uploaded (by Dzahn; owner: Dzahn):
[operations/puppet@production] db2011: remove ganglia

https://gerrit.wikimedia.org/r/396290

Change 396290 merged by Dzahn:
[operations/puppet@production] db2011: remove ganglia

https://gerrit.wikimedia.org/r/396290

Change 396291 had a related patch set uploaded (by Dzahn; owner: Dzahn):
[operations/puppet@production] mariadb::labs_deprecated: remove ganglia

https://gerrit.wikimedia.org/r/396291

Change 396291 merged by Dzahn:
[operations/puppet@production] mariadb::labs_deprecated: remove ganglia

https://gerrit.wikimedia.org/r/396291

Change 396292 had a related patch set uploaded (by Dzahn; owner: Dzahn):
[operations/puppet@production] labsdb: remove ganglia

https://gerrit.wikimedia.org/r/396292

Change 396292 merged by Dzahn:
[operations/puppet@production] labsdb: remove ganglia

https://gerrit.wikimedia.org/r/396292

Change 396294 had a related patch set uploaded (by Dzahn; owner: Dzahn):
[operations/puppet@production] labsdb::slave: keep ganglia because postgresql

https://gerrit.wikimedia.org/r/396294

Change 396294 merged by Dzahn:
[operations/puppet@production] labsdb::slave: keep ganglia because postgresql

https://gerrit.wikimedia.org/r/396294

Change 396088 merged by Dzahn:
[operations/puppet/kafkatee@master] kafkatee: remove Ganglia monitoring class and script

https://gerrit.wikimedia.org/r/396088

Change 397986 had a related patch set uploaded (by Dzahn; owner: Dzahn):
[operations/puppet@production] mail::mx: remove ganglia

https://gerrit.wikimedia.org/r/397986

Change 397986 merged by Dzahn:
[operations/puppet@production] mail::mx: remove ganglia

https://gerrit.wikimedia.org/r/397986

Change 397990 had a related patch set uploaded (by Dzahn; owner: Dzahn):
[operations/puppet@production] mariadb::parsercache: remove ganglia

https://gerrit.wikimedia.org/r/397990

Change 382916 merged by Dzahn:
[operations/puppet@production] exim4/ganglia: mx,otrs,lists,phab: rm Ganglia exim stats

https://gerrit.wikimedia.org/r/382916

Change 394727 abandoned by Dzahn:
ganglia: add decom bash script if on trusty (WIP)

https://gerrit.wikimedia.org/r/394727

Change 397990 merged by Dzahn:
[operations/puppet@production] mariadb::parsercache: remove ganglia

https://gerrit.wikimedia.org/r/397990

Change 398186 had a related patch set uploaded (by Dzahn; owner: Dzahn):
[operations/puppet@production] parsercache: remove ganglia from parsercache nodes

https://gerrit.wikimedia.org/r/398186

Change 398186 merged by Dzahn:
[operations/puppet@production] parsercache: remove ganglia from parsercache nodes

https://gerrit.wikimedia.org/r/398186

Change 398390 had a related patch set uploaded (by Dzahn; owner: Dzahn):
[operations/puppet@production] dbstore: remove ganglia

https://gerrit.wikimedia.org/r/398390

Change 398390 merged by Dzahn:
[operations/puppet@production] dbstore: remove ganglia

https://gerrit.wikimedia.org/r/398390

Change 398398 had a related patch set uploaded (by Dzahn; owner: Dzahn):
[operations/puppet@production] external storage codfw: remove ganglia

https://gerrit.wikimedia.org/r/398398

Change 398398 merged by Dzahn:
[operations/puppet@production] external storage codfw: remove ganglia

https://gerrit.wikimedia.org/r/398398

Change 398413 had a related patch set uploaded (by Dzahn; owner: Dzahn):
[operations/puppet@production] mysql codfw: remove ganglia

https://gerrit.wikimedia.org/r/398413

Change 398413 merged by Dzahn:
[operations/puppet@production] mysql codfw: remove ganglia

https://gerrit.wikimedia.org/r/398413

Mentioned in SAL (#wikimedia-operations) [2017-12-15T03:01:15Z] <mutante> db2016 thru db2019 - had to manually kill gmond process to decom ganglia, other db codfw hosts: didnt need it | running puppet on db205* and others in codfw to remove all ganglia (T177225)

Change 398526 had a related patch set uploaded (by Dzahn; owner: Dzahn):
[operations/puppet@production] external storage eqiad: remove ganglia

https://gerrit.wikimedia.org/r/398526

Change 398526 merged by Dzahn:
[operations/puppet@production] external storage eqiad: remove ganglia

https://gerrit.wikimedia.org/r/398526

Change 398528 had a related patch set uploaded (by Dzahn; owner: Dzahn):
[operations/puppet@production] dbproxy eqiad: remove ganglia

https://gerrit.wikimedia.org/r/398528

Change 398528 merged by Dzahn:
[operations/puppet@production] dbproxy eqiad: remove ganglia

https://gerrit.wikimedia.org/r/398528

Change 398531 had a related patch set uploaded (by Dzahn; owner: Dzahn):
[operations/puppet@production] db eqiad: remove ganglia

https://gerrit.wikimedia.org/r/398531

Change 398531 merged by Dzahn:
[operations/puppet@production] db eqiad: remove ganglia

https://gerrit.wikimedia.org/r/398531

Dzahn added a comment.Dec 15 2017, 8:51 PM

Alright, Ganglia is purged from everything across the board, except 17 hosts now! :) They are:

4 x maps codfw (osm/postgres)
4 x maps eqiad (osm/postgres)
3 x maps-test codfw (osm/postgres)
3 x labsdb eqiad (postgres)
2 x install (aggregators eqiad/codfw)
1 x ganglia-web (uranium)

All else DONE

Gehel added a subscriber: Gehel.Mon, Dec 18, 5:53 PM

Alright, Ganglia is purged from everything across the board, except 17 hosts now! :) They are:

4 x maps codfw (osm/postgres)
4 x maps eqiad (osm/postgres)
3 x maps-test codfw (osm/postgres)

We are not actively using ganglia for maps, so we can remove those without any issue.

Change 398899 had a related patch set uploaded (by Dzahn; owner: Dzahn):
[operations/puppet@production] maps: remove ganglia

https://gerrit.wikimedia.org/r/398899

Change 398899 merged by Dzahn:
[operations/puppet@production] maps: remove ganglia

https://gerrit.wikimedia.org/r/398899

Dzahn added a comment.Mon, Dec 18, 7:29 PM

We are not actively using ganglia for maps, so we can remove those without any issue.

Cool! Thanks for confirming. I had just kept it because there was no replacemente for postgres stats yet but that's also work in progress and coming up. Removed from maps and maps-test cluster just now :)

Change 394518 abandoned by Dzahn:
mysql eqiad: remove ganglia

Reason:
not neeed anymore, done in multiple other changes

https://gerrit.wikimedia.org/r/394518

Change 398903 had a related patch set uploaded (by Dzahn; owner: Dzahn):
[operations/puppet@production] labsdb100[467]: remove ganglia

https://gerrit.wikimedia.org/r/398903

Change 398903 merged by Dzahn:
[operations/puppet@production] labsdb100[467]: remove ganglia

https://gerrit.wikimedia.org/r/398903

Change 382905 merged by Dzahn:
[operations/puppet@production] osm: remove all ganglia support

https://gerrit.wikimedia.org/r/382905

Change 382906 merged by Dzahn:
[operations/puppet@production] postgresql: remove all ganglia support

https://gerrit.wikimedia.org/r/382906

Change 398904 had a related patch set uploaded (by Dzahn; owner: Dzahn):
[operations/puppet@production] osm/postgres: remove ganglia diskstat plugin inclusion

https://gerrit.wikimedia.org/r/398904

Change 398904 merged by Dzahn:
[operations/puppet@production] osm/postgres: remove ganglia diskstat plugin inclusion

https://gerrit.wikimedia.org/r/398904

Change 382904 merged by Dzahn:
[operations/puppet@production] ganglia/site: decom ganglia-web node, rm eqiad/codfw aggregators

https://gerrit.wikimedia.org/r/382904

Mentioned in SAL (#wikimedia-operations) [2017-12-18T20:53:35Z] <mutante> ganglia.wikimedia.org shut down just now after a deprecation period - service is out of commission - T177225

Change 382923 merged by Dzahn:
[operations/puppet@production] statsd: remove ganglia backend support

https://gerrit.wikimedia.org/r/382923

Change 382924 had a related patch set uploaded (by Dzahn; owner: Dzahn):
[operations/puppet@production] standard: decom ganglia plugin everywhere by default

https://gerrit.wikimedia.org/r/382924

Change 382924 merged by Dzahn:
[operations/puppet@production] standard: decom ganglia plugin everywhere by default

https://gerrit.wikimedia.org/r/382924

Change 382926 merged by Dzahn:
[operations/puppet@production] standard: actually drop 'has_ganglia' param entirely

https://gerrit.wikimedia.org/r/382926

Change 382932 merged by Dzahn:
[operations/puppet@production] ganglia: delete ganglia-web classes and role

https://gerrit.wikimedia.org/r/382932

Change 399119 had a related patch set uploaded (by Dzahn; owner: Dzahn):
[operations/puppet@production] network::constants: drop uranium from monitoring hosts

https://gerrit.wikimedia.org/r/399119

Change 399120 had a related patch set uploaded (by Dzahn; owner: Dzahn):
[operations/puppet@production] remove ganglia_aggregators settings from hiera

https://gerrit.wikimedia.org/r/399120

Change 399121 had a related patch set uploaded (by Dzahn; owner: Dzahn):
[operations/puppet@production] rm role/manifests/ganglia/config

https://gerrit.wikimedia.org/r/399121

Change 399121 merged by Dzahn:
[operations/puppet@production] rm role/manifests/ganglia/config

https://gerrit.wikimedia.org/r/399121

Change 399124 had a related patch set uploaded (by Dzahn; owner: Dzahn):
[operations/dns@master] remove ganglia.wikimedia.org

https://gerrit.wikimedia.org/r/399124

Change 399119 merged by Alexandros Kosiaris:
[operations/puppet@production] network::constants: drop uranium from monitoring hosts

https://gerrit.wikimedia.org/r/399119

Change 399120 merged by Dzahn:
[operations/puppet@production] remove ganglia_aggregators settings from hiera

https://gerrit.wikimedia.org/r/399120

Change 399124 merged by Dzahn:
[operations/dns@master] remove ganglia.wikimedia.org

https://gerrit.wikimedia.org/r/399124

Mentioned in SAL (#wikimedia-operations) [2017-12-19T19:49:23Z] <mutante> deleted ganglia.wikimedia.org from DNS - webserver was already down since yesterday - not used anymore (T177225)

Change 399248 had a related patch set uploaded (by Dzahn; owner: Dzahn):
[operations/puppet@production] redis: delete ganglia monitoring script

https://gerrit.wikimedia.org/r/399248

Change 382933 merged by Dzahn:
[operations/puppet@production] ganglia: delete the module

https://gerrit.wikimedia.org/r/382933

Change 399326 had a related patch set uploaded (by Dzahn; owner: Dzahn):
[operations/puppet@production] redis: delete ganglia monitoring script

https://gerrit.wikimedia.org/r/399326

Dzahn closed this task as Resolved.Wed, Dec 20, 3:57 AM

Ganglia has been uninstalled from the fleet, the aggregators are gone, the roles and the module is deleted, the DNS name is removed, for all purposes it's gone. The remaining hits for grepping "ganglia" across the repo are mostly related to the "ganglia_clusters" variable in Hiera which we should replace with LVS config or rename:

https://gerrit.wikimedia.org/r/#/c/382930/

(WIP) https://gerrit.wikimedia.org/r/#/c/382931/

And then it appears a couple times in modules/confluent/manifests/kafka/mirror/jmxtrans.pp and in an example in wmflib.

But none of this means Ganglia is still running at any capacity, so this ticket is resolved.

Change 399326 merged by Dzahn:
[operations/puppet@production] redis: delete ganglia monitoring script

https://gerrit.wikimedia.org/r/399326

Change 399686 had a related patch set uploaded (by Dzahn; owner: Dzahn):
[operations/puppet@production] confluent:kafka:jmxtrans: remove Ganglia support

https://gerrit.wikimedia.org/r/399686

Change 399686 abandoned by Dzahn:
confluent:kafka:jmxtrans: remove Ganglia support

Reason:
depends on a submodule

https://gerrit.wikimedia.org/r/399686

Change 399686 restored by Dzahn:
confluent:kafka:jmxtrans: remove Ganglia support

https://gerrit.wikimedia.org/r/399686

Change 399689 had a related patch set uploaded (by Ottomata; owner: Ottomata):
[operations/puppet/varnishkafka@master] Parameterize kafka.ssl.cipher.suites

https://gerrit.wikimedia.org/r/399689

Change 399691 had a related patch set uploaded (by Dzahn; owner: Dzahn):
[operations/puppet@production] apache: remove comments about Ganglia monitoring

https://gerrit.wikimedia.org/r/399691

Change 399691 merged by Dzahn:
[operations/puppet@production] apache: remove comments about Ganglia monitoring

https://gerrit.wikimedia.org/r/399691

Change 399699 had a related patch set uploaded (by Dzahn; owner: Dzahn):
[operations/puppet/jmxtrans@master] drop optional Ganglia params from metrics::jvm

https://gerrit.wikimedia.org/r/399699

Change 399689 merged by Ottomata:
[operations/puppet/varnishkafka@master] Parameterize kafka.ssl.cipher.suites

https://gerrit.wikimedia.org/r/399689

Change 399686 merged by Dzahn:
[operations/puppet@production] confluent:kafka:jmxtrans: remove Ganglia support

https://gerrit.wikimedia.org/r/399686

Change 399699 abandoned by Dzahn:
drop optional Ganglia params from metrics::jvm

Reason:
not needed for the other change to work

https://gerrit.wikimedia.org/r/399699

Change 399248 abandoned by Dzahn:
redis: delete ganglia monitoring script

Reason:
already done

https://gerrit.wikimedia.org/r/399248