Page MenuHomePhabricator

Puppet fails on labs instances due to Ganglia (ex: using apache::site puppet class)
Closed, ResolvedPublic

Description

Puppet fails on CI slaves with:

Notice: /Stage[main]/Ganglia::Monitor::Packages/File[/etc/ganglia/conf.d]/ensure: created
Error: Cannot create /usr/lib/ganglia/python_modules; parent directory /usr/lib/ganglia does not exist

That is because ganglia is no more installed, though on old instances /etc/ganglia is still around.

The root cause is that the browser tests virtual host uses apache::site which ends up including apache::monitoring` which include ganglia ...

Same goes with hhvm::monitoring

Looks like we can short circuit it via $::standard::has_ganglia.

Event Timeline

Change 287695 had a related patch set uploaded (by Hashar):
apache: skip ganglia when it is unwanted

https://gerrit.wikimedia.org/r/287695

Mentioned in SAL [2016-05-09T20:04:01Z] <hashar> CI, removing ganglia configuration entirely via: salt -v '*' cmd.run 'dpkg --purge ganglia-monitor' # T134808

Mentioned in SAL [2016-05-09T20:06:27Z] <hashar> CI, removing ganglia configuration entirely via: salt -v '*' cmd.run 'rm -fRv /etc/ganglia' # T134808

Mentioned in SAL [2016-05-09T20:13:45Z] <hashar> beta: salt -v '*' cmd.run 'dpkg --purge libganglia1 ganglia-monitor; rm -fR /etc/ganglia' # T134808

hashar renamed this task from Puppet fails on Jessie permanent CI slaves to Puppet fails on labs instances using apache::site puppet class.May 9 2016, 8:14 PM
hashar claimed this task.
hashar triaged this task as Medium priority.

Change 287743 had a related patch set uploaded (by Hashar):
hhvm: skip ganglia when it is unwanted

https://gerrit.wikimedia.org/r/287743

hashar renamed this task from Puppet fails on labs instances using apache::site puppet class to Puppet fails on labs instances due to Ganglia (ex: using apache::site puppet class).May 9 2016, 8:31 PM

@hashar re: ganglia on labs, also see T115330. ganglia-monitor should be killed on all, and there is this list of IPs

https://phabricator.wikimedia.org/T115330#2244694

Change 287976 had a related patch set uploaded (by Hashar):
ocg: skip ganglia when it is unwanted

https://gerrit.wikimedia.org/r/287976

Change 287976 merged by Dzahn:
ocg: skip ganglia when it is unwanted

https://gerrit.wikimedia.org/r/287976

Change 287695 merged by Dzahn:
apache: skip ganglia when it is unwanted

https://gerrit.wikimedia.org/r/287695

Change 287743 merged by Dzahn:
hhvm: skip ganglia when it is unwanted

https://gerrit.wikimedia.org/r/287743

@hashar all 3 merged (after checking with compiler and in prod)

puppet compiler is magic. Thank you!

All of CI / Beta instances are no more affected. So I guess that solves it.