Page MenuHomePhabricator

puppet agent disabled on beta cluster deployment-eventlogging02.eqiad.wmflabs instance
Closed, ResolvedPublic


On the beta cluster, the puppet agent is disabled on instance deployment-eventlogging02.eqiad.wmflabs. @coren and @thcipriani found that when doing a migration that required a recent puppet patch. It was not applied.

If there is no good reason to have puppet disabled, would you mind reenabling it with: puppet agent --enable ? Thanks!

Ccing @mforns @Nuria and @joal who appear in last output.

Event Timeline

hashar raised the priority of this task from to Needs Triage.
hashar updated the task description. (Show Details)
hashar added subscribers: hashar, coren, thcipriani and 3 others.

I have set a message pointing to this task by using:

puppet agent --enable; puppet agent --disable ''

I have just enabled puppet again, no reason to have it disabled anymore (we did so for testing purposes couple weeks back)

Seems it has not been reenabled so I have did it and ran puppet again. From the diff it changes a bunch of files under /etc/eventlogging.d/consumers/

--- /etc/eventlogging.d/consumers/client-side-events-log        2015-04-01 01:47:37.150152215 +0000
+++ /tmp/puppet-file20150424-32504-1v1nxs7-0    2015-04-24 08:54:51.685501072 +0000
@@ -1,2 +1,2 @@

/var/log/eventlogging is a symlink to /srv/log/eventlogging at least. The files are generated from an erb template and the line is the @output parameter. It is build from the role class based on $::eventlogging::log_dir which for some reason is yield empty :(

I have disabled puppet agent again on the eventlogging02 instance and manually fixed the paths in all /etc/eventlogging.d/consumers files.

I have restarted the consumers using:

restart eventlogging/consumer NAME=all-events-log
restart eventlogging/consumer NAME=server-side-events-log
restart eventlogging/consumer NAME=client-side-events-kafka-log
restart eventlogging/consumer NAME=client-side-events-log

Files under /srv/log/eventlogging are being written to again. That lost a good 30-40 minutes of logs.

Ccing @Ottomata for the puppet part.

The instance has the class role::eventlogging applied then it:

include eventlogging
$log_dir = $::eventlogging::log_dir
eventlogging::service::consumer {
        input  => "tcp://${processor}:8421?raw=1",
        output => "file://${log_dir}/server-side-events.log";

For some reason, the include is realized later and $log_dir = $::eventlogging::log_dir ends up being an empty string which causes the bad paths.

I tried to change the include eventlogging to a require eventlogging to ensure the log_dir variable is properly defined but then I end up with a puppet dependency cycle:

Error: Could not apply complete catalog: Found 1 dependency cycle:
=> Eventlogging::Service::Consumer[all-events.log]
=> Class[Role::Eventlogging]
=> Class[Role::Eventlogging]
=> Notify[eventlogging_log_dir]
=> Class[Role::Eventlogging])

I haven't looked further but it seems to me something is wrong in the puppet manifests :/

I'm enabling puppet and running it now so that people can log into it post NFS outage.

hashar assigned this task to yuvipanda.
hashar moved this task from To Triage to Externally Blocked on the Beta-Cluster-Infrastructure board.

Puppet has been reenabled by @yuvipanda and it is passing.