I have set a message pointing to this task by using:

puppet agent --enable; puppet agent --disable 'https://phabricator.wikimedia.org/T96921'

I have just enabled puppet again, no reason to have it disabled anymore (we did so for testing purposes couple weeks back)

Seems it has not been reenabled so I have did it and ran puppet again. From the diff it changes a bunch of files under /etc/eventlogging.d/consumers/

--- /etc/eventlogging.d/consumers/client-side-events-log        2015-04-01 01:47:37.150152215 +0000
+++ /tmp/puppet-file20150424-32504-1v1nxs7-0    2015-04-24 08:54:51.685501072 +0000
@@ -1,2 +1,2 @@
 tcp://127.0.0.1:8422?raw=1&socket_id=client-side-events.log
-file:///var/log/eventlogging/client-side-events.log
+file:///client-side-events.log

/var/log/eventlogging is a symlink to /srv/log/eventlogging at least. The files are generated from an erb template and the line is the @output parameter. It is build from the role class based on $::eventlogging::log_dir which for some reason is yield empty :(

I have disabled puppet agent again on the eventlogging02 instance and manually fixed the paths in all /etc/eventlogging.d/consumers files.

I have restarted the consumers using:

restart eventlogging/consumer NAME=all-events-log
restart eventlogging/consumer NAME=server-side-events-log
restart eventlogging/consumer NAME=client-side-events-kafka-log
restart eventlogging/consumer NAME=client-side-events-log

Files under /srv/log/eventlogging are being written to again. That lost a good 30-40 minutes of logs.

Ccing @Ottomata for the puppet part.

The instance has the class role::eventlogging applied then it:

include eventlogging
$log_dir = $::eventlogging::log_dir
eventlogging::service::consumer {
    'server-side-events.log':
        input  => "tcp://${processor}:8421?raw=1",
        output => "file://${log_dir}/server-side-events.log";
}

For some reason, the include is realized later and $log_dir = $::eventlogging::log_dir ends up being an empty string which causes the bad paths.

I tried to change the include eventlogging to a require eventlogging to ensure the log_dir variable is properly defined but then I end up with a puppet dependency cycle:

Error: Could not apply complete catalog: Found 1 dependency cycle:
(
   File[/etc/eventlogging.d/consumers/all-events-log]
=> Eventlogging::Service::Consumer[all-events.log]
=> Class[Role::Eventlogging]
=> Class[Role::Eventlogging]
=> Notify[eventlogging_log_dir]
=> Class[Role::Eventlogging])

I haven't looked further but it seems to me something is wrong in the puppet manifests :/

I'm enabling puppet and running it now so that people can log into it post NFS outage.