Page MenuHomePhabricator

Elasticsearch logs are not send to logstash after 2.3.3 upgrade
Closed, ResolvedPublic

Description

Since the upgrade of elasticsearch to 2.3.3, logs are not sent to logstash anymore.

tcpdump confirms that no logging packet is leaving the elasticsearch server. The issue can be reproduced on Beta.

Event Timeline

Restricted Application added subscribers: Zppix, Aklapper. · View Herald Transcript

Error seems related to the new elasticsearch security manager. It is now more strict...

log4j:ERROR Could not send GELF message: access denied ("java.net.SocketPermission" "localhost:0" "listen,resolve")
java.security.AccessControlException: access denied ("java.net.SocketPermission" "localhost:0" "listen,resolve")

This should be fixed by adding permissions like:

grant codeBase "file:/usr/share/elasticsearch/logstash-gelf.jar" {
  permission java.net.SocketPermission "localhost:0", "listen,resolve";
};

or

grant {
  permission java.net.SocketPermission "localhost:0", "listen,resolve";
};

Now I just need to find where to load that additional policy file...

Change 295129 had a related patch set uploaded (by Nicko):
T136696 Including a .policy file to grant permission to send logs to logstash

https://gerrit.wikimedia.org/r/295129

Current patch isn't working. Moving this to backlog until I get time to dig more into it.

debt triaged this task as Medium priority.Aug 22 2016, 5:38 PM
debt added a subscriber: debt.

Moving to the backlog at this time to tackle later on

Issue opened upstream to see if we can get some external help: https://github.com/elastic/elasticsearch/issues/21428

The problem is related to our old version of logstash-gelf (1.5.3). This version initializes the datagram socket only when the first message is sent. Elasticsearch initialises the security manager after configuring logger, which means that if the datagram socket is opened during logging configuration, there is no security manager in the way. The latest logstash-gelf version does moved socket creation earlier and works just fine with elasticsearch.

Fixing this issue require updating our logstash-gelf package.

logstash-gelf has been upgraded on disk, but a cluster restart is still needed to pick up that change on production servers. This is going to wait for the next cluster restart (I'm sure there is one coming up soon...). Let's keep this task open in the meantime...

Was the elastic search plugin updated on the install? Might have been because from logstash 2.x it now defaults to http instead of node.

@Paladox I'm not sure I understand what you mean with the plugin update...

Log messages can now be seen in logstash, so we seem to be good. I'll close this for now, feel free to reopen if needed.

Change 295129 abandoned by Gehel:
T136696 Including a .policy file to grant permission to send logs to logstash

https://gerrit.wikimedia.org/r/295129