This is merely a support question.
I have set up a labs instance logstash.integration.eqiad.wmflabs using Jessie. It comes with Java 7 and ElasticSearch 2.3.5. Plugins are loaded from a clone of our plugins. On start it raises some jar hell:
failed to load bundle [ file:/srv/deployment/elasticsearch/plugins/analysis-icu/lucene-analyzers-icu-5.5.0.jar, file:/srv/deployment/elasticsearch/plugins/analysis-icu/icu4j-54.1.jar, file:/srv/deployment/elasticsearch/plugins/analysis-icu/analysis-icu-2.3.5.jar ] due to jar hell elasticsearch[5744]: Likely root cause: java.util.zip.ZipException: error in opening zip file elasticsearch[5744]: at java.util.zip.ZipFile.open(Native Method)
SOLUTION
cd /srv/deployment/elasticsearch/plugins git fat init && git fat pull
Bunch of random traces/infos:
systemd[1]: Starting Elasticsearch... systemd[1]: Started Elasticsearch. elasticsearch[5744]: [2016-11-19 20:51:26,346][WARN ][bootstrap ] Unable to lock JVM Memory: error=12,reason=Cannot allocate memory elasticsearch[5744]: [2016-11-19 20:51:26,347][WARN ][bootstrap ] This can result in part of the JVM being swapped out. elasticsearch[5744]: [2016-11-19 20:51:26,347][WARN ][bootstrap ] Increase RLIMIT_MEMLOCK, soft limit: 65536, hard limit: 65536 elasticsearch[5744]: [2016-11-19 20:51:26,347][WARN ][bootstrap ] These can be adjusted by modifying /etc/security/limits.conf, for example: elasticsearch[5744]: # allow user 'elasticsearch' mlockall elasticsearch[5744]: elasticsearch soft memlock unlimited elasticsearch[5744]: elasticsearch hard memlock unlimited elasticsearch[5744]: [2016-11-19 20:51:26,347][WARN ][bootstrap ] If you are logged in interactively, you will have to re-login for the new limits to take effect. elasticsearch[5744]: [2016-11-19 20:51:26,628][INFO ][node ] [logstash] version[2.3.5], pid[5744], build[90f439f/2016-07-27T10:36:52Z] elasticsearch[5744]: [2016-11-19 20:51:26,628][INFO ][node ] [logstash] initializing ... elasticsearch[5744]: Exception in thread "main" java.lang.IllegalStateException: failed to load bundle [file:/srv/deployment/elasticsearch/plugins/analysis-icu/lucene-analyzers-icu-5.5.0.jar, file:/srv/deployment/elasticsearch/plugins/analysis-icu/icu4j-54.1.jar, file:/srv/deployment/elasticsearch/plugins/analysis-icu/analysis-icu-2.3.5.jar] due to jar hell elasticsearch[5744]: Likely root cause: java.util.zip.ZipException: error in opening zip file elasticsearch[5744]: at java.util.zip.ZipFile.open(Native Method) elasticsearch[5744]: at java.util.zip.ZipFile.<init>(ZipFile.java:215) elasticsearch[5744]: at java.util.zip.ZipFile.<init>(ZipFile.java:145) elasticsearch[5744]: at java.util.jar.JarFile.<init>(JarFile.java:154) elasticsearch[5744]: at java.util.jar.JarFile.<init>(JarFile.java:91) elasticsearch[5744]: at org.elasticsearch.bootstrap.JarHell.checkJarHell(JarHell.java:174) elasticsearch[5744]: at org.elasticsearch.plugins.PluginsService.loadBundles(PluginsService.java:419) elasticsearch[5744]: at org.elasticsearch.plugins.PluginsService.<init>(PluginsService.java:129) elasticsearch[5744]: at org.elasticsearch.node.Node.<init>(Node.java:158) elasticsearch[5744]: at org.elasticsearch.node.Node.<init>(Node.java:140) elasticsearch[5744]: at org.elasticsearch.node.NodeBuilder.build(NodeBuilder.java:143) elasticsearch[5744]: at org.elasticsearch.bootstrap.Bootstrap.setup(Bootstrap.java:178) elasticsearch[5744]: at org.elasticsearch.bootstrap.Bootstrap.init(Bootstrap.java:270) elasticsearch[5744]: at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:35) elasticsearch[5744]: Refer to the log for complete error details. systemd[1]: elasticsearch.service: main process exited, code=exited, status=1/FAILURE systemd[1]: Unit elasticsearch.service entered failed state.
# apt-cache policy elasticsearch
elasticsearch:
Installed: 2.3.5
Candidate: 2.3.5
Version table:
*** 2.3.5 0
1001 http://apt.wikimedia.org/wikimedia/ jessie-wikimedia/thirdparty amd64 Packages
100 /var/lib/dpkg/status
1.6.2+dfsg-1~bpo8+1 0
100 http://mirrors.wikimedia.org/debian/ jessie-backports/main amd64 Packages
1.0.3+dfsg-5+deb8u1 0
500 http://security.debian.org/ jessie/updates/main amd64 Packageshttps://gerrit.wikimedia.org/r/operations/software/elasticsearch/plugins is at c5de449cf10e5935cae7e58276c86abb4af1b351 "Upgrade elasticsearch plugins to 2.3.5 - T145404".