Similar to https://phabricator.wikimedia.org/T252913 ; the logstash-oss package on apifeatureusage* is currently configured to use the bundled OpenJDK instead of the system packages shipped by Debian.
And the bundled version is also very outdated; it's bundles openjdk 11.0.8 from July 2020, while the Debian packages ships 11.0.14 (so misses the last six Java security releases).
Change 759757 had a related patch set uploaded (by Cwhite; author: Cwhite):
[operations/puppet@production] logstash: use java home from profile::java
Change 759751 had a related patch set uploaded (by Cwhite; author: Cwhite):
[operations/puppet@production] opensearch: use java_home from profile::java
Change 759757 merged by Cwhite:
[operations/puppet@production] logstash: use java home from profile::java
Is another restart of the logstash service needed? On apifeatureusage1001 logstash still uses the bundled Java?
jmm@apifeatureusage1001:~$ ps aux | grep jdk logstash 27706 90.1 61.0 4868196 2466300 ? SNsl Feb08 867:45 /usr/share/logstash/jdk/bin/java -Xms2G -Xmx2G -XX:CMSInitiatingOccupancyFraction=75 -XX:+UseCMSInitiatingOccupancyOnly -XX:+DisableExplicitGC -Djava.awt.headless=true -Dfile.encoding=UTF-8 -Dlog4j2.formatMsgNoLookups=true -XX:+HeapDumpOnOutOfMemoryError -javaagent:/usr/share/java/prometheus/jmx_prometheus_javaagent.jar=10.64.48.107:7800:/etc/prometheus/logstash_jmx_exporter.yaml -cp /usr/share/logstash/logstash-core/lib/jars/animal-sniffer-annotations-1.14.jar:/usr/share/logstash/logstash-core/lib/jars/checker-compat-qual-2.0.0.jar:/usr/share/logstash/logstash-core/lib/jars/commons-codec-1.14.jar:/usr/share/logstash/logstash-core/lib/jars/commons-compiler-3.1.0.jar:/usr/share/logstash/logstash-core/lib/jars/commons-logging-1.2.jar:/usr/share/logstash/logstash-core/lib/jars/error_prone_annotations-2.1.3.jar:/usr/share/logstash/logstash-core/lib/jars/google-java-format-1.1.jar:/usr/share/logstash/logstash-core/lib/jars/gradle-license-report-0.7.1.jar:/usr/share/logstash/logstash-core/lib/jars/guava-24.1.1-jre.jar:/usr/share/logstash/logstash-core/lib/jars/j2objc-annotations-1.1.jar:/usr/share/logstash/logstash-core/lib/jars/jackson-annotations-2.9.10.jar:/usr/share/logstash/logstash-core/lib/jars/jackson-core-2.9.10.jar:/usr/share/logstash/logstash-core/lib/jars/jackson-databind-2.9.10.4.jar:/usr/share/logstash/logstash-core/lib/jars/jackson-dataformat-cbor-2.9.10.jar:/usr/share/logstash/logstash-core/lib/jars/janino-3.1.0.jar:/usr/share/logstash/logstash-core/lib/jars/javassist-3.26.0-GA.jar:/usr/share/logstash/logstash-core/lib/jars/jruby-complete-9.2.13.0.jar:/usr/share/logstash/logstash-core/lib/jars/jsr305-1.3.9.jar:/usr/share/logstash/logstash-core/lib/jars/log4j-api-2.13.3.jar:/usr/share/logstash/logstash-core/lib/jars/log4j-core-2.13.3.jar:/usr/share/logstash/logstash-core/lib/jars/log4j-jcl-2.13.3.jar:/usr/share/logstash/logstash-core/lib/jars/log4j-slf4j-impl-2.13.3.jar:/usr/share/logstash/logstash-core/lib/jars/logstash-core.jar:/usr/share/logstash/logstash-core/lib/jars/org.eclipse.core.commands-3.6.0.jar:/usr/share/logstash/logstash-core/lib/jars/org.eclipse.core.contenttype-3.4.100.jar:/usr/share/logstash/logstash-core/lib/jars/org.eclipse.core.expressions-3.4.300.jar:/usr/share/logstash/logstash-core/lib/jars/org.eclipse.core.filesystem-1.3.100.jar:/usr/share/logstash/logstash-core/lib/jars/org.eclipse.core.jobs-3.5.100.jar:/usr/share/logstash/logstash-core/lib/jars/org.eclipse.core.resources-3.7.100.jar:/usr/share/logstash/logstash-core/lib/jars/org.eclipse.core.runtime-3.7.0.jar:/usr/share/logstash/logstash-core/lib/jars/org.eclipse.equinox.app-1.3.100.jar:/usr/share/logstash/logstash-core/lib/jars/org.eclipse.equinox.common-3.6.0.jar:/usr/share/logstash/logstash-core/lib/jars/org.eclipse.equinox.preferences-3.4.1.jar:/usr/share/logstash/logstash-core/lib/jars/org.eclipse.equinox.registry-3.5.101.jar:/usr/share/logstash/logstash-core/lib/jars/org.eclipse.jdt.core-3.10.0.jar:/usr/share/logstash/logstash-core/lib/jars/org.eclipse.osgi-3.7.1.jar:/usr/share/logstash/logstash-core/lib/jars/org.eclipse.text-3.5.101.jar:/usr/share/logstash/logstash-core/lib/jars/reflections-0.9.11.jar:/usr/share/logstash/logstash-core/lib/jars/slf4j-api-1.7.25.jar org.logstash.Logstash --path.settings /etc/logstash jmm@apifeatureusage1001:~$ /usr/share/logstash/jdk/bin/java --version openjdk 11.0.8 2020-07-14 OpenJDK Runtime Environment AdoptOpenJDK (build 11.0.8+10) OpenJDK 64-Bit Server VM AdoptOpenJDK (build 11.0.8+10, mixed mode)
Change 761384 had a related patch set uploaded (by Cwhite; author: Cwhite):
[operations/puppet@production] logstash: set JAVA_HOME to system java runtime
Change 761384 merged by Cwhite:
[operations/puppet@production] logstash: set JAVA_HOME to system java runtime
Hah, looks like LS_JAVA_HOME is new in 7.16. Older versions use JAVA_HOME. Thanks for verifying!
Rolled out a patch to support both.
Change 759751 merged by Cwhite:
[operations/puppet@production] opensearch: use java_home from profile::java