Page MenuHomePhabricator

Puppet error on deployment-logtash03
Closed, ResolvedPublic

Description

Error: /Stage[main]/Logstash/Service[logstash]/ensure: change from stopped to running failed: Systemd start for logstash failed!

krenair@deployment-logstash03:~$ sudo service logstash start
Failed to start logstash.service: Unit logstash.service not found.

from logstash's init.pp:

# Older 1.x versions of logstash needed this file deployed,
# but 5.x comes with a sensible service definition for systemd
# in /etc/systemd/logstash.service
file { '/lib/systemd/system/logstash.service':
    ensure  => absent,
}

service { 'logstash':
    ensure     => running,
    provider   => systemd,
    enable     => true,
    hasstatus  => true,
    hasrestart => true,
}
cannot open `/etc/systemd/logstash.service' (No such file or directory)
root@deployment-logstash03:~# dpkg -L logstash | grep systemd
/usr/share/logstash/vendor/bundle/jruby/1.9/gems/pleaserun-0.0.30/templates/systemd
/usr/share/logstash/vendor/bundle/jruby/1.9/gems/pleaserun-0.0.30/templates/systemd/default
/usr/share/logstash/vendor/bundle/jruby/1.9/gems/pleaserun-0.0.30/templates/systemd/default/program.service
/usr/share/logstash/vendor/bundle/jruby/1.9/gems/pleaserun-0.0.30/templates/systemd/default/default
/usr/share/logstash/vendor/bundle/jruby/1.9/gems/pleaserun-0.0.30/templates/systemd/default/prestart.sh
/usr/share/logstash/vendor/bundle/jruby/1.9/gems/pleaserun-0.0.30/templates/systemd-user
/usr/share/logstash/vendor/bundle/jruby/1.9/gems/pleaserun-0.0.30/templates/systemd-user/default
/usr/share/logstash/vendor/bundle/jruby/1.9/gems/pleaserun-0.0.30/templates/systemd-user/default/program.service
/usr/share/logstash/vendor/bundle/jruby/1.9/gems/pleaserun-0.0.30/templates/systemd-user/default/prestart.sh
/usr/share/logstash/vendor/bundle/jruby/1.9/gems/pleaserun-0.0.30/spec/pleaserun/platform/systemd_spec.rb
/usr/share/logstash/vendor/bundle/jruby/1.9/gems/pleaserun-0.0.30/lib/pleaserun/platform/systemd-user.rb
/usr/share/logstash/vendor/bundle/jruby/1.9/gems/pleaserun-0.0.30/lib/pleaserun/platform/systemd.rb
root@deployment-logstash03:~#
root@deployment-logstash03:~# apt-cache show logstash
Package: logstash
Priority: extra
Section: default
Installed-Size: 202586
Maintainer: <vagrant@packer-virtualbox-iso-1540737293>
Architecture: all
Version: 1:5.6.15-1
Filename: pool/thirdparty/elastic56/l/logstash/logstash_5.6.15-1_all.deb
SHA512: 3efd99d3bc56a9158cd5fe55bb694b705df6657382fe18d14a6da69cca2edef32dc6a1c7109d2f001f2d4d449927851c92f48c783a68611422da84dbb70c85e5
Size: 111596290
SHA256: b736d5bcc3e606c29965b60f3e1e7d02cdd580f21b4a89a9c8fc261f6ea7545e
SHA1: 6ca41c00cc2c9d68c12303b15dfc67dfcdb0ae18
MD5sum: d3acf7ee28fd3fd02d4caa4623edbbf7
Description: An extensible logging pipeline
Description-md5: 2cc8837023e818d510770f6e573a838a
Homepage: http://www.elasticsearch.org/overview/logstash/
License: Apache 2.0
Vendor: Elasticsearch

root@deployment-logstash03:~# apt-cache policy logstash
logstash:
  Installed: 1:5.6.15-1
  Candidate: 1:5.6.15-1
  Version table:
 *** 1:5.6.15-1 1001
       1001 http://apt.wikimedia.org/wikimedia stretch-wikimedia/thirdparty/elastic56 amd64 Packages
        100 /var/lib/dpkg/status
root@deployment-logstash03:~# curl -sI 'http://apt.wikimedia.org/wikimedia/pool/thirdparty/elastic56/l/logstash/logstash_5.6.15-1_all.deb' | grep Last-Modified
Last-Modified: Tue, 19 Mar 2019 14:52:47 GMT

Event Timeline

Krenair created this task.Aug 16 2019, 1:17 PM
Restricted Application added a subscriber: Aklapper. · View Herald TranscriptAug 16 2019, 1:17 PM
Krenair updated the task description. (Show Details)Aug 16 2019, 1:18 PM
herron added a subscriber: herron.Aug 16 2019, 1:33 PM

Not sure what caused the system to be in this state, but after the following steps logstash is back up and running.

root@deployment-logstash03:~# apt remove logstash
Reading package lists... Done
Building dependency tree
Reading state information... Done
The following packages will be REMOVED:
  logstash
0 upgraded, 0 newly installed, 1 to remove and 2 not upgraded.
After this operation, 207 MB disk space will be freed.
Do you want to continue? [Y/n]
(Reading database ... 112287 files and directories currently installed.)
Removing logstash (1:5.6.15-1) ...
root@deployment-logstash03:~# puppet agent -t
Info: Using configured environment 'production'
Info: Retrieving pluginfacts
Info: Retrieving plugin
Info: Loading facts
Info: Caching catalog for deployment-logstash03.deployment-prep.eqiad.wmflabs
Notice: /Stage[main]/Base::Environment/Tidy[/var/tmp/core]: Tidying 0 files
Info: Applying configuration version '1565961887'
Notice: openstack::clientpackages::vms::mitaka::stretch: no special configuration yet
Notice: /Stage[main]/Openstack::Clientpackages::Vms::Mitaka::Stretch/Notify[openstack::clientpackages::vms::mitaka::stretch: no special configuration yet]/message: defined 'message' as 'openstack::clientpackages::vms::mitaka::stretch: no special configuration yet'
Notice: The LDAP client stack for this host is: classic/sudoldap
Notice: /Stage[main]/Profile::Ldap::Client::Labs/Notify[LDAP client stack]/message: defined 'message' as 'The LDAP client stack for this host is: classic/sudoldap'
Notice: /Stage[main]/Logstash/Package[logstash]/ensure: created
Notice: /Stage[main]/Logstash/File[/etc/default/logstash]/content:
--- /etc/default/logstash	2019-08-16 13:25:21.857585283 +0000
+++ /tmp/puppet-file20190816-24520-1ozfy26	2019-08-16 13:25:22.449585569 +0000
@@ -1,11 +1,5 @@
-JAVACMD="/usr/bin/java"
-LS_HOME="/usr/share/logstash"
-LS_SETTINGS_DIR="/etc/logstash"
-LS_PIDFILE="/var/run/logstash.pid"
-LS_USER="logstash"
-LS_GROUP="logstash"
-LS_GC_LOG_FILE="/var/log/logstash/gc.log"
-LS_OPEN_FILES="16384"
-LS_NICE="19"
-SERVICE_NAME="logstash"
-SERVICE_DESCRIPTION="logstash"
+# Use LS_JAVA_OPTS to add jmx_exporter when launching logstash as a daemon.
+# Using /etc/logstash/jvm.options is not going to work because that file is used for all logstash jvm
+# invocations, resulting in attempts to double-bind while e.g. launching logstash-plugin.
+LS_JAVA_OPTS="-javaagent:/usr/share/java/prometheus/jmx_prometheus_javaagent.jar=172.16.1.184:7801:/etc/prometheus/logstash_jmx_exporter.yaml"
+

Info: Computing checksum on file /etc/default/logstash
Info: FileBucket got a duplicate file {md5}9078440886369167bb5a19a50f2a9782
Info: /Stage[main]/Logstash/File[/etc/default/logstash]: Filebucketed /etc/default/logstash to puppet with sum 9078440886369167bb5a19a50f2a9782
Notice: /Stage[main]/Logstash/File[/etc/default/logstash]/content:

Notice: /Stage[main]/Logstash/File[/etc/default/logstash]/content: content changed '{md5}9078440886369167bb5a19a50f2a9782' to '{md5}07441ffa2c4cb1c9b0d0d4e6f32fb8df'
Info: /Stage[main]/Logstash/File[/etc/default/logstash]: Scheduling refresh of Service[logstash]
Notice: /Stage[main]/Logstash/Service[logstash]/ensure: ensure changed 'stopped' to 'running'
Info: /Stage[main]/Logstash/Service[logstash]: Unscheduling refresh on Service[logstash]
Notice: Applied catalog in 31.69 seconds
root@deployment-logstash03:~# systemctl status logstash
● logstash.service - logstash
   Loaded: loaded (/etc/systemd/system/logstash.service; enabled; vendor preset: enabled)
   Active: active (running) since Fri 2019-08-16 13:26:36 UTC; 11s ago
 Main PID: 25408 (java)
    Tasks: 27 (limit: 4915)
   CGroup: /system.slice/logstash.service
           └─25408 /usr/bin/java -XX:+UseParNewGC -XX:+UseConcMarkSweepGC -XX:CMSInitiatingOccupancyFraction=75 -XX:+UseCMSInitiatingOccupancyOnly -XX:+DisableExplicitGC -Djava.awt.head

Aug 16 13:26:36 deployment-logstash03 systemd[1]: Started logstash.

Interesting, okay, so the file is /etc/systemd/system/logstash.service (init.pp left out the system/ part), and that doesn't seem to come from a package:
dpkg-query: no path found matching pattern /etc/systemd/system/logstash.service

Krenair closed this task as Resolved.Aug 16 2019, 1:46 PM
Krenair assigned this task to herron.