https://logstash-beta.wmflabs.org/ is mostly empty.
Looking for events from hosts `deployment-logstash2` or `deployment-logstash03` , the logstash process there started failing around 2019-08-16 13:25
The first error I found:
```
[2019-08-16T13:26:00,667][ERROR][logstash.plugins.registry] Problems loading a plugin with
{:type=>"filter", :name=>"truncate", :path=>"logstash/filters/truncate", :error_message=>"NameError", :error_class=>NameError, :error_backtrace=>["/usr/share/logstash/logstash-core/lib/logstash/plugins/registry.rb:226:in `namespace_lookup'",
"/usr/share/logstash/logstash-core/lib/logstash/plugins/registry.rb:162:in `legacy_lookup'",
"/usr/share/logstash/logstash-core/lib/logstash/plugins/registry.rb:138:in `lookup'",
"/usr/share/logstash/logstash-core/lib/logstash/plugins/registry.rb:180:in `lookup_pipeline_plugin'",
"/usr/share/logstash/logstash-core/lib/logstash/plugin.rb:140:in `lookup'",
"/usr/share/logstash/logstash-core/lib/logstash/pipeline.rb:103:in `plugin'",
"(eval):157:in `initialize'",
"org/jruby/RubyKernel.java:1079:in `eval'",
"/usr/share/logstash/logstash-core/lib/logstash/pipeline.rb:75:in `initialize'",
"/usr/share/logstash/logstash-core/lib/logstash/pipeline.rb:165:in `initialize'",
"/usr/share/logstash/logstash-core/lib/logstash/agent.rb:296:in `create_pipeline'",
"/usr/share/logstash/logstash-core/lib/logstash/agent.rb:95:in `register_pipeline'",
"/usr/share/logstash/logstash-core/lib/logstash/runner.rb:313:in `execute'",
"/usr/share/logstash/vendor/bundle/jruby/1.9/gems/clamp-0.6.5/lib/clamp/command.rb:67:in `run'",
"/usr/share/logstash/logstash-core/lib/logstash/runner.rb:204:in `run'",
"/usr/share/logstash/vendor/bundle/jruby/1.9/gems/clamp-0.6.5/lib/clamp/command.rb:132:in `run'",
"/usr/share/logstash/lib/bootstrap/environment.rb:71:in `(root)'"]}
```
For recent errors:
```
[2019-09-17T17:24:33,329][INFO ][logstash.inputs.udp ] Starting UDP listener {:address=>"0.0.0.0:11514"}
[2019-09-17T17:24:28,325][ERROR][logstash.inputs.udp ] UDP listener died {
:exception=>#<Errno::EADDRINUSE: Address already in use - bind - Address already in use>,
:backtrace=>[
"org/jruby/ext/socket/RubyUDPSocket.java:161:in `bind'",
"/usr/share/logstash/vendor/bundle/jruby/1.9/gems/logstash-input-udp-3.3.4/lib/logstash/inputs/udp.rb:116:in `udp_listener'",
"/usr/share/logstash/vendor/bundle/jruby/1.9/gems/logstash-input-udp-3.3.4/lib/logstash/inputs/udp.rb:68:in `run'",
"/usr/share/logstash/logstash-core/lib/logstash/pipeline.rb:470:in `inputworker'",
"/usr/share/logstash/logstash-core/lib/logstash/pipeline.rb:463:in `start_input'"]
}
```
rsyslog is already listening on 0.0.0.0:11514
deployment-logstash2 is the instance serving https://logstash-beta.wmflabs.org/
https://horizon.wikimedia.org/project/instances/15f1bf2e-30e0-487a-ab2a-f6fcc3e4c170/
Puppet classes:
role::logstash::apifeatureusage
role::logstash::collector prometheus_nodes: hiera('prometheus_nodes', [])
role::logstash::puppetreports
role::labs::lvm::srv
role::kafka::logging
role::kibana auth_type: None; vhost: None; auth_realm: undef; auth_file: undef; ldap_authurl: undef; ldap_binddn: undef; require_ssl: true; ldap_groups: []; serveradmin: None
profile:elastalert
Hiera:
```
elasticsearch::graylog_hosts: none
elasticsearch::version: 5
profile::elasticsearch::common_settings:
auto_create_index: +logstash-*,-*
cluster_hosts:
- deployment-logstash2.deployment-prep.eqiad.wmflabs
cluster_name: labs-logstash-eqiad
expected_nodes: 1
heap_memory: 5G
minimum_master_nodes: 1
recover_after_nodes: 1
recover_after_time: 1m
short_cluster_name: labs-logstash-eqiad
unicast_hosts:
- deployment-logstash2.deployment-prep.eqiad.wmflabs
profile::kafka::broker::kafka_cluster_name: logging-eqiad
profile::kafka::broker::ssl_enabled: true
profile::kafka::broker::ssl_password: qwerty
profile::logstash::collector::jmx_exporter_port: 7801
puppetmaster: deployment-puppetmaster03.deployment-prep.eqiad.wmflabs
```