https://logstash-beta.wmflabs.org/ is mostly empty. If you've come here looking how to find log events on beta... You can ssh to deployment-fluorine02.deployment-prep.eqiad.wmflabs!
Looking for events from hosts deployment-logstash2 or deployment-logstash03 , the logstash process there started failing around 2019-08-16 13:25
The first error I found:
[2019-08-16T13:26:00,667][ERROR][logstash.plugins.registry] Problems loading a plugin with {:type=>"filter", :name=>"truncate", :path=>"logstash/filters/truncate", :error_message=>"NameError", :error_class=>NameError, :error_backtrace=>["/usr/share/logstash/logstash-core/lib/logstash/plugins/registry.rb:226:in `namespace_lookup'", "/usr/share/logstash/logstash-core/lib/logstash/plugins/registry.rb:162:in `legacy_lookup'", "/usr/share/logstash/logstash-core/lib/logstash/plugins/registry.rb:138:in `lookup'", "/usr/share/logstash/logstash-core/lib/logstash/plugins/registry.rb:180:in `lookup_pipeline_plugin'", "/usr/share/logstash/logstash-core/lib/logstash/plugin.rb:140:in `lookup'", "/usr/share/logstash/logstash-core/lib/logstash/pipeline.rb:103:in `plugin'", "(eval):157:in `initialize'", "org/jruby/RubyKernel.java:1079:in `eval'", "/usr/share/logstash/logstash-core/lib/logstash/pipeline.rb:75:in `initialize'", "/usr/share/logstash/logstash-core/lib/logstash/pipeline.rb:165:in `initialize'", "/usr/share/logstash/logstash-core/lib/logstash/agent.rb:296:in `create_pipeline'", "/usr/share/logstash/logstash-core/lib/logstash/agent.rb:95:in `register_pipeline'", "/usr/share/logstash/logstash-core/lib/logstash/runner.rb:313:in `execute'", "/usr/share/logstash/vendor/bundle/jruby/1.9/gems/clamp-0.6.5/lib/clamp/command.rb:67:in `run'", "/usr/share/logstash/logstash-core/lib/logstash/runner.rb:204:in `run'", "/usr/share/logstash/vendor/bundle/jruby/1.9/gems/clamp-0.6.5/lib/clamp/command.rb:132:in `run'", "/usr/share/logstash/lib/bootstrap/environment.rb:71:in `(root)'"]}
For recent errors:
[2019-09-17T17:24:33,329][INFO ][logstash.inputs.udp ] Starting UDP listener {:address=>"0.0.0.0:11514"} [2019-09-17T17:24:28,325][ERROR][logstash.inputs.udp ] UDP listener died { :exception=>#<Errno::EADDRINUSE: Address already in use - bind - Address already in use>, :backtrace=>[ "org/jruby/ext/socket/RubyUDPSocket.java:161:in `bind'", "/usr/share/logstash/vendor/bundle/jruby/1.9/gems/logstash-input-udp-3.3.4/lib/logstash/inputs/udp.rb:116:in `udp_listener'", "/usr/share/logstash/vendor/bundle/jruby/1.9/gems/logstash-input-udp-3.3.4/lib/logstash/inputs/udp.rb:68:in `run'", "/usr/share/logstash/logstash-core/lib/logstash/pipeline.rb:470:in `inputworker'", "/usr/share/logstash/logstash-core/lib/logstash/pipeline.rb:463:in `start_input'"] }
rsyslog is already listening on 0.0.0.0:11514
deployment-logstash2 is the instance serving https://logstash-beta.wmflabs.org/
https://horizon.wikimedia.org/project/instances/15f1bf2e-30e0-487a-ab2a-f6fcc3e4c170/
Puppet classes:
role::logstash::apifeatureusage
role::logstash::collector prometheus_nodes: hiera('prometheus_nodes', [])
role::logstash::puppetreports
role::labs::lvm::srv
role::kafka::logging
role::kibana auth_type: None; vhost: None; auth_realm: undef; auth_file: undef; ldap_authurl: undef; ldap_binddn: undef; require_ssl: true; ldap_groups: []; serveradmin: None
profile:elastalert
Hiera:
elasticsearch::graylog_hosts: none elasticsearch::version: 5 profile::elasticsearch::common_settings: auto_create_index: +logstash-*,-* cluster_hosts: - deployment-logstash2.deployment-prep.eqiad.wmflabs cluster_name: labs-logstash-eqiad expected_nodes: 1 heap_memory: 5G minimum_master_nodes: 1 recover_after_nodes: 1 recover_after_time: 1m short_cluster_name: labs-logstash-eqiad unicast_hosts: - deployment-logstash2.deployment-prep.eqiad.wmflabs profile::kafka::broker::kafka_cluster_name: logging-eqiad profile::kafka::broker::ssl_enabled: true profile::kafka::broker::ssl_password: qwerty profile::logstash::collector::jmx_exporter_port: 7801 puppetmaster: deployment-puppetmaster03.deployment-prep.eqiad.wmflabs