Page MenuHomePhabricator

"agent": false crashes Logstash.
Closed, ResolvedPublicSecurity

Description

Logstash crashes upon encountering an api-feature-usage log message setting "agent": false

{
  "message": "Pipeline worker error, the pipeline will be stopped",
  "exception": {
    "metaClass": {
      "metaClass": {
        "exception": "Java::OrgJrubyExceptions::NoMethodError",
        "backtrace": [
          "usr.share.logstash.vendor.bundle.jruby.$2_dot_5_dot_0.gems.logstash_minus_filter_minus_useragent_minus_3_dot_3_dot_2_minus_java.lib.logstash.filters.useragent.filter(/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/logstash-filter-useragent-3.3.2-java/lib/logstash/filters/useragent.rb:111)",
          "usr.share.logstash.logstash_minus_core.lib.logstash.filters.base.do_filter(/usr/share/logstash/logstash-core/lib/logstash/filters/base.rb:159)",
          "usr.share.logstash.logstash_minus_core.lib.logstash.filters.base.multi_filter(/usr/share/logstash/logstash-core/lib/logstash/filters/base.rb:178)",
          "org.jruby.RubyArray.each(org/jruby/RubyArray.java:1821)",
          "usr.share.logstash.logstash_minus_core.lib.logstash.filters.base.multi_filter(/usr/share/logstash/logstash-core/lib/logstash/filters/base.rb:175)",
          "org.logstash.config.ir.compiler.AbstractFilterDelegatorExt.multi_filter(org/logstash/config/ir/compiler/AbstractFilterDelegatorExt.java:134)",
          "RUBY.start_workers(/usr/share/logstash/logstash-core/lib/logstash/java_pipeline.rb:299)"
        ],
        "thread": "#<Thread:0x40bdb6b8@/usr/share/logstash/logstash-core/lib/logstash/java_pipeline.rb:129 sleep>"
      }
    }
  },
  "pipeline_id": "main",
  "error": "(NoMethodError) undefined method `empty?' for false:FalseClass"
}

To restore logs flow, we disabled usage of the useragent filter plugin. Our usage is:

filter {
  if [type] == "mediawiki" {
    if [channel] == "api-feature-usage" {
      useragent {
        source => "agent"
        prefix => "ua_"
        id => "filter/useragent/mw/parse
      }
    }
  }
}

Hosts previously emitting "agent": "" are now somehow being emitted as "agent": false. (logstash)

Creating as security issue because it's a Logstash DoS vector.

Details

Risk Rating
Medium
Author Affiliation
WMF Technology Dept

Event Timeline

colewhite updated the task description. (Show Details)

This is mitigated now on the Logstash side by mutating the field into a string. Puppetized and deployed to both the logging and api-feature-usage pipelines.

sbassett subscribed.

@colewhite - Can we make this task public now? Thanks.

@colewhite - Can we make this task public now? Thanks.

Yes, we have mitigated the issue sufficiently. Marking resolved on our end so it doesn't linger about waiting for action from upstream.

sbassett changed Author Affiliation from N/A to WMF Technology Dept.
sbassett changed the visibility from "Custom Policy" to "Public (No Login Required)".
sbassett changed the edit policy from "Custom Policy" to "All Users".
sbassett changed Risk Rating from N/A to Medium.