Page MenuHomePhabricator

Ingestion errors for production logs on ELK7
Open, HighPublic

Description

I went through the exercise of checking all indexing errors reported on elk7 for the latest 30 days, some of these issues are shared with production pipeline and some are specific to elk7 type conflicts in default logstash mapping (those marked with "mapping")

  1. mediawiki: "failed to parse field [index] of type [long] in document"
  2. mediawiki: "object mapping for [response] tried to parse field [response] as object, but found a concrete value" T239458
  3. mediawiki: "failed to parse field [errors] of type [text] in document with"
  4. mediawiki: "failed to parse field [status] of type [long] in document with"
  5. mediawiki: "failed to parse field [response] of type [text] in document with"
  6. mediawiki: "object mapping for [errors] tried to parse field [errors] as object, but found a concrete value"
  7. restbase: "failed to parse field [res.body] of type [text] in document" T239090
  8. restbase: "object mapping for [err.body] tried to parse field [body] as object, but found a concrete value" T239090
  9. restbase: "Can't merge a non object mapping [res.body] with an object mapping [res.body]" T239090
  10. restbase: "mapper [res.body] of different type, current_type [text], merged_type [ObjectMapper]" T239090
  11. restbase: "object mapping for [res.body] tried to parse field [body] as object, but found a concrete value" T239090
  12. syslog: "Can't merge a non object mapping [err.headers.x-wm-req.backend] with an object mapping [err.headers.x-wm-req.backend]"
  13. logstash: (recommendation api) "failed to parse field [error] of type [text] in document with"
  14. logstash: "object mapping for [error.body] tried to parse field [body] as object, but found a concrete value"
  15. logstash: "object mapping for [error] tried to parse field [error] as object, but found a concrete value"
  16. logstash: "object mapping for [request] tried to parse field [request] as object, but found a concrete value"
  17. (mapping) deploy: "failed to parse field [exception.code] of type [long] in document"
  18. (mapping) mediawiki: "failed to parse field [filter_id] of type [long] in document"
  19. (mapping) mediawiki: "mapper [user_id] of different type, current_type [text], merged_type [long]"
  20. (mapping) mediawiki: "failed to parse field [lag] of type [boolean] in document with"
  21. (mapping) mediawiki: "failed to parse field [lag] of type [float] in document with"
  22. (mapping) mediawiki: "failed to parse field [lag] of type [long] in document with"
  23. (mapping) mediawiki: "failed to parse field [code] of type [long] in document with"
  24. (mapping) mediawiki: "failed to parse field [globalId] of type [long] in document with"
  25. (mapping) mediawiki: "mapper [elasticsearch_requests.max_score] cannot be changed from type [float] to [long]"
  26. (mapping) mediawiki: "failed to parse field [user_id] of type [long] in document with"
  27. (mapping) mediawiki: "failed to parse field [indexType] of type [boolean] in document with"
  28. (mapping) logstash: "mapper [@version] of different type, current_type [long], merged_type [text]"
  29. (mapping) logstash: "mapper [@version] of different type, current_type [text], merged_type [long]"
  30. (mapping) logstash: "mapper [err.args] of different type, current_type [text], merged_type [long]"
  31. (mapping) logstash: "mapper [timestamp] of different type, current_type [text], merged_type [date]"
  32. (mapping) logstash: "mapper [timestamp] of different type, current_type [long], merged_type [date]"
  33. (mapping) logstash: "failed to parse date field [Thu Dec 12 23:59:03.450851 2019] with format [strict_date_optional_time||epoch_millis]"
  34. (mapping) logstash: "failed to parse field [timestamp] of type [long] in document"

The final query I used is below, link: https://logstash.wikimedia.org/goto/d02810decd49d3ccaac4a146f55a6432

"Could not index event to Elasticsearch" AND host:logstash102* AND  NOT ("failed to parse field [timestamp] of type [long] in document" OR "object mapping for [error.body] tried to parse field [body] as object, but found a concrete value" OR  "object mapping for [err.body] tried to parse field [body] as object, but found a concrete value" OR "mapper [@version] of different type, current_type [text], merged_type [long]" OR "Can't merge a non object mapping [res.body] with an object mapping [res.body]" OR "failed to parse field [error] of type [text] in document with" OR "Can't merge a non object mapping [err.headers.x-wm-req.backend] with an object mapping [err.headers.x-wm-req.backend]" OR "failed to parse field [indexType] of type [boolean] in document with" OR "failed to parse field [user_id] of type [long] in document with" OR "object mapping for [error] tried to parse field [error] as object, but found a concrete value" OR "failed to parse field [lag] of type [long] in document with" OR "mapper [err.args] of different type, current_type [text], merged_type [long]" OR "failed to parse field [status] of type [long] in document with" OR "object mapping for [res.body] tried to parse field [body] as object, but found a concrete value" OR "mapper [res.body] of different type, current_type [text], merged_type [ObjectMapper]" OR "mapper [@version] of different type, current_type [long], merged_type [text]" OR "failed to parse field [lag] of type [float] in document with" OR "mapper [elasticsearch_requests.max_score] cannot be changed from type [float] to [long]" OR  "failed to parse field [globalId] of type [long] in document with" OR "object mapping for [request] tried to parse field [request] as object, but found a concrete value" OR "failed to parse field [response] of type [text] in document with" OR  "failed to parse field [code] of type [long] in document with" OR "object mapping for [errors] tried to parse field [errors] as object, but found a concrete value" OR "failed to parse field [lag] of type [boolean] in document with" OR "failed to parse date field" OR "mapper [timestamp] of different type, current_type [long], merged_type [date]" OR "mapper [user_id] of different type, current_type [text], merged_type [long]" OR "mapper [timestamp] of different type, current_type [text], merged_type [date]" OR "failed to parse field [filter_id] of type [long] in document" OR "failed to parse field [exception.code] of type [long] in document" OR "failed to parse field [timestamp] of type [long] in document" OR "failed to parse field [res.body] of type [text] in document" OR "failed to parse field [index] of type [long] in document" OR"object mapping for [response] tried to parse field [response] as object, but found a concrete value" OR "failed to parse field [errors] of type [text] in document with" )

Details

Related Gerrit Patches:

Event Timeline

fgiunchedi updated the task description. (Show Details)Dec 13 2019, 2:03 PM
fgiunchedi updated the task description. (Show Details)Dec 13 2019, 2:09 PM
fgiunchedi updated the task description. (Show Details)
fgiunchedi updated the task description. (Show Details)Dec 13 2019, 2:13 PM

Change 558043 had a related patch set uploaded (by Filippo Giunchedi; owner: Filippo Giunchedi):
[operations/puppet@production] prometheus: bump Logstash Elasticsearch indexing failures thresholds

https://gerrit.wikimedia.org/r/558043

Change 558043 merged by Filippo Giunchedi:
[operations/puppet@production] prometheus: bump Logstash Elasticsearch indexing failures thresholds

https://gerrit.wikimedia.org/r/558043

jcrespo triaged this task as High priority.Dec 26 2019, 11:11 AM
jcrespo added a subscriber: jcrespo.

This seems high importance, feel free to tune down if necessary.

fgiunchedi moved this task from Backlog to Doing on the User-fgiunchedi board.Mon, Feb 10, 10:25 AM