Page MenuHomePhabricator

Gitlab Sidekiq mapper parsing exceptions since 2021-11-15@1825
Closed, ResolvedPublic

Description

It appears the format changed.

Could not index event to Elasticsearch. status: 400, action: ["index", {:_id=>nil, :_index=>"ecs-1.7.0-5-default-2021.46", :routing=>nil, :_type=>"_doc"}, #<LogStash::Event:0x71d7f90e>], response: {"index"=>{"_index"=>"ecs-1.7.0-5-default-2021.46", "_type"=>"_doc", "_id"=>"a4jeJH0B1RMACsoStrkT", "status"=>400, "error"=>{"type"=>"mapper_parsing_exception", "reason"=>"failed to parse field [http.response.status_code] of type [long] in document with id 'a4jeJH0B1RMACsoStrkT'. Preview of field's value: 'done'", "caused_by"=>{"type"=>"illegal_argument_exception", "reason"=>"For input string: \"done\""}}}}

Event Timeline

Change 739018 had a related patch set uploaded (by Cwhite; author: Cwhite):

[operations/puppet@production] logstash: amend gitlab sidekiq log mutation to use new format

https://gerrit.wikimedia.org/r/739018

Change 739018 merged by Cwhite:

[operations/puppet@production] logstash: reconstruct gitlab sidekiq message field

https://gerrit.wikimedia.org/r/739018

colewhite claimed this task.

Reformatting the logs has eliminated the source of mapping errors.