Page MenuHomePhabricator

illegal_argument_exception
Open, MediumPublic

Description

We are trying to stream Push-Notification-Service logs to elasticsearch, and we initially were able to

https://logstash-next.wikimedia.org/goto/02c8f00c4a4d36524e74d0f47b4796ef

But there are no logs after the 4th of Sept, even though the application is logging.

We found that elasticsearch has been unable to index:

https://logstash-next.wikimedia.org/goto/7ca65191c2e4fe0089d56c4353596e9f

[2020-09-04T14:16:52,000][WARN ][logstash.outputs.elasticsearch] Could not index event to Elasticsearch. {:status=>400, :action=>["index", {:_id=>nil, :_index=>"logstash-2020.09.04", :_type=>"fcm_send_failed", :_routing=>nil}, 2020-09-04T14:16:49.719Z kubestage1002 500: fcm_send_failed], :response=>{"index"=>{"_index"=>"logstash-2020.09.04", "_type"=>"fcm_send_failed", "_id"=>"AXRZeiZoNoG2jwpw1wTA", "status"=>400, "error"=>{"type"=>"illegal_argument_exception", "reason"=>"[detail] is defined as an object in mapping [fcm_send_failed] but this name is already used for a field in other types"}}}}

How can we fix it? Thank you!

Event Timeline

jijiki triaged this task as Medium priority.Wed, Sep 9, 3:18 PM
jijiki created this task.
Restricted Application added a subscriber: Aklapper. · View Herald Transcript
JMeybohm updated the task description. (Show Details)Wed, Sep 9, 3:23 PM

Translated in english, the message [detail] is defined as an object in mapping [fcm_send_failed] but this name is already used for a field in other types" essentially means that detail is being sent as a different type (an object in this case) than previously defined/seen field of the same name (i.e. detail as a string).

This is unfortunately known and one of the current limitations we have with json logs from disparate producers when field types clash (an instance/manifestation of this problem T180051). A short term mitigation would be to drop the detail object altogether (i.e. "pull" its members up one level of nesting) or rename the field or indeed push it as a string if that makes sense

HTH!

lmata moved this task from Inbox to Radar on the observability board.Mon, Sep 14, 3:24 PM