Page MenuHomePhabricator

Failed to index error logs from SwiftFileBackend::doStoreInternal
Closed, DeclinedPublic

Description

Found when looking into T226937: Not possible to server-side upload certain images: "An unknown error occurred in storage backend "local-swift-eqiad"" looks like ATM messages from doStoreInternal fail to index

	[2019-06-27T22:49:00,307][WARN ][logstash.outputs.elasticsearch] Could not index event to Elasticsearch. {:status=>400, :action=>["index", {:_id=>nil, :_index=>"logstash-2019.06.27", :_type=>"mediawiki", :_routing=>nil}, 2019-06-27T22:48:59.000Z mwmaint1002 HTTP 100 (Continue) in 'SwiftFileBackend::doStoreInternal' (given '{"async":false,"src":"./Hurtigruten.05.11.1920x1080.NRK2.webm","dst":"mwstore://local-swift-eqiad/local-public/3/30/Hurtigruten.05.11.1920x1080.NRK2.webm","overwrite":true,"headers":{"x-content-dimensions":""},"dstExists":false}'): (curl error: 28) Timeout was reached], :response=>{"index"=>{"_index"=>"logstash-2019.06.27", "_type"=>"mediawiki", "_id"=>"AWubILNzQFnOyvY0oCnA", "status"=>400, "error"=>{"type"=>"illegal_argument_exception", "reason"=>"[err] is defined as a field in mapping [mediawiki] but this name is already used for an object in other types"}}}}

Filing for reference, this is unfortunately a long standing issue with field types and their explosion in our logstash indices

Event Timeline

Restricted Application added subscribers: Liuxinyu970226, Aklapper. · View Herald TranscriptJul 1 2019, 10:46 AM
fgiunchedi closed this task as Declined.Dec 9 2019, 11:38 AM

We've separated indices now so this specific error has been resolved, there are other logging conflicts still left of course.

Restricted Application removed a subscriber: Liuxinyu970226. · View Herald TranscriptDec 9 2019, 11:38 AM