Got these indexing failures from restbase today while API was slow:
[2019-11-25T11:35:17,571][WARN ][logstash.outputs.elasticsearch] Could not index event to Elasticsearch. {:status=>400, :action=>["index", {:_id=>nil, :_index=>"logstash-restbase-2019.11.25", :_type=>"restbase", :_routing=>nil}, 2019-11-25T11:35:17.195Z restbase2013 503: http_error], :response=>{"index"=>{"_index"=>"logstash-restbase-2019.11.25", "_type"=>"restbase", "_id"=>"AW6iWCpDarkxubcmNpxo", "status"=>400, "error"=>{"type"=>"mapper_parsing_exception", "reason"=>"object mapping for [res.body] tried to parse field [body] as object, but found a concrete value"}}}}I am assuming that's because res.body has been sent as a string in the offending log while normally (?) it is a nested object, like this:
{
"_index": "logstash-restbase-2019.11.25",
"_type": "restbase",
"_id": "AW6iXidmarkxubcmN-mu",
"_score": 1,
"_source": {
"res": {
"name": "HTTPError",
"headers": {
"x-request-id": "8e2c9500-0f78-11ea-b73b-974fc939a41f",
"server": "restbase1021",
"x-webkit-csp": "default-src 'none'; frame-ancestors 'none'",
"access-control-allow-headers": "accept, content-type, content-length, cache-control, accept-language, api-user-agent, if-match, if-modified-since, if-none-match, dnt, accept-encoding",
"x-frame-options": "SAMEORIGIN",
"access-control-allow-methods": "GET,HEAD",
"access-control-expose-headers": "etag",
"access-control-allow-origin": "*",
"content-security-policy": "default-src 'none'; frame-ancestors 'none'",
"x-content-type-options": "nosniff",
"x-xss-protection": "1; mode=block",
"x-content-security-policy": "default-src 'none'; frame-ancestors 'none'",
"referrer-policy": "origin-when-cross-origin",
"content-location": "https://en.wikipedia.org/api/rest_v1/page/summary/crusted%20buttercream%20icing",
"content-type": "application/problem+json",
"cache-control": "private, max-age=0, s-maxage=0, must-revalidate"
},
"message": "ETIMEDOUT",
"body": {
"internalErr": "ETIMEDOUT",
"internalURI": "https://api-rw.discovery.wmnet/w/api.php",
"internalMethod": "post",
"detail": "ETIMEDOUT",
"type": "internal_http_error",
"internalStack": "Error: ETIMEDOUT\n at Timeout.setTimeout (/srv/deployment/restbase/deploy-cache/revs/b98706885857b8c2621271e987e59274aa231ff5/node_modules/preq/index.js:15:27)\n at ontimeout (timers.js:386:14)\n at tryOnTimeout (timers.js:250:5)\n at Timer.listOnTimeout (timers.js:214:5)"
},
"status": 504
...Note that we've seen conflicts happening on both res and body in T240667 namely:
"object mapping for [res.body] tried to parse field [body] as object, but found a concrete value" "failed to parse field [res.body] of type [text] in document" "object mapping for [err.body] tried to parse field [body] as object, but found a concrete value" "Can't merge a non object mapping [res.body] with an object mapping [res.body]"