Page MenuHomePhabricator

Audit paths for making requests to elasticsearch to ensure they have timeouts set.
Closed, ResolvedPublic

Description

We should be setting both server side and client side timeouts on all requests made to the elasticsearch cluster. While digging through hive logs related to a recent cluster instability problem (T142506) I found that requests made through CirrusSearch\Searcher::findNamespace don't have any explicit timeout set, it eventually times out after 5 (!!!) minutes. This should be fixed, and we should double check the rest of the places that make requests to ensure everything has an appropriate timeout. We could set a default client side timeout in CirrusSearch\Connection::__construct but we also want to set the server side timeout (with the 'timeout' query parameter).

Might also be worth looking into if we can set default server side timeouts more reasonably as well.

Event Timeline

Restricted Application added projects: Discovery, Discovery-Search. · View Herald TranscriptAug 15 2016, 5:06 PM
Restricted Application added a subscriber: Aklapper. · View Herald Transcript
EBernhardson updated the task description. (Show Details)
EBernhardson updated the task description. (Show Details)
Gehel added a subscriber: Gehel.Aug 17 2016, 8:50 AM

Change 306306 had a related patch set uploaded (by EBernhardson):
[WIP] Ensure timeouts are always used

https://gerrit.wikimedia.org/r/306306

Change 306306 merged by jenkins-bot:
Ensure timeouts are always used

https://gerrit.wikimedia.org/r/306306

debt closed this task as Resolved.Sep 1 2016, 8:44 PM