Change Details

We run all search queries with both server side and client side timeouts. Unfortunately twice now (T172653 and T195491) we have found edge cases where queries can run 10+ minutes on a shard. We don't currently have any good way of identifying they exist, they have been noticed either because the cluster was struggling due to the added load of 10+ minute queries, or because a search engineer noticed a query running longer than expected by chance. Elasticsearch has slow logging functionality, we should look into how we can make this visible.

We run all search queries with both server side and client side timeouts. Unfortunately multiple times now (T152895#2872562, T172653 and T195491) we have found edge cases where queries can run 10+ minutes on a shard. We don't currently have any good way of identifying they exist, they have been noticed either because the cluster was struggling due to the added load of 10+ minute queries, or because a search engineer noticed a query running longer than expected by chance. Elasticsearch has slow logging functionality, we should look into how we can make this visible.

We run all search queries with both server side and client side timeouts. Unfortunately twicemultiple times now (T152895#2872562, T172653 and T195491) we have found edge cases where queries can run 10+ minutes on a shard. We don't currently have any good way of identifying they exist, they have been noticed either because the cluster was struggling due to the added load of 10+ minute queries, or because a search engineer noticed a query running longer than expected by chance. Elasticsearch has slow logging functionality, we should look into how we can make this visible.