Page MenuHomePhabricator

Limit CPU consumption of blazegraph
Closed, ResolvedPublic


As investigated on T200563: on the public WDQS cluster, we have peaks of high CPU usage by blazegraph, depending on the number and type of queries being running. This leads to CPU exhaustion and all kind of bad things happen (NIC dropping packets, starvation of other processes, ...).

It is not clear how we can achieve this. The obvious solution would be to have bounded threadpools in blazegrahp, but that's not possible. We might be able to restrict the number of active queries, or the number of concurrent HTTP requests, but it isn't clear that the CPU cost is similar enough between queries / requests. Both because CPU vs IO can vary for different query types (a regex based query would be more CPU intensive) and because Blazegraph parallelize query execution, and the level of parallelism might be different for different queries.

Side note: the ability to constraint resource usage should be a criteria if we evaluate an alternative to Blazegraph.