Page MenuHomePhabricator

Mjolnir kafka msearch daemon should scale up/down based on elasticsearch load
Closed, DeclinedPublic

Description

Currently the msearch daemons are restricted primarily with static limits on various things that when multiplied across the daemons results in a reasonable, if conservative, load.

It seems the daemon could be more aggressive though, these limits don't take into account the complexity of individual queries and only work because it the current values have been tuned for the result we want. Instead it seems like we could auto-scale the number of workers up over time, and gracefully handle thread pool rejection by scaling back and re-sending.

Event Timeline

Looked into this a little, it's probably not worth the effort. If it was user facing maybe, but as something that will primarily be triggered through an oozie workflow I'm inclined to skip this.