As a wdqs user I would like servers that are lagged to be depooled so that I don't experience stale results.
We (wdqs maintainers) often have to depool wdqs servers manually because they are heavily lagged, this has several drawbacks:
- it relies on a manual intervention
- the operator that depooled the server in the first place must remember to repool the server once the lag is back to acceptable values
AC:
- a server should be automatically depooled if the lag reached a certain threshold (re-use the same threshold used by icinga?)
- a server should be automatically repooled when its lag is back to normal values
- do not automatically depool more than what we currently can serve