Per parent ticket, we are in the process of enabling envoy TLS termination on our OpenSearch clusters. This represents a large step forward in terms of observability and control at layer 7, but we there are also some risks. Creating this ticket to figure out:
- How big is the performance penalty from double TLS termination? We aren't going to roll back just because of this, but we might consider beefing up our Envoy pods.
The penalty seems to be around 1m to 7ms, but it is not uniform.
It seems to be that any ingressgateway pod can do the TLS termination, including those running on nodes where there is no opensearch master pod running.
When the ingressgateway pod has analysed the payload and made a routing decision based on the HTTP request, this is then sent back to any opensearch master pod, which could be on another machine.
It might be possible to mitigate this by using the locality based load-balancing features of istio, with a sub-zone label on the pods to keep the upstream traffic to the same rack.
However, since the performance penalty is relatively small, this might not be necessary.
- TLS Certificates
- Will OpenSearch and/or Envoy refuse to connect when the pod certificates expire? Answer: yes The current version of the opensearch-cluster helm chart forces us to terminate TLS at the pod level. It also doesn't support hot reloading of certificates. If we put an Envoy-hosted certificate in front of the pods, our current TLS probes will no longer be able to see the expiration date of the pod certificates. Without monitoring, it's much more likely we'll forget to renew the certificates and cause an outage.
- Does the new version of the chart force you to use TLS on the HTTP API like the old version? If not, we can disable TLS on the HTTP API (ref T414217 ) and avoid double TLS termination.
We should not roll out Envoy TLS termination to production clusters until we have satisfactory answers to the above questions.
