In T294414 we realized that having the Istio sidecar proxy machinery would simplify a lot the configuration of our InferenceService resources. This task tracks a spike to test if it is feasible to turn on the TLS mesh in our ml-serve-{eqiad,codfw} clusters.
We have never tried this road since we thought that cfssl-manager + PKI were both needed, but in reality this is not true. The istiod pod creates a root CA (self signed) to manage certificates in the mesh if nothing is specified, so we could try to use it before other roads. The upstream docs suggest to use an intermediate PKI CA dedicated to istiod, but in our case our cluster is separated from the rest and it will host very similar services (all InferenceService based), so the security gain of using an extra intermediate CA doesn't seem big. In the future we could swap the self signed root CA for a dedicated PKI intermediate, it should be easy enough (if we want).