This is the follow up task for T287007.
Already merged changes regarding this:
- custom_deploy: Add istio manifest for main clusters
- admin_ng: Support managing of system namespaces with helmfile
- admin_ng/main: Create istio-system namespace
Those changes, together with a WIP one, made it possible to generally install istio to staging-codfw but there are still some open questions/things to fix:
- Decide how we want the kube-apiserver to reach webooks running inside of the cluster, see: T290967
- Create proper NetworkPolicies, ideally automated or global, for Ingress-Gateway to be able to reach services etc.
- Figure out how to deal with the internal ca that istio manages. It is by default used to secure communication with itsiod as well as establish trust between the Ingress-Gateway and services.
- Make Ingress-Gateway trust Puppet-CA (e.g. tls-proxy) certificates
- Make prometheus scrape istiod and Ingress-Gateway
- Decide on how we want to run the Ingress-Gateway and ultimately how we want PyBal to healthcheck it/the k8s nodes.
I'm keeping some additional, unordered notes at https://wikitech.wikimedia.org/wiki/User:JMeybohm/Kubernetes/Ingress