Istio can be configured with native ingress resources, using the annotation:

kubernetes.io/ingress.class: istio

see https://istio.io/latest/docs/tasks/traffic-management/ingress/kubernetes-ingress/

The other quirk is that secret added to the ingress will need to be located in the istio namespace (which isn't that surprising, but a deviation from the norm, and also means we need root intervention whenever we want to add an TLS-terminated ingress).

Of course, istio also offers its own custom resource definitions for a richer configuration:

the istio gateway (https://istio.io/latest/docs/reference/config/networking/gateway/) provides what looks like a simplified configuration for envoy, more or less, using three types of CRDs:

• Gateway to expose the ports we want to listen to
• VirtualService to indicate the routing prefixes

but I don't think we really need it at this point in time; we might as well wait for the new api gateway interface in newer kubernetes installations, that will probably add more features support.

Metrics can easily be collected with prometheus - in fact, istio ships with the correct annotations and thus should easily be picked up by our prometheus without adding any new rule.

Change 719265 had a related patch set uploaded (by JMeybohm; author: JMeybohm):

[operations/deployment-charts@master] custom_deploy: Add istio manifest for main clusters

https://gerrit.wikimedia.org/r/719265

Change 719265 merged by jenkins-bot:

[operations/deployment-charts@master] custom_deploy: Add istio manifest for main clusters

https://gerrit.wikimedia.org/r/719265

A few questions/points:

• I see a bullet point TLS certificates need to be placed in the namespace of istio-ingressgateway and a comment by @Joe above, but it doesn't clarify it to me. Does this mean that certs AND private keys get placed in the namespace as Secrets/Configmaps?
• Care to add an example of Just plain host and path prefix matching (1st case) vs Host and path (prefix and exact) matching (2nd and 3rd case).
• Interesting that the "Medium" has "Role model" for binding restrictions while "Advanced" doesn't. Is that correct?
• Piggybacking on the work already done by the ML-team is indeed a huge contributing factor for choosing istio over other ingresses that have way smaller complexity (e.g. linkerd, contour, etc).
• We 've talked about this already in some channels, but stating it here as well for completeness. My take is that we go as simple as possible on this one. Debugging on kubernetes can already be a difficult task due to complexities of the architecture, adding istio will only add to that complexity. Let's keep the extra things added as lean as possible. So, I guess that means choice #1 for now.

In T287007#7354949, @akosiaris wrote:

A few questions/points:

• I see a bullet point TLS certificates need to be placed in the namespace of istio-ingressgateway and a comment by @Joe above, but it doesn't clarify it to me. Does this mean that certs AND private keys get placed in the namespace as Secrets/Configmaps?

Yes, exactly that. Cert and private key for all hostnames we want to use via Ingress need to be placed in the istio-ingressgateway Namespace as secrets. We already have those as ConfigMaps in the namespaces of the services, though.

In T287007#7354949, @akosiaris wrote:

• Care to add an example of Just plain host and path prefix matching (1st case) vs Host and path (prefix and exact) matching (2nd and 3rd case).

For 1st case a definition like path: /foo will always be treated as path: /foo.*. For 2nd and 3rd we may choose to have an exact match only instead ( path: /foo != path: /foo.+ ).

In T287007#7354949, @akosiaris wrote:

• Interesting that the "Medium" has "Role model" for binding restrictions while "Advanced" doesn't. Is that correct?

I was surprised as well, but I did not find anything regarding that in Istio.

In T287007#7354949, @akosiaris wrote:

• We 've talked about this already in some channels, but stating it here as well for completeness. My take is that we go as simple as possible on this one. Debugging on kubernetes can already be a difficult task due to complexities of the architecture, adding istio will only add to that complexity. Let's keep the extra things added as lean as possible. So, I guess that means choice #1 for now.

Agreed. It's also important to add that we could mix and match (assuming we add the required CRDs for the 2nd option) if we see the need. Although I would suppose not to to that to much as that will diverge the way services are configured quite heavily (adding even more complexity).