The following is a write-up of what I see as possible stumbling blocks for services on LiftWing using gRPC in addition to our current HTTP(S)/REST mode of operation. Note that none of these concerns are deal-breakers or insurmountable, but rather are aspects that need addressing, and may prove to be more than just 30m of work, due to the complexities of Istio, kserve and k8s in general.

One key difference between gRPC and our existing REST usage is that for gRPC, being specified on top of HTTP/2, is always using TLS, i.e. there is no HTTP (plain text) equivalent in gRPC (or HTTP/2).

For LW services making outgoing gRPC requests, the details that need to be addressed are:

• we currently use Istio's Envoy proxy to do TLS termination, even for outgoing connections. Specifically, LW services only talk plain HTTP and Istio transparently does TLS on behalf of the services. The upside of this is that we can use Istio metrics (and logs, via Logstash) to get insight into the rate of e.g. HTTP status codes (2xx vs. 5xxx etc). With TLS being mandatory, this insight goes away, and we would need to get the same information from the LW services somehow.
• since the LW services atm use plain HTTP, they need not care about TLS certificates (client or server). With gRPC, we would have to make sure that the LW services have access to the relevant CA information that is used for the certs of the services they talk to.
• It is unclear (to me) if Envoy needs to be (re)configured to allow HTTP/2 (as opposed to just 1.0 and 1.1), but I suspect this is at worst a minor config change

For LW services acting as servers that provide gRPC endpoints:

• similar to the client situation, the TLS termination for HTTP/1.x services is currently done by Istio. If we want to do end-to-end gRPC, we would need to issue TLS certificates for services and more importantly, get them into the serving pods/containers for use by the actual isvc.
• again similar to the client mode above, any metrics we currently collect from Istio that rely on traffic inspection (endpoints, status codes etc) would go away and would need to be gathered from the isvc directly.
• there may be a way for Istio to do the TLS termination here, i.e. the isvc would never see TLS traffic just like now, but there are questions about the feasibility of this (can all gRPC requests be translated into REST?) as well as the question if there would even be a point to this, since using gRPC may be motivated by features that need to be carried end-to-end (contracts/transactions).

For both modes, there is also a minor question of performance, in that isvcs would now handle the TLS overhead. While Istio then would be relieved of this, we may need to adjust resources for high-traffic services. Similar, e2e latency behavior may change. I don't suspect either of these would be major issues, but let's keep it in mind.

There likely also are "unknown unknowns", in the sense of gRPC on LW being entirely new territory, so it is likely that things we're not even aware of will break or behave in an unexpected manner.

Finally, ML team so far has no experience in running gRPC services at all, so there would definitely be a learning curve, both in the deployment/development stage, as well as learning the ropes of debugging gRPC services, especially if the metrics we are used to are now coming from elsewhere (or may not even exist in the beginning); ditto for dashboards, alerts, SLO metrics and the like.

Hey! Adding a few notes/thoughts:

In T421903#11778065, @klausman wrote:

For LW services making outgoing gRPC requests, the details that need to be addressed are:

• we currently use Istio's Envoy proxy to do TLS termination, even for outgoing connections. Specifically, LW services only talk plain HTTP and Istio transparently does TLS on behalf of the services. The upside of this is that we can use Istio metrics (and logs, via Logstash) to get insight into the rate of e.g. HTTP status codes (2xx vs. 5xxx etc). With TLS being mandatory, this insight goes away, and we would need to get the same information from the LW services somehow.

In theory this shouldn't be a problem from what I can read, kserve running grpc ingress can call HTTP services in egress using the same transparent proxy functionality (keeping metrics etc..).
In the event that kserve needs to call grpc in egress, IIUC the same HTTP to HTTPs trick that we use could happen via h2c -> http2 (to have metrics etc..). No idea how metrics would look like, we'd need to test it.

• since the LW services atm use plain HTTP, they need not care about TLS certificates (client or server). With gRPC, we would have to make sure that the LW services have access to the relevant CA information that is used for the certs of the services they talk to.

This shouldn't be an issue, kserve would still use istio but in grpc/http/2 mode so the TLS certs will be handled by cert-manager transparently.

• It is unclear (to me) if Envoy needs to be (re)configured to allow HTTP/2 (as opposed to just 1.0 and 1.1), but I suspect this is at worst a minor config change

I think it should happen automatically via istio configs.

For LW services acting as servers that provide gRPC endpoints:

• similar to the client situation, the TLS termination for HTTP/1.x services is currently done by Istio. If we want to do end-to-end gRPC, we would need to issue TLS certificates for services and more importantly, get them into the serving pods/containers for use by the actual isvc.

Same as above, this bit will be taken care by istio itself afaics.

• again similar to the client mode above, any metrics we currently collect from Istio that rely on traffic inspection (endpoints, status codes etc) would go away and would need to be gathered from the isvc directly.
• there may be a way for Istio to do the TLS termination here, i.e. the isvc would never see TLS traffic just like now, but there are questions about the feasibility of this (can all gRPC requests be translated into REST?) as well as the question if there would even be a point to this, since using gRPC may be motivated by features that need to be carried end-to-end (contracts/transactions).

What do you have in mind here? Istio should support grpc and it should be able to provide metrics for it. Do you have an example in mind?

For both modes, there is also a minor question of performance, in that isvcs would now handle the TLS overhead. While Istio then would be relieved of this, we may need to adjust resources for high-traffic services. Similar, e2e latency behavior may change. I don't suspect either of these would be major issues, but let's keep it in mind.

There likely also are "unknown unknowns", in the sense of gRPC on LW being entirely new territory, so it is likely that things we're not even aware of will break or behave in an unexpected manner.

True, there are surely some nuances that we don't know yet, like adapting the istio configs.

Finally, ML team so far has no experience in running gRPC services at all, so there would definitely be a learning curve, both in the deployment/development stage, as well as learning the ropes of debugging gRPC services, especially if the metrics we are used to are now coming from elsewhere (or may not even exist in the beginning); ditto for dashboards, alerts, SLO metrics and the like.

Here it depends how much hoarde will pay off in the future with caching and general availability.

To clarify a basic assumption I have: gRPC only works over HTTP/2, and HTTP/2 is always TLS-encrypted, i.e. there is no way to speak gRPC over a plaintext connection, or at least not with the standard libraries for gRPC. If that is not the case, things might be simpler (or way more complex ;) ).

In T421903#11805086, @elukey wrote:

In T421903#11778065, @klausman wrote:

For LW services making outgoing gRPC requests, the details that need to be addressed are:

• we currently use Istio's Envoy proxy to do TLS termination, even for outgoing connections. Specifically, LW services only talk plain HTTP and Istio transparently does TLS on behalf of the services. The upside of this is that we can use Istio metrics (and logs, via Logstash) to get insight into the rate of e.g. HTTP status codes (2xx vs. 5xxx etc). With TLS being mandatory, this insight goes away, and we would need to get the same information from the LW services somehow.

In theory this shouldn't be a problem from what I can read, kserve running grpc ingress can call HTTP services in egress using the same transparent proxy functionality (keeping metrics etc..).
In the event that kserve needs to call grpc in egress, IIUC the same HTTP to HTTPs trick that we use could happen via h2c -> http2 (to have metrics etc..). No idea how metrics would look like, we'd need to test it.

So to clarify, this would still look like a normal JSON-request-in-http to the actual isvc, and Istio would transmogrify that into a gRPC request? Wouldn't that mean that Istio needs a .proto file so it knows what REST parameters to turn into what gRPC fields?

Or does the isvc make binary-blob-like requests in protobuf shape over plain HTTP and then they get TLS-ified by Envoy? I was under the impression that the former (using plain HTTP) is not supported by most gRPC libraries, hence my statement above.

• since the LW services atm use plain HTTP, they need not care about TLS certificates (client or server). With gRPC, we would have to make sure that the LW services have access to the relevant CA information that is used for the certs of the services they talk to.

This shouldn't be an issue, kserve would still use istio but in grpc/http/2 mode so the TLS certs will be handled by cert-manager transparently.

So isvc<->Istio would be TLS, right? And Envoy looks transparent by way of having a TLS cert of the CN/SAN of the service, and the isvc trusts it because it trusts the certificate manager?

For LW services acting as servers that provide gRPC endpoints:

• similar to the client situation, the TLS termination for HTTP/1.x services is currently done by Istio. If we want to do end-to-end gRPC, we would need to issue TLS certificates for services and more importantly, get them into the serving pods/containers for use by the actual isvc.

Same as above, this bit will be taken care by istio itself afaics.

The question here would be if the isvc would expect to receive some TLS certificate to load into its http/2 server, since HTTP/2 is TLS-only, and how the certificate would be injected into the isvc. The Istio side (i.e. what the external client of such a service sees) is likely to work automagically just like it is happening with HTTP/1.1+TLS at the moment.

• again similar to the client mode above, any metrics we currently collect from Istio that rely on traffic inspection (endpoints, status codes etc) would go away and would need to be gathered from the isvc directly.
• there may be a way for Istio to do the TLS termination here, i.e. the isvc would never see TLS traffic just like now, but there are questions about the feasibility of this (can all gRPC requests be translated into REST?) as well as the question if there would even be a point to this, since using gRPC may be motivated by features that need to be carried end-to-end (contracts/transactions).

What do you have in mind here? Istio should support grpc and it should be able to provide metrics for it. Do you have an example in mind?

The question is whether Istio can see e.g. the HTTP status codes etc. This hinges on it doing a "TLS added and removed here" MITM-kind splitting of the TLS stream. If it does not split open the stream, it can't see the payload and we wouldn't have the metrics as it is now.

In T421903#11819290, @klausman wrote:

To clarify a basic assumption I have: gRPC only works over HTTP/2, and HTTP/2 is always TLS-encrypted, i.e. there is no way to speak gRPC over a plaintext connection, or at least not with the standard libraries for gRPC. If that is not the case, things might be simpler (or way more complex ;) ).

In T421903#11805086, @elukey wrote:

In T421903#11778065, @klausman wrote:

For LW services making outgoing gRPC requests, the details that need to be addressed are:

• we currently use Istio's Envoy proxy to do TLS termination, even for outgoing connections. Specifically, LW services only talk plain HTTP and Istio transparently does TLS on behalf of the services. The upside of this is that we can use Istio metrics (and logs, via Logstash) to get insight into the rate of e.g. HTTP status codes (2xx vs. 5xxx etc). With TLS being mandatory, this insight goes away, and we would need to get the same information from the LW services somehow.

In theory this shouldn't be a problem from what I can read, kserve running grpc ingress can call HTTP services in egress using the same transparent proxy functionality (keeping metrics etc..).
In the event that kserve needs to call grpc in egress, IIUC the same HTTP to HTTPs trick that we use could happen via h2c -> http2 (to have metrics etc..). No idea how metrics would look like, we'd need to test it.

So to clarify, this would still look like a normal JSON-request-in-http to the actual isvc, and Istio would transmogrify that into a gRPC request? Wouldn't that mean that Istio needs a .proto file so it knows what REST parameters to turn into what gRPC fields?

Or does the isvc make binary-blob-like requests in protobuf shape over plain HTTP and then they get TLS-ified by Envoy? I was under the impression that the former (using plain HTTP) is not supported by most gRPC libraries, hence my statement above.

I think we are not understanding each other, let me try to explain. In my head, there are two metrics:

1. Ingress HTTP responses - this will be replaced by http2+grpc by Istio, and the isvc python code will need to be updated accordingly. I am not 100% sure what metrics kserve exposes when grpc is used for that, but I am reasonably sure they are not none.
2. Egress service calls - here we use the http -> https egress origination trick, and the isvc will keep using HTTP for those, not grpc.

So we should figure out what metrics are provided for gRPC by Kserve with a poc before closing a task, this is what I am saying.

• since the LW services atm use plain HTTP, they need not care about TLS certificates (client or server). With gRPC, we would have to make sure that the LW services have access to the relevant CA information that is used for the certs of the services they talk to.

This shouldn't be an issue, kserve would still use istio but in grpc/http/2 mode so the TLS certs will be handled by cert-manager transparently.

So isvc<->Istio would be TLS, right? And Envoy looks transparent by way of having a TLS cert of the CN/SAN of the service, and the isvc trusts it because it trusts the certificate manager?

My understanding is that it will be plaintext grpc, not using any encryption (because it will be a localhost data exchange), so you won't need any extra TLS certificate.

For LW services acting as servers that provide gRPC endpoints:

• similar to the client situation, the TLS termination for HTTP/1.x services is currently done by Istio. If we want to do end-to-end gRPC, we would need to issue TLS certificates for services and more importantly, get them into the serving pods/containers for use by the actual isvc.

Same as above, this bit will be taken care by istio itself afaics.

The question here would be if the isvc would expect to receive some TLS certificate to load into its http/2 server, since HTTP/2 is TLS-only, and how the certificate would be injected into the isvc. The Istio side (i.e. what the external client of such a service sees) is likely to work automagically just like it is happening with HTTP/1.1+TLS at the moment.

HTTP/2 can work also in plaintext mode with h2c, that I believe is used for the "localhost" Istio/envoy to isvc communications, but it would need to be tested. Kserve docs doesn't seem to talk about any extra TLS cert needed.

• again similar to the client mode above, any metrics we currently collect from Istio that rely on traffic inspection (endpoints, status codes etc) would go away and would need to be gathered from the isvc directly.
• there may be a way for Istio to do the TLS termination here, i.e. the isvc would never see TLS traffic just like now, but there are questions about the feasibility of this (can all gRPC requests be translated into REST?) as well as the question if there would even be a point to this, since using gRPC may be motivated by features that need to be carried end-to-end (contracts/transactions).

What do you have in mind here? Istio should support grpc and it should be able to provide metrics for it. Do you have an example in mind?

The question is whether Istio can see e.g. the HTTP status codes etc. This hinges on it doing a "TLS added and removed here" MITM-kind splitting of the TLS stream. If it does not split open the stream, it can't see the payload and we wouldn't have the metrics as it is now.

My understanding is that Istio will provide the same metrics, but related to grpc calls (Failed, successful, etc..) and also probably using the HTTP/2 response codes as well (but I am not sure to what extent). Again this is a good thing to test in a poc :)

My recommendation would be to test an isvc in grpc mode, even a basic one, and then make a summary of pros/cons/limitations/etc.. It shouldn't take much, I can help with the Istio bits if needed!

After a clarifying chat with Luca about the intricacies of gRPC, HTTP/2 etc, I now have a better picture of what will need building (probably :) )

The key insight was that you can do plaintext communications for gRPC, i.e. it is possible to run a gRPC service without ever needing an TLS certificate. This means that our current setup of having Istio handle all TLS stuff can be replicated for gRPC.

There still are of course some questions about the specific configuration of Istio, how to make kserve offer gRPC and the monitoring/metrics side, but I feel those are much more predictable in how much time it will take to solve them.

IIUC, Bartosz has already run a gRPC-enable version of an isvc locally, so maybe he can weigh in on the stuff that is needed on the side of the isvc.

My recommendation would be to test an isvc in grpc mode, even a basic one, and then make a summary of pros/cons/limitations/etc.. It shouldn't take much, I can help with the Istio bits if needed!

Totally agreed, that should be our next step.

To speak on enabling gRPC for ISVC, our plan would be to use the Kserve's V2 Inference Protocol, which supports both gRPC and HTTP/REST interfaces. Currently, all our services were built with V1 protocol in mind, which only supports HTTP/REST interface.

The good part is that Kserve registers both V1 and V2 endpoints by default, so all our services already expose all V1 HTTP, V2 HTTP and V2 gRPC endpoints. I checked this locally by running the article topic via docker-compose, uvicorn binds HTTP on 8080 and kserve's gRPC server binds 8081, both are reachable.

What's required for V2 to work with our ISVC code is slightly stronger typing on inputs and outputs. Currently, our services just get plain untyped dict on input and respond with untyped dict as well, which works for V1. For V2, we need to handle a little stronger types of InferRequest and InferResponse, but the changes can be quite minimal - there is already a WIP patch from Ilias showing how could this look like: https://gerrit.wikimedia.org/r/c/machinelearning/liftwing/inference-services/+/1254844. Here V1 HTTP stays fully functional along V2 so everything is backwards compatible.

What is left is making sure we can support this on K8S side. From my understanding, there are a few things here outside of the Istio/Envoy issues discussed above:

• AFAIK the biggest concern is that Knative layer has a single-port constraint as Knative revision routes to only one container port, either http1 or h2c. This probably means that we have to go running 2 ISVCs simultaneously(?) from the same docker image. This would mean using 2x the resources, but having cache means we don't have to scale that much.
• The wmf-stable/kserve-inference hardcodes a single HTTP port (main_app.port: 8080 in values.yaml), we'd need to add a knob for declaring a service as gRPC.
• We'd need the gRPC port added to ingress rules to NetworkPolicy
• For cross-cluster communication, Istio gateway would also need a gRPC listener(?) so that Hoarde could reach it.

We’ve filed T424049: k8s changes needed to allow article topic (and other future isvcs) to use the kserve v2 inference protocol (and gRPC), which aims to address the minimum Kubernetes changes required to support gRPC requests to the articletopic service