Change #1327059 had a related patch set uploaded (by Dpogorzelski; author: Dpogorzelski):

[operations/docker-images/production-images@master] kserve: update images to 0.20

https://gerrit.wikimedia.org/r/1327059

Change #1327059 merged by Dpogorzelski:

[operations/docker-images/production-images@master] kserve: update images to 0.20

https://gerrit.wikimedia.org/r/1327059

Change #1327557 had a related patch set uploaded (by Dpogorzelski; author: Dpogorzelski):

[operations/deployment-charts@master] kserve: 0.20 upstream alignment

https://gerrit.wikimedia.org/r/1327557

Change #1329506 had a related patch set uploaded (by Dpogorzelski; author: Dpogorzelski):

[operations/deployment-charts@master] kserve: add LLMInferenceService support

https://gerrit.wikimedia.org/r/1329506

Change #1331389 had a related patch set uploaded (by Dpogorzelski; author: Dpogorzelski):

[operations/deployment-charts@master] kserve: expose LLMInferenceService via istio

https://gerrit.wikimedia.org/r/1331389

LLMInferenceService

A second resource type is available on ml-staging-codfw, in the experimental namespace: LLMInferenceService.

It runs vLLM against a model directory in Swift. The vLLM image is shared, so a model needs no image from the inference-services repository. You declare the model path and the resources in the values file.

InferenceService is unchanged. Both types work in the same namespace.

Differences from InferenceService

Consequences:

1. :predict does not exist. The paths are /v1/models, /v1/chat/completions, /v1/completions and /v1/embeddings. A client that uses :predict must change.
2. The name must be unique in the namespace. An LLMInferenceService and an InferenceService with one name would claim the same hostname.
3. Only /openai/v1/ is reachable from outside the cluster. /metrics is not.
4. The pods carry no Istio sidecar.

How to deploy a model

Step 1. Put the model in Swift, under s3://wmf-ml-models/.

Step 2. Add the model to helmfile.d/ml-services/experimental/values-llm.yaml.

llm_inference_services:
  llm-my-model:
    model:
      uri: s3://wmf-ml-models/llm/my-model/20260101120000/
      name: my-model          # the name that clients send in the "model" field
    replicas: 1
    template:
      containers:
        - name: main
          resources:
            requests:
              cpu: "4"
              memory: 16Gi
              ephemeral-storage: 20Gi
              amd.com/gpu: "1"
            limits:
              cpu: "8"
              memory: 32Gi
              ephemeral-storage: 30Gi
              amd.com/gpu: "1"
          env:
            - name: VLLM_ADDITIONAL_ARGS
              value: "--max-model-len 16384 --gpu-memory-utilization 0.90"

Some rough edges

In theory KServe can route an LLMInferenceService with the Gateway API, through spec.router. It can also use the Gateway API Inference Extension for LLM-aware routing but we currently use neither. The Gateway API switch looks busy and the Inference Extension needs Istio 1.27 or later. ml-serve runs Istio 1.24.2 and this is shared infrastructure across WMF.

The chart writes an Istio VirtualService instead. These are the consequences.

Consequences of no Gateway API

• No route attachment control. The Gateway API gives the gateway owner control, with allowedRoutes and ReferenceGrant. Istio has no equivalent. The deploy-kserve role now permits virtualservices, so all 18 ml-serve namespaces can claim any hostname on the shared gateway. One has to be careful to not use same service name twice.
• No conflict rules and no status. An HTTPRoute has defined precedence rules, and it reports a condition if the gateway rejects it. Two VirtualServices that claim one hostname give an undefined result, and no object shows the problem. You must read the gateway access log.

Consequences of no Inference Extension

You can give an LLM model more than one replica. It works. The controller creates the pods, the Service gets more endpoints, and Istio balances the requests across them. Every request gets an answer.

What you do not get is LLM-aware routing. The Inference Extension adds an endpoint picker. The picker uses data that a general-purpose balancer does not have:

• which replica holds the prompt prefix in its KV cache;
• the queue depth and the KV cache use of each replica;
• which replica has a given LoRA adapter in memory;
• a priority class, so batch traffic does not block interactive traffic.

Without the picker:

1. The prefill work repeats. A request goes to any replica. If another replica already holds the prompt prefix in its KV cache, that cache is not used. The new replica computes the prefill again. This costs GPU time, and it increases the time to the first token. The cost grows with long system prompts, with RAG context, and with multi-turn chat.
2. Saturation is invisible. A replica with a full KV cache and a deep queue looks the same as an idle replica. The cost of an LLM request varies by a large factor, so a wrong choice costs much more than for normal HTTP traffic.
3. No LoRA routing and no priority classes. Request priority is the one item that also applies to a single replica, because it acts before the dispatch of the request. We cannot shed or delay low-priority traffic today.

I think it's a good baseline to test and then try to switch to gateway api otherwise it's way too many moving parts at once

This sounds awesome, thanks for the summary! So iiuc we can't benefit any of the goodies that llm-d offers (kv cache aware routing etc) untill we use the Gateway api. Still, this seems much nicer and cleaner and reusing the base vllm image is great (our docker registry will be very happy)

yea exactly, it doesn't hurt but there is no application specific aware routing because we can't support Inference Extension in this setup so can't take advantage of some goodies. still, it's a jump forward not having to build the docker container and just ship the model. also a lot of the benefits that come with Inference Extension apply only when you run more than one replica, with 1 replica it shouldn't make difference

Change #1327557 merged by Dpogorzelski:

[operations/deployment-charts@master] kserve: 0.20 upstream alignment

https://gerrit.wikimedia.org/r/1327557

Change #1329506 merged by Dpogorzelski:

[operations/deployment-charts@master] kserve: add LLMInferenceService support

https://gerrit.wikimedia.org/r/1329506

Change #1331389 merged by Dpogorzelski:

[operations/deployment-charts@master] kserve: expose LLMInferenceService via istio

https://gerrit.wikimedia.org/r/1331389

We're waiting on Machine-Learning-Team to confirm that we can move forward.

@DPogorzelski-WMF thank you for working on this. We've followed the steps in T433973#12264511 and deployed an llmisvc but can't access it the same way we access other LiftWing services. Please see details in: T439659#12380045.

ml-serve-codfw is updated. will do eqiad next