Change #1151645 had a related patch set uploaded (by Santhosh; author: Santhosh):

[machinelearning/liftwing/inference-services@main] WIP: Openvinoi model server integration

https://gerrit.wikimedia.org/r/1151645

I tried to integrate Openvino model server to liftwing. Learnings for the first iteration(see the above WIP patch):

The Opevino Model server's latest version 2025.1 require Python 3.12+ and GLIBC 2.38+. So we cannot use our bookworm images to prepare based image. Because of this, I used the Opevino model server docker images(official image provided by Intel) for this iteration.

@isarantopoulos I am not sure about the best practice for the base images in Liftwing. Please advice.

Using older versions of OpenVino is not an option as I had tried them in the past and not performant enough. It defeats the purpose.

I am working on the KServe API bridging with OpenVino server.

Regarding the KServe API and Openvino model server:

The Kserver compatiblae OVMS Rest api is documented at https://docs.openvino.ai/2025/model-server/ovms_docs_rest_api_kfs.html#inference-api
It provides APIs compatible with V2 of Kserve API standard. https://github.com/kserve/kserve/blob/master/docs/predict-api/v2/required_api.md#httprest

We are using V1 of kserve currently. V2 introduces /infer API. But this infer API accepts token ids, shape etc- basically the raw inputs to the model. To call that API, you need to calculate the model inputs - You need to do tokenization and shaping, which is not at all trivial and cannot expect API consumers to do that from their end. (Or I am completely misunderstanding the spec)

There is a problem with both v1 and v1 kserve api for LLM inference though. Both are incompatible with OpenAI API spec which is kind of standard for any LLM inference(thanks to OpenRouter and many SDKs which adopted it). OVMS recommends that API and readily provides it too under /v3 path. I got this working. It is generic - you can give roles(system, assistant, user, tool) and more options for inference. I would like to move in that direction rather than kserve's limited infer API.

At present, the following works based on my patch.

http
POST http://localhost:8080/v3/chat/completions
Content-Type: application/json

{
  "model": "Phi-4-mini-instruct-int4-ov",
  "max_tokens": 300,
  "temperature": 0,
  "stream": false,
  "messages": [
    { "role": "system", "content": "You are a helpful assistant." },
      { "role": "user", "content": "Hello, How are you!" }
    ]
}

Output(from my dev laptop: X1 Carbon 9th gen. Intel i7 CPU):

{
  "choices": [
    {
      "finish_reason": "stop",
      "index": 0,
      "logprobs": null,
      "message": {
        "content": "Hello! I'm just a computer program, so I don't have feelings, but I'm here and ready to assist you. How can I help you today?",
        "role": "assistant"
      }
    }
  ],
  "created": 1748516549,
  "model": "Phi-4-mini-instruct-int4-ov",
  "object": "chat.completion",
  "usage": {
    "prompt_tokens": 17,
    "completion_tokens": 31,
    "total_tokens": 48
  }
}

Since you people tried VLLM inference, I am sure you will have thoughts on this as well. Please let me know. @isarantopoulos , @kevinbazira

Docker images: The images that we can use as base images in WMF's production clusters have to be in production-images repo. Looking at the dockefile instructions available in the openvino repo if we wanted to use this image in prod we would have to port one of the dockerfiles created for ubuntu to start from a debian distribution. Regarding python 3.12 we could install this python version in bookworm.

v1/v2 protocol
At the moment on LiftWing we don't support v2 but we can plan the work required to do it for new services. One can however test the v2 protocol by building an image locally at the moment.
kserve v2 follows the Open Inference Protocol (OIP) which aims to become a standard across platforms (kserve, seldon, triton, openvino etc.). Looking at this kserve github page it says that openvino has also adopted this, although the link is broken so I don't know what to assume here.
Regarding the input required for v2 you can either pass the tensor directly or any other input you want. Here is an example of how a v1 request is translated to a v2 request:

v1:

curl -s localhost:8080/v2/models/articlequality/infer -X POST -d '{"rev_id": 12345, "lang": "en"}'

v2:

curl -s localhost:8080/v2/models/articlequality/infer -X POST \
  -d '{
        "inputs": [
          {
            "name": "lang",
            "shape": [1],
            "datatype": "BYTES",
            "data": ["en"]
          },
          {
            "name": "rev_id",
            "shape": [1],
            "datatype": "INT64",
            "data": [12345]
          }
        ]
      }'

Have you tried to test if v1 would work? We have been using the huggingfaceserver in kserve which supports the OpenAI protocol out of the box.

I'm not against going a different route than kserve but we would have to scope the work required to do so before taking a final decision. Apart from the docker image work there is the kubernetes side of things (helm charts, prometheus metrics)

In T395012#10867030, @santhosh wrote:

Regarding the KServe API and Openvino model server:
...
Since you people tried VLLM inference, I am sure you will have thoughts on this as well. Please let me know. @isarantopoulos , @kevinbazira

As per Ilias' comment in T395012#10874407, I ran a simple comparison between vllm and ovms using KServe's v1 protocol on ml-lab1002. The goal was to get an understanding of how ovms' setup and performance compares with vllm using a simple Kserve inference service. Here are the results

vLLM

Using the docker image built in 1146891, that supports vllm and kserve, I tested a simple isvc as shown below:

1.Set up simple vllm model-server

2.Start the vllm isvc to serve phi-4-mini-instruct on GPU

3.Query the vllm isvc

$ time curl -s localhost:8080/v1/models/simple-vllm:predict -X POST -d '{ "instances": [ { "text": "Who are you?" } ] }' -i -H "Content-type: application/json"
HTTP/1.1 200 OK
date: Mon, 02 Jun 2025 13:28:27 GMT
server: uvicorn
content-length: 152
content-type: application/json

{"predictions":[{"generated_text":"I am Phi, your assistant. I'm here to help you with any questions or tasks you have. What can I do for you today?"}]}
real	0m0.471s
user	0m0.014s
sys	0m0.000s

OVMS

Using the docker image built in 1151645, which supports ovms and kserve, I tested a simple isvc as shown below:

1.Set up simple ovms model-server

2.Start the ovms isvc to serve phi-4-mini-instruct-int8-ov on CPU

3.Query the ovms isvc

$ time curl -s localhost:8080/v1/models/simple-ovms:predict -X POST -d '{ "instances": [ { "text": "Who are you?" } ] }' -i -H "Content-type: application/json"
HTTP/1.1 200 OK
date: Mon, 02 Jun 2025 13:42:03 GMT
server: uvicorn
content-length: 44
content-type: application/json

{"predictions":[{"text":" sold the? What"}]}
real	0m1.174s
user	0m0.000s
sys	0m0.011s

My takeaways:

1.OVMS on CPU (~1.17s) is slower than vLLM on GPU (~0.47s) in this simple test. If we plan to host LLMs on CPU with ovms, we might have a performance drop.
2.Whereas vLLM can serve original models from HuggingFace, ovms requires an extra model conversion step to OpenVINO IR format (xml/bin), then place it under a version folder. This will likely require another evaluation step to ensure the converted model matches the original model's accuracy.
3.On a brighter note, we can use ovms with the KServe v1 protocol.

Thanks @kevinbazira and @isarantopoulos for these details. Very useful information.

Kevin, What you tried with openvino is low level openvino API. That code has some issues it seems. The output you got does not make sense("Sold the? What"). It could be because the input does not match with the expected input of Phi-4 which expect messages with roles. You mentioned it as OVMS. It is not OVMS

OVMS is Openvino Model Server(https://github.com/openvinotoolkit/model_server) - very similar to huggingfaceserver. I had tried the low level openvino APIs, and it works but quite hard to work since you need to configure each model with its internal architecture. I am not proposing that for this exploration. I am proposing Openvino model server setup. I image that it would be similar to our huggingface_server setup.

There won't be any python code we add, other then plugging the OVMS with an entrypoint.sh and corresponding pipeline blubber. See WIP patch https://gerrit.wikimedia.org/r/c/machinelearning/liftwing/inference-services/+/1151645

@kevinbazira I learned that ml-lab-1002 has AMD CPUs and not Intel CPUs. This is interesting information. OpenVINO is targetting Intel CPUs, NPUs and GPUs. I had heard that it will also work on AMD GPUs but this is quite undocumented. I think your experiment shows it works on AMDs too(AMD EPYC 7643P is what ml-lab-1002 uses).

The speed difference between AMD GPU and CPU is expected. As I was discussing with Sucheta and Ilias, I foresee this as inference capability where realtime computation is not required for the usecase. However, trying to expose smaller LMs for realtime usage is not out of scope in my exploration. I found BERT based models running in OVMS with my intel CPUs in ~10-20 millisecond range.

@isarantopoulos, after reading your reply, I read how huggingfaceserver is working and configured(T357986) and I think we can get OVMS working in the same way.

The huggingface server exposes openai completion API like this

curl https://inference-staging.svc.codfw.wmnet:30443/openai/v1/completions" -H "Host: llama3.experimental.wikimedia.org" -H "Content-Type: application/json" -X POST -d '{"model": "llama3", "prompt": "Write me a poem about Machine Learning.", "stream":false, "max_tokens": 50}'

And OVMS exposes it like this:

curl http://localhost:8080/v3/chat/completions -H "Content-Type: application/json" -X POST -d '{
  "model": "Phi-4-mini-instruct-int4-ov",
  "max_tokens": 300,
  "temperature": 0,
  "stream": false,
  "messages": [
    { "role": "system", "content": "You are a helpful assistant." },
      { "role": "user", "content": "Who are you?" }
  ]
}'

The only difference is openai/v1 vs v3 in the API URL part. I guess this is manageable with our proxies and and optional rewrites I guess? Output is same for both.
(ref https://docs.openvino.ai/2025/model-server/ovms_docs_genai.html). @isarantopoulos please let me know.

I have added chat, completion and embedding examples that uses the openi api in my WIP patch https://gerrit.wikimedia.org/r/c/machinelearning/liftwing/inference-services/+/1151645 - please have a look

Now, regarding Kserve v1 and v2 APIs, OVMS documents it as Tensorflow API and KServe API respectively. I could not get :predict(V1) or /infer APIs running so far. The /models API path lists my models, but both predict and infer APIs says the model name could not found. This is something I will try again. Sending the input as BYTES is documented, but that is also failing my experiments(it says not a valid type). This is also another item to figure out.

However, @isarantopoulos, do you think it is important to expose APIs other than openai api endpoint? My understanding is exposing OpenAI compatible API endpoint makes the models in OVMS immediately usable, and v1-predict or v2-infer APIs are nice to have. Please correct me if I am wrong.

One interesting feature in OVMS is the model respository and configuration uses Mediapipe graphs. So unlike huggingface, you cannot just download a model and run it. You need to convert to Openvino IR format(this is easy and takes a few minutes) and then the graphs need to be prepared(they are text files with node definitions).

@isarantopoulos, The docker image preparation based on production image as base image - I will take a look into it. But if I am stuck I will report here - I am not really good with Docker image building :-)

I am trying to build a production docker image with WMF debian bookworm+python3.11+openvino 2025.1.0+ OVMS 2025.1.0. I am referring the Ubuntu Dockerfile.

There are prebuilt binaries of Openvino for Ubuntu, Redhat, and Debian 10(buster), but not for Debian stable(bookworm). So I am trying to compile it from source on Debian. The compilations of the cpp code is quite time consuming(2 hours and still going on. So many submodules and dependencies)

Change #1153988 had a related patch set uploaded (by Santhosh; author: Santhosh):

[operations/docker-images/production-images@master] Add Openvino modelserver

https://gerrit.wikimedia.org/r/1153988

Updates: I successfully created a production docker image on top of WMF production wookworm image.

Upstream provides Dockerfile for Redhat 8 and Ubuntu 24.04 and not for Debian stable.

For the Dockerfile, I referred Ubuntu Dockerfile, but had to make several changes to get this working. I am considering to request an official Dockerfile for Debian(may be I can contribute)

Openvino and Openvino model server are written in C++ and we compile them from source code - including dependencies. They are big projects and in my development laptop, it took more than 5+ hours! Upstream provides openvino prebuilt binaries for Ubuntu 24.04 and Debian 10. But that wont work in Debian 12(bookworm) because of glibc and other
shared library version conflicts.

Took lot of time for me to prepare the final version. docker-pkg build with --use-cache helped a lot for incremental preparation of Dockerfile. It is 238 lines and perhaps the largest one in production-images repo :-)

The final image is 734MB.

I tested it with the model repository I had prepared. Everything is working as expected so far.

I will try to create a screencast or video recording showing the models in action.

Here is the screencast of everything working together: https://drive.google.com/file/d/1YDSvTm3ePv585-ittck2tYWH2XZ-AHLx/view
(MP4 video, 15mins, 88MB)

Screencast of inference alone(30seconds), this time not hindered by CPU usage of screenrecording.

https://asciinema.org/a/722578

Update: New version of Opevino and Openvino model server was released a few days ago. I updated my production bookworm based docker image patch: https://gerrit.wikimedia.org/r/c/operations/docker-images/production-images/+/1153988 to that version. This new version adds support for agentic AI(tool calling and MCP).

Since preparing the production image for bookworm is the most time taking effort, I filed a ticket in upstream to consider providing a docker image for stable debian releases- https://github.com/openvinotoolkit/model_server/issues/3452

Now that we have proof of concept, ML team need to take a decision on this so that they can include this work in the planned activities. That will help steamlining code review, and next steps.

The model repository setup is easy with the new versions. OVMS can pull models from huggingface repos and setup all configurations. I wrote a shellscript to get all listed models in it and start the server. This script is used with the official docker images of OVMS. https://gerrit.wikimedia.org/r/c/machinelearning/liftwing/inference-services/+/1151645 will require updates based on this

#!/bin/bash
# Get all the models and prepapre configuration
# Refer https://docs.openvino.ai/nightly/model-server/ovms_demos_continuous_batching_rag.html

# Configuration
MODELS_PATH="/data/scratch/santhosh/models"
USER_GROUP="$(id -u):$(id -g)"
IMAGE="openvino/model_server:latest"
BASE_CMD="docker run --user $USER_GROUP --rm -v $MODELS_PATH:/models"

# Model definitions: "model_name:task"
MODELS=(
        "OpenVINO/gemma-2b-it-int4-ov:text_generation"
        "OpenVINO/bge-base-en-v1.5-fp16-ov:embeddings"
        "OpenVINO/bge-reranker-base-fp16-ov:rerank"
        "OpenVINO/FLUX.1-schnell-int4-ov:image_generation"
)

echo "Setting up OpenVINO models..."

# Download models if they don't exist
for model_task in "${MODELS[@]}"; do
        model="${model_task%:*}"
        task="${model_task#*:}"
        model_dir="$MODELS_PATH/${model##*/}"

        if [[ ! -d "$model_dir" ]]; then
                echo "Downloading $model..."
                $BASE_CMD:rw $IMAGE --pull --model_repository_path /models --source_model "$model" --task "$task"
        else
                echo "Model $model already exists, skipping download"
        fi
done

# Add models to config
echo "Configuring models..."
for model_task in "${MODELS[@]}"; do
        model="${model_task%:*}"
        $BASE_CMD:rw $IMAGE --add_to_config /models --model_name "$model" --model_path "$model"
done

# Check if config exists before running server
if [[ -f "$MODELS_PATH/config.json" ]]; then
        echo "Starting OpenVINO model server..."
        $BASE_CMD:ro -p 9000:9000 -p 8000:8000 $IMAGE \
                --config_path /models/config.json --port 8000 --rest_port 9000
else
        echo "Error: Configuration file not found at $MODELS_PATH/config.json"
        exit 1
fi

https://gerrit.wikimedia.org/r/c/operations/docker-images/production-images/+/1153988

This patch to prepare a WMF production image is outdated again as new version of Debian is released.

Since LiftWing hosting plan is pending and under consideration by Machine-Learning-Team , I setup https://ovms.wmcloud.org/ with upstream docker image and above shellscript. Wrote an nginx proxy configuration for landing page, API key authentication and /api/v1 mounting. OpenAI api compatible APIs are available. This instance uses 8 CPU cores, 32GB RAM, 20GB Disk(models are in scratch volume). This wmcloud instance does not cancel this ticket, and for experimentation purpose.

Change #1153988 abandoned by Santhosh:

[operations/docker-images/production-images@master] Add Openvino modelserver

Reason:

Abandoning as this is old and outdated and there is no plan from ML team on this front.

https://gerrit.wikimedia.org/r/1153988

I am abandoning the patches as they are outdated and I do not plan to keep them updated with the changes mentioned above.

Change #1151645 abandoned by Santhosh:

[machinelearning/liftwing/inference-services@main] WIP: Openvino model server integration

Reason:

Abandoning as this is old and outdated and there is no plan from ML team on this front.

https://gerrit.wikimedia.org/r/1151645