Update kserve to 0.15.2
Closed, DeclinedPublic8 Estimated Story Points
Actions

Assigned To

None

Authored By

	isarantopoulos
	Jun 10 2024, 12:16 PM

Project Tags

Referenced Files

	F67626690: build.log
	Oct 27 2025, 9:23 AM

Subscribers

Description

[ This task was initially aimed to upgrade to 0.13.0 but we have changed the scope to target a more recent version ]
We want to upgrade kserve to 0.15.2 but before we want to go through the changelog and investigate if there are any breaking changes that will required us to update our infrastructure (e.g. update istio).
As part of this task we will attempt to upgrade the kserve python package in huggingfaceserver which doesn't support any production traffic as well as revertrisk and document any findings.

We need to assess what is the risk (if any) of running a different version of the chart vs the python package and plan for the chart upgrade as well.

Complete list of info: https://github.com/kserve/kserve/releases/tag/v0.15.2

Details

Related Changes in Gerrit:

Subject	Author	Repo	Branch	Lines +/-
logo-detection: update kserve to 0.14.1	Ilias Sarantopoulos	machinelearning/liftwing/inference-services	main	+1 -1
haggingface_models: Update kserve to 0.15.2	Gkyziridis	machinelearning/liftwing/inference-services	main	+8 -7
ml-services: update revscoring to kserve 0.13.1	Ilias Sarantopoulos	operations/deployment-charts	master	+7 -7
revscoring: upgrade kserve to 0.13.1	Ilias Sarantopoulos	machinelearning/liftwing/inference-services	main	+12 -12
ml-services: update article-descriptions kserve to 0.13.1	Ilias Sarantopoulos	operations/deployment-charts	master	+1 -1
article-descriptions: update kserve to 0.13.1	Ilias Sarantopoulos	machinelearning/liftwing/inference-services	main	+1 -1
ml-services: bump kserve in langid to 0.13.1	Ilias Sarantopoulos	operations/deployment-charts	master	+2 -2
langid: bump kserve to 0.13.1 and fasttext to 0.9.3	Ilias Sarantopoulos	machinelearning/liftwing/inference-services	main	+2 -2
kserve: Bump version to 0.13	Klausman	operations/docker-images/production-images	master	+40 -11
huggingface: kserve 0.13.0	Ilias Sarantopoulos	machinelearning/liftwing/inference-services	main	+32 -28

Customize query in gerrit

Related Objects
Search...

Status	Assigned	Task
Declined	None	T367048 Update kserve to 0.15.2
		Restricted Task
Resolved	BWojtowicz-WMF	T383119 Update revertrisk to kserve 0.15.2

Event Timeline

There are a very large number of changes, so older changes are hidden. Show Older Changes

Restricted Application added a subscriber: Aklapper. · View Herald TranscriptJun 10 2024, 12:16 PM

Change #1041082 had a related patch set uploaded (by Ilias Sarantopoulos; author: Ilias Sarantopoulos):

[machinelearning/liftwing/inference-services@main] huggingface: kserve 0.13.0

https://gerrit.wikimedia.org/r/1041082

gerritbot added a project: Patch-For-Review.Jun 10 2024, 12:19 PM

• elukey updated the task description. (Show Details)Jun 10 2024, 12:28 PM

Please also review T367050 :)

Change #1041082 merged by jenkins-bot:

[machinelearning/liftwing/inference-services@main] huggingface: kserve 0.13.0

https://gerrit.wikimedia.org/r/1041082

isarantopoulos mentioned this in rMLIS0df3628a6a54: huggingface: kserve 0.13.0.Jun 11 2024, 9:48 AM

Maintenance_bot removed a project: Patch-For-Review.Jun 11 2024, 10:30 AM

isarantopoulos claimed this task.Jun 11 2024, 2:40 PM

isarantopoulos set the point value for this task to 3.

isarantopoulos moved this task from Unsorted to In Progress on the Machine-Learning-Team board.

I have manually tested the new kserve version with the huggingface image and the bert model that is deployed in the experimental namespace.

isarantopoulos triaged this task as Medium priority.Jul 16 2024, 4:19 PM

Change #1046617 had a related patch set uploaded (by Klausman; author: Klausman):

[operations/docker-images/production-images@master] kserve: Bump version to 0.13

https://gerrit.wikimedia.org/r/1046617

gerritbot added a project: Patch-For-Review.Aug 21 2024, 1:07 PM

Change #1046617 merged by Klausman:

[operations/docker-images/production-images@master] kserve: Bump version to 0.13

https://gerrit.wikimedia.org/r/1046617

isarantopoulos moved this task from In Progress to Ready To Go on the Machine-Learning-Team board.Sep 24 2024, 2:39 PM

Maintenance_bot removed a project: Patch-For-Review.Sep 24 2024, 3:32 PM

Change #1078605 had a related patch set uploaded (by Ilias Sarantopoulos; author: Ilias Sarantopoulos):

[machinelearning/liftwing/inference-services@main] langid: bump kserve to 0.13.1

https://gerrit.wikimedia.org/r/1078605

gerritbot added a project: Patch-For-Review.Oct 8 2024, 8:09 AM

Change #1078605 merged by jenkins-bot:

[machinelearning/liftwing/inference-services@main] langid: bump kserve to 0.13.1 and fasttext to 0.9.3

https://gerrit.wikimedia.org/r/1078605

isarantopoulos mentioned this in rMLISe5d7f49011ce: langid: bump kserve to 0.13.1 and fasttext to 0.9.3.Oct 14 2024, 2:27 PM

Maintenance_bot removed a project: Patch-For-Review.Oct 14 2024, 2:31 PM

Change #1080052 had a related patch set uploaded (by Ilias Sarantopoulos; author: Ilias Sarantopoulos):

[operations/deployment-charts@master] ml-services: bump kserve in langid to 0.13.1

https://gerrit.wikimedia.org/r/1080052

gerritbot added a project: Patch-For-Review.Oct 14 2024, 2:55 PM

Change #1080052 merged by jenkins-bot:

[operations/deployment-charts@master] ml-services: bump kserve in langid to 0.13.1

https://gerrit.wikimedia.org/r/1080052

Maintenance_bot removed a project: Patch-For-Review.Oct 14 2024, 3:32 PM

Change #1080247 had a related patch set uploaded (by Ilias Sarantopoulos; author: Ilias Sarantopoulos):

[machinelearning/liftwing/inference-services@main] article-descriptions: update kserve to 0.13.1

https://gerrit.wikimedia.org/r/1080247

gerritbot added a project: Patch-For-Review.Oct 15 2024, 10:02 AM

Change #1080247 merged by jenkins-bot:

[machinelearning/liftwing/inference-services@main] article-descriptions: update kserve to 0.13.1

https://gerrit.wikimedia.org/r/1080247

isarantopoulos mentioned this in rMLIS50f40e1e5a0a: article-descriptions: update kserve to 0.13.1.Oct 17 2024, 1:35 PM

Maintenance_bot removed a project: Patch-For-Review.Oct 17 2024, 2:30 PM

Change #1081392 had a related patch set uploaded (by Ilias Sarantopoulos; author: Ilias Sarantopoulos):

[operations/deployment-charts@master] ml-services: update article-descriptions kserve to 0.13.1

https://gerrit.wikimedia.org/r/1081392

gerritbot added a project: Patch-For-Review.Oct 18 2024, 11:47 AM

Change #1081392 merged by jenkins-bot:

[operations/deployment-charts@master] ml-services: update article-descriptions kserve to 0.13.1

https://gerrit.wikimedia.org/r/1081392

Maintenance_bot removed a project: Patch-For-Review.Oct 18 2024, 1:31 PM

isarantopoulos moved this task from Ready To Go to In Progress on the Machine-Learning-Team board.Oct 18 2024, 1:56 PM

I just upgraded artilce descriptions model.

This is the current state of the services

service	kserve version
reference quality	0.13.1
revscoring	0.11.2
article country	0.13.1
article descriptions	0.13.1
langid	0.13.1
logo detection	0.11.2
outlink topic model	0.11.2
readability	0.13.1
revertrisk	0.12.1

From the above the services that should be prioritized for an upgrade are:

revscoring
outlink
logo detection

Also 2 days ago kserve 0.14 was released https://github.com/kserve/kserve/releases/tag/v0.14.0

I tried upgrading kserve to 0.14.0 in revscoring and after dealing with some dependency conflicts I bumped into this

57.28 The conflict is caused by:
57.28     The user requested numpy<2.0 and >=1.18.5
57.28     gensim 4.3.3 depends on numpy<2.0 and >=1.18.5
57.28     konlpy 0.5.2 depends on numpy>=1.6
57.28     kserve 0.14.0 depends on numpy<2.0.0 and >=1.26.0
57.28     revscoring 2.11.13 depends on numpy<1.25.0 and >=1.21.5

In order to resolve this numpy dependency in revscoring should be bumped to allow >=1.6

To surpass the above issue we need to update the revscoring repository which is already archived.
In the meantime I am upgrading to 0.13.1 which seems feasible with some python dependency hell management. I managed to build the image and plan to deploy it to staging to make sure it works.

encountered the following issue when running revscoring:

Traceback (most recent call last):
  File "/srv/rev/revscoring_model/model.py", line 5, in <module>
    import kserve
  File "/opt/lib/python/site-packages/kserve/__init__.py", line 18, in <module>
    from .model_server import ModelServer
  File "/opt/lib/python/site-packages/kserve/model_server.py", line 33, in <module>
    from .protocol.dataplane import DataPlane
  File "/opt/lib/python/site-packages/kserve/protocol/dataplane.py", line 33, in <module>
    from .rest.openai import OpenAIModel
  File "/opt/lib/python/site-packages/kserve/protocol/rest/openai/__init__.py", line 15, in <module>
    from .openai_model import (
  File "/opt/lib/python/site-packages/kserve/protocol/rest/openai/openai_model.py", line 21, in <module>
    from kserve.protocol.rest.openai.types import (
  File "/opt/lib/python/site-packages/kserve/protocol/rest/openai/types/__init__.py", line 17, in <module>
    from kserve.protocol.rest.openai.types.openapi import (
  File "/opt/lib/python/site-packages/kserve/protocol/rest/openai/types/openapi.py", line 23, in <module>
    from pydantic import (
ImportError: cannot import name 'ConfigDict' from 'pydantic' (/opt/lib/python/site-packages/pydantic/__init__.cpython-39-x86_64-linux-gnu.so)

After upgrading pydantic from 1.8.2 to 2.9.2 the above issue seems to have been resolved.

Change #1085625 had a related patch set uploaded (by Ilias Sarantopoulos; author: Ilias Sarantopoulos):

[machinelearning/liftwing/inference-services@main] revscoring: upgrade kserve to 0.13.1

https://gerrit.wikimedia.org/r/1085625

gerritbot added a project: Patch-For-Review.Nov 4 2024, 8:35 AM

Change #1085625 merged by jenkins-bot:

[machinelearning/liftwing/inference-services@main] revscoring: upgrade kserve to 0.13.1

https://gerrit.wikimedia.org/r/1085625

isarantopoulos mentioned this in rMLISc146d4520ff3: revscoring: upgrade kserve to 0.13.1.Nov 4 2024, 9:14 AM

Change #1087130 had a related patch set uploaded (by Ilias Sarantopoulos; author: Ilias Sarantopoulos):

[operations/deployment-charts@master] ml-services: update revscoring to kserve 0.13.1

https://gerrit.wikimedia.org/r/1087130

Change #1087130 merged by jenkins-bot:

[operations/deployment-charts@master] ml-services: update revscoring to kserve 0.13.1

https://gerrit.wikimedia.org/r/1087130

Maintenance_bot removed a project: Patch-For-Review.Nov 4 2024, 10:31 AM

isarantopoulos changed the point value for this task from 3 to 8.Nov 7 2024, 11:03 AM

I ran some load tests on revscoring with wrk

before:

wrk -c 1 -t 1 --timeout 2s -s inference-damaging.lua https://inference.svc.codfw.wmnet:30443/v1/models/enwiki-damaging:predict --latency -d 60
Running 1m test @ https://inference.svc.codfw.wmnet:30443/v1/models/enwiki-damaging:predict
  1 threads and 1 connections
  Thread Stats   Avg      Stdev     Max   +/- Stdev
    Latency   212.26ms  162.90ms   1.63s    93.38%
    Req/Sec     6.28      2.67    10.00     59.53%
  Latency Distribution
     50%  167.36ms
     75%  183.48ms
     90%  290.37ms
     99%    1.03s
  299 requests in 1.00m, 103.37KB read
  Socket errors: connect 0, read 0, write 0, timeout 1

after upgrade

wrk -c 1 -t 1 --timeout 2s -s inference-damaging.lua https://inference-staging.svc.codfw.wmnet:30443/v1/models/enwiki-damaging:predict --latency -d 60
Running 1m test @ https://inference-staging.svc.codfw.wmnet:30443/v1/models/enwiki-damaging:predict
  1 threads and 1 connections
  Thread Stats   Avg      Stdev     Max   +/- Stdev
    Latency   213.90ms  149.36ms   1.58s    94.70%
    Req/Sec     5.94      2.64    10.00     58.13%
  Latency Distribution
     50%  168.12ms
     75%  204.16ms
     90%  312.74ms
     99%  960.32ms
  289 requests in 1.00m, 99.91KB read
  Socket errors: connect 0, read 0, write 0, timeout 1

I added a test for enwiki-damaging and the locust tests showed the following for a 2' duration which shows that we need to work a bit at some point of our load testing.
This doesn't have much to do with this task but I'm just documenting it for future reference

MODEL=revscoring my_locust_venv/bin/locust --headless --csv results/revscoring
[2024-11-07 10:57:26,457] stat1008/INFO/locust.main: Run time limit set to 120 seconds
[2024-11-07 10:57:26,457] stat1008/INFO/locust.main: Starting Locust 2.31.5
[2024-11-07 10:57:26,458] stat1008/INFO/locust.runners: Ramping to 2 users at a rate of 10.00 per second
[2024-11-07 10:57:26,458] stat1008/INFO/locust.runners: All users spawned: {"EnwikiDamaging": 1, "EnwikiGoodfaith": 1} (2 total users)
[2024-11-07 10:59:25,865] stat1008/INFO/locust.main: --run-time limit reached, shutting down
Load test results are within the threshold
[2024-11-07 10:59:25,953] stat1008/INFO/locust.main: Shutting down (exit code 1)
Type     Name                                 # reqs      # fails |    Avg     Min     Max    Med |   req/s  failures/s
--------|-----------------------------------|-------|-------------|-------|-------|-------|-------|--------|-----------
POST     /v1/models/enwiki-damaging:predict      231     0(0.00%) |    515     172    3484    270 |    1.93        0.00

--------|-----------------------------------|-------|-------------|-------|-------|-------|-------|--------|-----------
         Aggregated                              270   39(14.44%) |    465     167    3484    250 |    2.26        0.33

Response time percentiles (approximated)
Type     Name                                         50%    66%    75%    80%    90%    95%    98%    99%  99.9% 99.99%   100% # reqs
--------|---------------------------------------|--------|------|------|------|------|------|------|------|------|------|------|------
POST     /v1/models/enwiki-damaging:predict           270    350    420    450    910   2700   3300   3400   3500   3500   3500    231

--------|---------------------------------------|--------|------|------|------|------|------|------|------|------|------|------|------
         Aggregated                                   250    330    390    430    760   2500   3300   3400   3500   3500   3500    270

I'm proceeding with prod deployments for revscoring.

Revscoring has been upgraded to kserve 0.13.1
This is the updated state on kserve versions:

service	kserve version
reference quality	0.13.1
revscoring	0.13.1
article country	0.13.1
article descriptions	0.13.1
langid	0.13.1
logo detection	0.11.2
outlink topic model	0.11.2
readability	0.13.1
revertrisk	0.12.1

Next in line that should be upgraded is outlink_topic_model.

isarantopoulos renamed this task from Investigate kserve 0.13.0 upgrade to Update kserve to 0.13.1.Nov 19 2024, 3:48 PM

achou moved this task from In Progress to 2025-2026 Q1 Done on the Machine-Learning-Team board.Nov 19 2024, 3:52 PM

achou moved this task from 2025-2026 Q1 Done to In Progress on the Machine-Learning-Team board.

isarantopoulos lowered the priority of this task from Medium to Low.Nov 26 2024, 3:31 PM

isarantopoulos moved this task from In Progress to Ready To Go on the Machine-Learning-Team board.Dec 16 2024, 8:57 AM

Change #1113099 had a related patch set uploaded (by Ilias Sarantopoulos; author: Ilias Sarantopoulos):

[machinelearning/liftwing/inference-services@main] logo-detection: update kserve to 0.14.1

https://gerrit.wikimedia.org/r/1113099

gerritbot added a project: Patch-For-Review.Jan 21 2025, 10:51 AM

isarantopoulos removed isarantopoulos as the assignee of this task.Jan 28 2025, 2:40 PM

We should probably rename this task to upgrade to 0.15 https://github.com/kserve/kserve/releases/tag/v0.15.0
The only possible issue would be that we still have the charts in a much older version. I'm not sure if this would cause an issue or not. We could update a service and test.
@klausman any thoughts?

isarantopoulos added a project: Essential-Work.Jul 8 2025, 1:50 PM

isarantopoulos renamed this task from Update kserve to 0.13.1 to Update kserve to 0.15.2.Jul 15 2025, 2:44 PM

isarantopoulos renamed this task from Update kserve to 0.15.2 to Update kserve to 0.13.1.

isarantopoulos renamed this task from Update kserve to 0.13.1 to Update kserve to 0.15.2.Jul 16 2025, 12:26 PM

isarantopoulos updated the task description. (Show Details)

isarantopoulos added a subtask: T383119: Update revertrisk to kserve 0.15.2.

BWojtowicz-WMF closed subtask T383119: Update revertrisk to kserve 0.15.2 as Resolved.Sep 10 2025, 7:17 AM

Change #1195632 had a related patch set uploaded (by Gkyziridis; author: Gkyziridis):

[machinelearning/liftwing/inference-services@main] haggingface_models: Update kserve to 0.15.2

https://gerrit.wikimedia.org/r/1195632

Update

Testing Locally

Trying to build the Apple Silicon version (reproducing the current one in inference-services) following the Readme.md getting the following error:

$ docker build --target production -f .pipeline/huggingface/blubber_m1.yaml -t hf:kserve-m1 .
[+] Building 2.8s (18/24)                                                                                                                                                                                                                                                                                                                                                                                        docker:desktop-linux
 => [internal] load build definition from blubber_m1.yaml                                                                                                                                                                                                                                                                                                                                                                        
 => => transferring dockerfile: 1.74kB                                                                                                                                                                                                                                                                                                                                                                                           
 => resolve image config for docker-image://docker-registry.wikimedia.org/repos/releng/blubber/buildkit:v0.23.0                                                                                                                                                                                                                                                                                                                  
 => CACHED docker-image://docker-registry.wikimedia.org/repos/releng/blubber/buildkit:v0.23.0@sha256:6b1535a39497bb6c5e0a733595721a91cee33dba99ab59d8323d077665073a53                                                                                                                                                                                                                                                            
 => => resolve docker-registry.wikimedia.org/repos/releng/blubber/buildkit:v0.23.0@sha256:6b1535a39497bb6c5e0a733595721a91cee33dba99ab59d8323d077665073a53                                                                                                                                                                                                                                                                       
 => local://dockerfile                                                                                                                                                                                                                                                                                                                                                                                                           
 => => transferring dockerfile: 1.74kB                                                                                                                                                                                                                                                                                                                                                                                           
 => local://context                                                                                                                                                                                                                                                                                                                                                                                                              
 => => transferring context: 34B                                                                                                                                                                                                                                                                                                                                                                                                 
 => [internal] load metadata for docker.io/arm64v8/debian:stable                                                                                                                                                                                                                                                                                                                                                                 
 => [internal] load build context                                                                                                                                                                                                                                                                                                                                                                                                
 => => transferring context: 261B                                                                                                                                                                                                                                                                                                                                                                                                
 => CACHED [build  1/10] FROM docker.io/arm64v8/debian:stable@sha256:5168358b65bf8037b57bb3f7bdf8abf59c3853d7a0c1b846f65ad764b57aa356                                                                                                                                                                                                                                                                                            
 => => resolve docker.io/arm64v8/debian:stable@sha256:5168358b65bf8037b57bb3f7bdf8abf59c3853d7a0c1b846f65ad764b57aa356                                                                                                                                                                                                                                                                                                           
 => CACHED [build  2/10] RUN apt-get update && apt-get install -y "build-essential" "git" "python3-pip" "python3-dev" "python3-setuptools" "python3-venv" && rm -rf /var/lib/apt/lists/*                                                                                                                                                                                                                                         
 => CACHED [build  3/10] RUN (getent group "65533" || groupadd -o -g "65533" -r "somebody") && (getent passwd "65533" || useradd -l -o -m -d "/home/somebody" -r -g "65533" -u "65533" "somebody") && mkdir -p "/srv/app" && chown "65533":"65533" "/srv/app" && mkdir -p "/opt/lib" && chown "65533":"65533" "/opt/lib"                                                                                                       
 => CACHED [build  4/10] RUN (getent group "900" || groupadd -o -g "900" -r "runuser") && (getent passwd "900" || useradd -l -o -m -d "/home/runuser" -r -g "900" -u "900" "runuser")                                                                                                                                                                                                                                            
 => CACHED [build  5/10] WORKDIR /srv/app                                                                                                                                                                                                                                                                                                                                                                                        
 => CACHED [build  6/10] RUN git "clone" "--branch" "apple-silicon" "https://github.com/wikimedia/kserve.git" "kserve_repo"                                                                                                                                                                                                                                                                                                      
 => CACHED [build  7/10] COPY --chown=65533:65533 [src/models/huggingface_modelserver/requirements_apple_silicon.txt, src/models/huggingface_modelserver/]                                                                                                                                                                                                                                                                       
 => CACHED [build  8/10] RUN python3 "-m" "venv" "/opt/lib/venv" "--system-site-packages"                                                                                                                                                                                                                                                                                                                                        
 => CACHED [build  9/10] RUN python3 "-m" "pip" "install" "-U" "setuptools!=60.9.0" && python3 "-m" "pip" "install" "-U" "wheel" "tox" "pip"                                                                                                                                                                                                                                                                                     
 => ERROR [build 10/10] RUN python3 "-m" "pip" "install" "-r" "src/models/huggingface_modelserver/requirements_apple_silicon.txt"                                                                                                                                                                                                                                                                                                
 => CANCELED [production 2/8] RUN apt-get update && apt-get install -y "python3" "python3-distutils" "python3-pip" "python3-setuptools" && rm -rf /var/lib/apt/lists/*                                                                                                                                                                                                                                                           
------                                                                                                                                                                                                                                                                                                                                                                                                                                
 > [build 10/10] RUN python3 "-m" "pip" "install" "-r" "src/models/huggingface_modelserver/requirements_apple_silicon.txt":                                                                                                                                                                                                                                                                                                           
0.308 Processing ./kserve_repo/python/kserve (from -r src/models/huggingface_modelserver/requirements_apple_silicon.txt (line 1))                                                                                                                                                                                                                                                                                                     
0.309   Installing build dependencies: started                                                                                                                                                                                                                                                                                                                                                                                        
0.752   Installing build dependencies: finished with status 'done'                                                                                                                                                                                                                                                                                                                                                                    
0.752   Getting requirements to build wheel: started                                                                                                                                                                                                                                                                                                                                                                                  
0.862   Getting requirements to build wheel: finished with status 'done'                                                                                                                                                                                                                                                                                                                                                              
0.863   Preparing metadata (pyproject.toml): started                                                                                                                                                                                                                                                                                                                                                                                  
1.030   Preparing metadata (pyproject.toml): finished with status 'done'                                                                                                                                                                                                                                                                                                                                                              
1.033 Obtaining file:///srv/app/kserve_repo/python/huggingfaceserver (from -r src/models/huggingface_modelserver/requirements_apple_silicon.txt (line 2))                                                                                                                                                                                                                                                                             
1.034   Installing build dependencies: started                                                                                                                                                                                                                                                                                                                                                                                        
1.288   Installing build dependencies: finished with status 'done'                                                                                                                                                                                                                                                                                                                                                                    
1.288   Checking if build backend supports build_editable: started
1.398   Checking if build backend supports build_editable: finished with status 'done'
1.399   Getting requirements to build editable: started
1.462   Getting requirements to build editable: finished with status 'done'
1.463   Preparing editable metadata (pyproject.toml): started
1.574   Preparing editable metadata (pyproject.toml): finished with status 'done'
1.579 INFO: pip is looking at multiple versions of kserve to determine which version is compatible with other requirements. This could take a while.
1.651 ERROR: Package 'kserve' requires a different Python: 3.13.5 not in '<3.13,>=3.9'
------
ERROR: failed to solve: process "/bin/sh -c python3 \"-m\" \"pip\" \"install\" \"-r\" \"src/models/huggingface_modelserver/requirements_apple_silicon.txt\"" did not complete successfully: exit code: 1

Trying to update kserve==0.15.2 for haggingface adding it in the requirements.txt.
It seems that I can build the docker image locally:

$ docker build --target production -f .pipeline/huggingface/blubber.yaml --platform=linux/amd64 -t hf:kserve .
[+] Building 238.5s (19/19) FINISHED                                                                                                                                                                                                                                                                                                                                                                             docker:desktop-linux
 => [internal] load build definition from blubber.yaml                                                                                                                                                                                                                                                                                                                                                                           
 => => transferring dockerfile: 1.69kB                                                                                                                                                                                                                                                                                                                                                                                           
 => resolve image config for docker-image://docker-registry.wikimedia.org/repos/releng/blubber/buildkit:v1.1.0                                                                                                                                                                                                                                                                                                                   
 => CACHED docker-image://docker-registry.wikimedia.org/repos/releng/blubber/buildkit:v1.1.0@sha256:ed64673e7c362f5f264f92ea36bf379f47c90699ee36d32ed6a96aba7de858f2                                                                                                                                                                                                                                                             
 => => resolve docker-registry.wikimedia.org/repos/releng/blubber/buildkit:v1.1.0@sha256:ed64673e7c362f5f264f92ea36bf379f47c90699ee36d32ed6a96aba7de858f2                                                                                                                                                                                                                                                                        
 => [internal] load .dockerignore                                                                                                                                                                                                                                                                                                                                                                                                
 => => transferring context: 88B                                                                                                                                                                                                                                                                                                                                                                                                 
 => [internal] load build context                                                                                                                                                                                                                                                                                                                                                                                                
 => => transferring context: 1.71MB                                                                                                                                                                                                                                                                                                                                                                                              
 => [build]      🌐 docker-registry.wikimedia.org/amd-pytorch23:2.3.0rocm6.0-2@sha256:61bca3a10ca76c96aa663f53b0adc9598b9f07b735846059e441f24f5ca77084                                                                                                                                                                                                                                                                          
 => => resolve docker-registry.wikimedia.org/amd-pytorch23:2.3.0rocm6.0-2@sha256:61bca3a10ca76c96aa663f53b0adc9598b9f07b735846059e441f24f5ca77084                                                                                                                                                                                                                                                                                
 => => sha256:2418fb0183302501eff228114f00be99215561edf1137cd3e1c122cc463df2e7 2.50GB / 2.50GB                                                                                                                                                                                                                                                                                                                                 
 => => extracting sha256:2418fb0183302501eff228114f00be99215561edf1137cd3e1c122cc463df2e7                                                                                                                                                                                                                                                                                                                                       
 => [production] 🖥️ # (getent group "65533" || groupadd -o -g "65533" -r "somebody") && (getent passwd "65533" || useradd -l -o -m -d "/home/somebody" -r -g "65533" -u "65533" "somebody") && mkdir -p "/srv/app" && chown "65533":"65533" "/srv/app" && mkdir -p "/opt/lib" && chown "65533":"65533" "/opt/lib"                                                                                                                
 => [build]      🖥️ # apt-get update && apt-get install -y "build-essential" "git" "python3-venv" && rm -rf /var/lib/apt/lists/*                                                                                                                                                                                                                                                                                                
 => [production] 🖥️ # (getent group "900" || groupadd -o -g "900" -r "runuser") && (getent passwd "900" || useradd -l -o -m -d "/home/runuser" -r -g "900" -u "900" "runuser")                                                                                                                                                                                                                                                   
 => [build]      🖥️ # (getent group "65533" || groupadd -o -g "65533" -r "somebody") && (getent passwd "65533" || useradd -l -o -m -d "/home/somebody" -r -g "65533" -u "65533" "somebody") && mkdir -p "/srv/app" && chown "65533":"65533" "/srv/app" && mkdir -p "/opt/lib" && chown "65533":"65533" "/opt/lib"
 => [build]      🖥️ # (getent group "900" || groupadd -o -g "900" -r "runuser") && (getent passwd "900" || useradd -l -o -m -d "/home/runuser" -r -g "900" -u "900" "runuser")                                                                                                                                                                                                                                                   
 => [build]      📂 [src/models/huggingface_modelserver/requirements.txt] -> src/models/huggingface_modelserver/                                                                                                                                                                                                                                                                                                                  
 => [build]      🖥️ @65533 $ python3 "-m" "venv" "/opt/lib/venv"                                                                                                                                                                                                                                                                                                                                                                 
 => [build]      🖥️ @65533 $ python3 "-m" "pip" "install" "-U" "setuptools!=60.9.0" && python3 "-m" "pip" "install" "-U" "wheel" "tox" "pip"                                                                                                                                                                                                                                                                                     
 => [build]      🖥️ @65533 $ python3 "-m" "pip" "install" "-r" "src/models/huggingface_modelserver/requirements.txt"                                                                                                                                                                                                                                                                                                            
 => [production] 📦 {build}[/opt/lib/python/site-packages] -> /opt/lib/python/site-packages                                                                                                                                                                                                                                                                                                                                       
 => [production] 📦 {build}[/opt/lib/venv/lib/python3.11/site-packages/] -> /opt/lib/venv/lib/python3.11/site-packages/                                                                                                                                                                                                                                                                                                           
 => [production] 📂 [src/models/huggingface_modelserver/entrypoint.sh] -> .                                                                                                                                                                                                                                                                                                                                                       
 => exporting to image                                                                                                                                                                                                                                                                                                                                                                                                           
 => => exporting layers                                                                                                                                                                                                                                                                                                                                                                                                          
 => => exporting manifest sha256:abf4a516922b56ace4d0ee806c59913e9632463a81e411abb79682f37cd79cfd                                                                                                                                                                                                                                                                                                                                
 => => exporting config sha256:3dd855bb5159730dc5a12da47537ebb2512bc538758dd6c3f189933dbef9ee91                                                                                                                                                                                                                                                                                                                                  
 => => exporting attestation manifest sha256:be3ea92b76c3e132319b83a34b5544607895f5cf1b10f1004a856c4d81fbff9e                                                                                                                                                                                                                                                                                                                   
 => => exporting manifest list sha256:705dd69b803362093c919214a0f748222e50dc01c5960059042c5f93e4dace66                                                                                                                                                                                                                                                                                                                          
 => => naming to docker.io/library/hf:kserve                                                                                                                                                                                                                                                                                                                                                                                    

View build details: docker-desktop://dashboard/build/desktop-linux/desktop-linux/8cci7g2bcfi0g84bfdxur35gs

But I cannot run it (probably due to Mac) I am getting this error:

$ docker run -p 8080:8080 -e MODEL_ID=bert-base-uncased -e MODEL_NAME=bert hf:kserve_arm
WARNING: The requested image's platform (linux/amd64) does not match the detected host platform (linux/arm64/v8) and no specific platform was requested
./entrypoint.sh: line 9:     7 Illegal instruction     python3 -m huggingfaceserver --model_id $MODEL_ID --model_name $MODEL_NAME "${@}"

Even when running an interactive session:

$ docker run -it --entrypoint=/bin/bash hf:kserve                                       
WARNING: The requested image's platform (linux/amd64) does not match the detected host platform (linux/arm64/v8) and no specific platform was requested
somebody@648b34389843:/srv/app$ ls -l
total 444
-rw-r--r-- 1 somebody somebody    219 Oct 13 11:31 Makefile
-rw-r--r-- 1 somebody somebody   7282 Oct 13 11:31 README.md
-rwxr-xr-x 1 somebody somebody    342 Apr 24 08:02 entrypoint.sh
drwxr-xr-x 3 somebody somebody   4096 Oct 13 11:31 huggingfaceserver
-rw-r--r-- 1 somebody somebody 427653 Oct 13 11:31 poetry.lock
-rw-r--r-- 1 somebody somebody   1135 Oct 13 11:31 pyproject.toml
somebody@648b34389843:/srv/app$ pip list | grep kserve 
kserve              0.15.2
somebody@648b34389843:/srv/app$ python3 -m huggingfaceserver --model-id bert-base-uncased --model-name "bert" 
Illegal instruction

Testing on remote machines

Machines ml-lab1 and ml-lab2 do not support blubber because they do not run the buildkit daemon.
So, I needed to translate the blubber.yaml to Dockerfile in order to test it on those machines.
I used this simple Dockerfile:

# Use the base image
FROM docker-registry.wikimedia.org/amd-pytorch23:2.3.0rocm6.0-2

# # Switch to root user to gain necessary permissions
USER root

ENV http_proxy=http://webproxy:8080
ENV https_proxy=http://webproxy:8080
ENV no_proxy=127.0.0.1,::1,localhost,.wmnet,.wikimedia.org,.wikipedia.org,.wikibooks.org,.wikiquote.org,.wiktionary.org,.wikisource.org,.wikispecies.org,.wikiversity.org,.wikidata.org,.mediawiki.org,.wikinews.org,.wikivoyage.org

# Update package lists and install required packages
RUN apt-get update && \
    apt-get install -y build-essential python3-pip python3-dev python3-setuptools python3-venv git g++ wget vim curl

Building it on ml-lab1002 I received the error:

ml-lab1002:~$ docker build -t hf:update_kserve .
Sending build context to Docker daemon  2.753GB
Step 1/6 : FROM docker-registry.wikimedia.org/amd-pytorch23:2.3.0rocm6.0-2
 ---> 312bc9118c1c
Step 2/6 : USER root
 ---> Using cache
 ---> cc1e9478f398
Step 3/6 : ENV http_proxy=http://webproxy:8080
 ---> Using cache
 ---> 3a694ddf6968
Step 4/6 : ENV https_proxy=http://webproxy:8080
 ---> Using cache
 ---> 3598bf14ee4e
Step 5/6 : ENV no_proxy=127.0.0.1,::1,localhost,.wmnet,.wikimedia.org,.wikipedia.org,.wikibooks.org,.wikiquote.org,.wiktionary.org,.wikisource.org,.wikispecies.org,.wikiversity.org,.wikidata.org,.mediawiki.org,.wikinews.org,.wikivoyage.org
 ---> Using cache
 ---> 54a86e473460
Step 6/6 : RUN apt-get update &&     apt-get install -y build-essential python3-pip python3-dev python3-setuptools python3-venv git g++ wget vim curl
 ---> Running in 74e915980371
Ign:1 http://security.debian.org/debian-security bookworm-security InRelease
Ign:2 http://apt.wikimedia.org/wikimedia bookworm-wikimedia InRelease
Ign:3 http://mirrors.wikimedia.org/debian bookworm InRelease
Ign:4 http://mirrors.wikimedia.org/debian bookworm-updates InRelease
Ign:1 http://security.debian.org/debian-security bookworm-security InRelease
Ign:2 http://apt.wikimedia.org/wikimedia bookworm-wikimedia InRelease
Ign:5 http://mirrors.wikimedia.org/debian bookworm-backports InRelease
Ign:2 http://apt.wikimedia.org/wikimedia bookworm-wikimedia InRelease
Ign:1 http://security.debian.org/debian-security bookworm-security InRelease
Ign:3 http://mirrors.wikimedia.org/debian bookworm InRelease
Ign:4 http://mirrors.wikimedia.org/debian bookworm-updates InRelease
Err:2 http://apt.wikimedia.org/wikimedia bookworm-wikimedia InRelease
  Temporary failure resolving 'apt.wikimedia.org'
Err:1 http://security.debian.org/debian-security bookworm-security InRelease
  Temporary failure resolving 'webproxy'
Ign:5 http://mirrors.wikimedia.org/debian bookworm-backports InRelease
Ign:3 http://mirrors.wikimedia.org/debian bookworm InRelease
Ign:4 http://mirrors.wikimedia.org/debian bookworm-updates InRelease
Ign:5 http://mirrors.wikimedia.org/debian bookworm-backports InRelease
Err:3 http://mirrors.wikimedia.org/debian bookworm InRelease
  Temporary failure resolving 'mirrors.wikimedia.org'
Err:4 http://mirrors.wikimedia.org/debian bookworm-updates InRelease
  Temporary failure resolving 'mirrors.wikimedia.org'
Err:5 http://mirrors.wikimedia.org/debian bookworm-backports InRelease
  Temporary failure resolving 'mirrors.wikimedia.org'
Reading package lists...
W: Failed to fetch http://mirrors.wikimedia.org/debian/dists/bookworm/InRelease  Temporary failure resolving 'mirrors.wikimedia.org'
W: Failed to fetch http://security.debian.org/debian-security/dists/bookworm-security/InRelease  Temporary failure resolving 'webproxy'
W: Failed to fetch http://mirrors.wikimedia.org/debian/dists/bookworm-updates/InRelease  Temporary failure resolving 'mirrors.wikimedia.org'
W: Failed to fetch http://mirrors.wikimedia.org/debian/dists/bookworm-backports/InRelease  Temporary failure resolving 'mirrors.wikimedia.org'
W: Failed to fetch http://apt.wikimedia.org/wikimedia/dists/bookworm-wikimedia/InRelease  Temporary failure resolving 'apt.wikimedia.org'
W: Some index files failed to download. They have been ignored, or old ones used instead.
Reading package lists...
Building dependency tree...
Reading state information...
Package python3-venv is not available, but is referred to by another package.
This may mean that the package is missing, has been obsoleted, or
is only available from another source

Package build-essential is not available, but is referred to by another package.
This may mean that the package is missing, has been obsoleted, or
is only available from another source

Package python3-dev is not available, but is referred to by another package.
This may mean that the package is missing, has been obsoleted, or
is only available from another source

E: Package 'build-essential' has no installation candidate
E: Package 'python3-dev' has no installation candidate
E: Package 'python3-venv' has no installation candidate
E: Unable to locate package git
E: Unable to locate package wget
E: Unable to locate package vim
E: Unable to locate package curl
The command '/bin/sh -c apt-get update &&     apt-get install -y build-essential python3-pip python3-dev python3-setuptools python3-venv git g++ wget vim curl' returned a non-zero code: 100

On ml-lab1001:

ml-lab1001:~$ docker build -t hf:update_kserve .
Sending build context to Docker daemon  49.26GB
Error response from daemon: Error processing tar file(exit status 1): write /.edit_pyt25/lib/python3.11/site-packages/torch/lib/rocblas/library/TensileLibrary_Type_HH_HPA_ExperimentalGrid_Contraction_l_Ailk_Bjlk_Cijk_Dijk_gfx90a.co: no space left on device

Machine ml-testing supports blubber so I could test it directly, however it seems that there is not space left:

ml-testing:~/inference-services$ docker build --target production -f .pipeline/huggingface/blubber.yaml --platform=linux/amd64 -t hf:kserve .
[+] Building 257.5s (6/19)                                                                                                                                                                                                                                                                                                                                                                                               docker:default
 => [internal] load build definition from blubber.yaml                                                                                                                                                                                                                                                                                                                                                                             
 => => transferring dockerfile: 1.58kB                                                                                                                                                                                                                                                                                                                                                                                             
 => resolve image config for docker-image://docker-registry.wikimedia.org/repos/releng/blubber/buildkit:v1.1.0                                                                                                                                                                                                                                                                                                                     
 => CACHED docker-image://docker-registry.wikimedia.org/repos/releng/blubber/buildkit:v1.1.0@sha256:ed64673e7c362f5f264f92ea36bf379f47c90699ee36d32ed6a96aba7de858f2                                                                                                                                                                                                                                                               
 => [internal] load build definition from blubber.yaml                                                                                                                                                                                                                                                                                                                                                                             
 => [internal] load .dockerignore                                                                                                                                                                                                                                                                                                                                                                                                  
 => => transferring context: 88B                                                                                                                                                                                                                                                                                                                                                                                                   
 => [internal] load build context                                                                                                                                                                                                                                                                                                                                                                                                  
 => => transferring context: 22.94kB                                                                                                                                                                                                                                                                                                                                                                                               
 => [build]      🌐 docker-registry.wikimedia.org/amd-pytorch23:2.3.0rocm6.0-2@sha256:61bca3a10ca76c96aa663f53b0adc9598b9f07b735846059e441f24f5ca77084                                                                                                                                                                                                                                                                            
 => => resolve docker-registry.wikimedia.org/amd-pytorch23:2.3.0rocm6.0-2@sha256:61bca3a10ca76c96aa663f53b0adc9598b9f07b735846059e441f24f5ca77084                                                                                                                                                                                                                                                                                  
 => => sha256:9e94c62ce5a28407b472ba3bb2abf6eecf59be4fa411fdd96397eadb5136a441 29.16MB / 29.16MB                                                                                                                                                                                                                                                                                                                                   
 => => sha256:7bd1fb5b49550afebb19d2e1bd8a7c06e3b4f76de454ce7167c4a3770bfaad4c 23.89MB / 23.89MB                                                                                                                                                                                                                                                                                                                                   
 => => sha256:f3a13a9bc46298e62919538bf88ac8d31175f909d30c6567bad79910eb2a2948 476.06kB / 476.06kB                                                                                                                                                                                                                                                                                                                                 
 => => sha256:61bca3a10ca76c96aa663f53b0adc9598b9f07b735846059e441f24f5ca77084 1.17kB / 1.17kB                                                                                                                                                                                                                                                                                                                                     
 => => sha256:312bc9118c1cc1cee0fabc3506593e953a25da88165d96cfa5ef9784170bd73f 4.01kB / 4.01kB                                                                                                                                                                                                                                                                                                                                     
 => => sha256:2418fb0183302501eff228114f00be99215561edf1137cd3e1c122cc463df2e7 2.50GB / 2.50GB                                                                                                                                                                                                                                                                                                                                   
 => => extracting sha256:9e94c62ce5a28407b472ba3bb2abf6eecf59be4fa411fdd96397eadb5136a441                                                                                                                                                                                                                                                                                                                                          
 => => extracting sha256:7bd1fb5b49550afebb19d2e1bd8a7c06e3b4f76de454ce7167c4a3770bfaad4c                                                                                                                                                                                                                                                                                                                                          
 => => extracting sha256:f3a13a9bc46298e62919538bf88ac8d31175f909d30c6567bad79910eb2a2948                                                                                                                                                                                                                                                                                                                                          
 => => extracting sha256:2418fb0183302501eff228114f00be99215561edf1137cd3e1c122cc463df2e7                                                                                                                                                                                                                                                                                                                                               
------
 > [build]      🌐 docker-registry.wikimedia.org/amd-pytorch23:2.3.0rocm6.0-2@sha256:61bca3a10ca76c96aa663f53b0adc9598b9f07b735846059e441f24f5ca77084:
------
ERROR: failed to solve: failed to register layer: write /opt/lib/python/base-packages/triton/_C/libtriton.so: no space left on device

I am not sure where and how to test this.
I think that we general need to create a space where we can easily test the actual images that we are using on production.
This space needs to support:

Blubber (avoid translating blubber into Dockerfile in order to test)
Enough space for testing images (e.g. rocm images)

The issue on ml-lab1001 is (was, now) that docker did not use the big multi-TB filesystem as storage for images, but it does now. I copied the Dockerfile from Georgios' homedir to a subdir of mine and ran docker build. It seems to have worked fine:

 ---> 82a4c1b28e66
Successfully built 82a4c1b28e66
Successfully tagged hf:update_kserve
ml-lab1001 hf $ docker image ls
REPOSITORY                                                                                       TAG              IMAGE ID       CREATED          SIZE
hf                                                                                               update_kserve    82a4c1b28e66   15 seconds ago   14GB

The build not working on 1002 is due to its special network setup. It could probably be fixed, though I doubt it's worth it.

build.log46 KBDownload

And here is the complete build log of the run mentioned above.

Thank you so much for working on that one @klausman!

Since the ml-lab1001 now uses the big TB filesystem, would it be possible to enable the buildkit daemon in order to use blubber, so ml-lab could be the dedicated machine for testing our production images and blubbers without translating the blubber files into Dockerfiles?

In T367048#11335401, @gkyziridis wrote:

Thank you so much for working on that one @klausman!

Since the ml-lab1001 now uses the big TB filesystem, would it be possible to enable the buildkit daemon in order to use blubber, so ml-lab could be the dedicated machine for testing our production images and blubbers without translating the blubber files into Dockerfiles?

I vaguely remember some dependencies are missing for buildkit/blubber on the host's disto (Bookworm), but that was months ago, I will re-investigate.

Edite to add: buildkitd is only available with Docker version >=24, whereas Bookworm is at v20. if-when we switch to Trixie, v26 would be available. I'll see if backports are available.

Hey folks, to re-iterate another time - please stop making manual configurations to these hosts and puppetize what is needed before proceeding further. The puppetization is also meant to raise awareness for the SRE team, that we are using ml-lab nodes for this purpose. Security-wise running docker is not trivial and it needs to be done the the proper way :)

klausman mentioned this in T327241: Move the kserve custom helm chart to the upstream one.Nov 26 2025, 1:02 PM

TBurmeister subscribed.Mar 5 2026, 4:53 PM

TBurmeister mentioned this in T419455: Generate OpenAPI descriptions for Lift Wing APIs.Mar 9 2026, 5:53 PM

kserve was updated to 0.17 via another task, closing

DPogorzelski-WMF moved this task from Ready To Go to 2025-2026 Q2 Done on the Machine-Learning-Team board.Apr 8 2026, 8:32 AM

This task refers to the python package and not the chart. I've opened a separate task to track this update T422591: Update kserve Python package to 0.17 across all inference services which we should tackle after rolling the chart to prod.

isarantopoulos changed the task status from Declined to Resolved.Apr 20 2026, 5:32 AM

isarantopoulos changed the task status from Resolved to Declined.Apr 20 2026, 5:34 AM

Change #1113099 abandoned by Ilias Sarantopoulos:

[machinelearning/liftwing/inference-services@main] logo-detection: update kserve to 0.14.1

Reason:

this is an outdated patch