Page MenuHomePhabricator

logo-detection image build fails: kserve 0.11.2 conflicts with pydantic v2 in shared requirements
Closed, ResolvedPublic

Description

Problem

The inference-services-pipeline-logo-detection-publish CI job fails at the pip install step:

  • kserve==0.11.2 (model server requirements) pins fastapi<0.96, which requires pydantic<2
  • python/requirements.txt (shared) has pydantic-settings==2.8.1, which requires pydantic>=2.7

This makes resolution impossible:

ERROR: ResolutionImpossible
pydantic-settings 2.8.1 depends on pydantic>=2.7.0
fastapi 0.95.2 depends on pydantic<2.0.0

Failed build: https://integration.wikimedia.org/ci/job/inference-services-pipeline-logo-detection-publish/40/console

Solution

Bump kserve to 0.15.2 in src/models/logo_detection/model_server/requirements.txt.
Verified locally that resolution succeeds (pydantic 2.13.4, fastapi 0.141.1, no ray dependency → smaller image), compatible with tensorflow-cpu 2.16.1 / keras 3.2.1.

TODO

  • Bump kserve to 0.15.2

Event Timeline

Change #1329278 had a related patch set uploaded (by Gkyziridis; author: Gkyziridis):

[machinelearning/liftwing/inference-services@main] logo_detection: bump kserve to 0.15.2 for pydantic v2 compatibility.

https://gerrit.wikimedia.org/r/1329278

Change #1329278 merged by jenkins-bot:

[machinelearning/liftwing/inference-services@main] logo_detection: bump kserve to 0.15.2 for pydantic v2 compatibility.

https://gerrit.wikimedia.org/r/1329278

Change #1329292 had a related patch set uploaded (by Gkyziridis; author: Gkyziridis):

[operations/deployment-charts@master] ml-services: Deploy latest logo-detection model version on staging.

https://gerrit.wikimedia.org/r/1329292

Change #1329292 merged by jenkins-bot:

[operations/deployment-charts@master] ml-services: Deploy latest logo-detection model version on staging.

https://gerrit.wikimedia.org/r/1329292

Change #1329515 had a related patch set uploaded (by Gkyziridis; author: Gkyziridis):

[operations/deployment-charts@master] ml-services: Deploy latest logo-detection model version on prod.

https://gerrit.wikimedia.org/r/1329515

Change #1329515 merged by jenkins-bot:

[operations/deployment-charts@master] ml-services: Deploy latest logo-detection model version on prod.

https://gerrit.wikimedia.org/r/1329515

Update

Logo-detection model CI/CD failure is fixed.
Merged and deployed on staging + prod.

Pods:

$ kube_env logo-detection ml-staging-codfw
$ kubectl get pods
NAME                                                         READY   STATUS    RESTARTS   AGE
logo-detection-predictor-00002-deployment-7647fcd9c5-d8xkk   3/3     Running   0          78m

$ kube_env logo-detection ml-serve-eqiad
$ kubectl get pods
NAME                                                         READY   STATUS    RESTARTS   AGE
logo-detection-predictor-00003-deployment-766d67fd76-8dqbb   3/3     Running   0          4m31s

$ kube_env logo-detection ml-serve-codfw
$ kubectl get pods
NAME                                                         READY   STATUS    RESTARTS   AGE
logo-detection-predictor-00003-deployment-5ffbb4554b-qsb6f   3/3     Running   0          4m36s