Revertrisk multilingual fails locally when ran with docker compose
Closed, ResolvedPublicBUG REPORT
Actions

Assigned To

Authored By

	isarantopoulos
	Oct 23 2025, 8:17 AM

Description

We get the following ImportError when running revertrisk-multilingual locally

docker compose up revertrisk-multilingual
Attaching to revertrisk-multilingual-1
revertrisk-multilingual-1  | + source common_settings.sh
revertrisk-multilingual-1  | +++ /usr/bin/python3 -c 'from python.resource_utils import get_cpu_count; print(get_cpu_count())'
revertrisk-multilingual-1  | ++ CPU_COUNT=20
revertrisk-multilingual-1  | ++ echo 'CPU count detected from get_cpu_count: 20'
revertrisk-multilingual-1  | ++ export OMP_NUM_THREADS=20
revertrisk-multilingual-1  | ++ OMP_NUM_THREADS=20
revertrisk-multilingual-1  | ++ echo 'OMP_NUM_THREADS set to: 20'
revertrisk-multilingual-1  | + MODEL_SERVER_PATH=model_server/model.py
revertrisk-multilingual-1  | + exec /usr/bin/python3 model_server/model.py
revertrisk-multilingual-1  | CPU count detected from get_cpu_count: 20
revertrisk-multilingual-1  | OMP_NUM_THREADS set to: 20
revertrisk-multilingual-1  | Traceback (most recent call last):
revertrisk-multilingual-1  |   File "/srv/revert_risk_model/model_server/model.py", line 5, in <module>
revertrisk-multilingual-1  |     import kserve
revertrisk-multilingual-1  |   File "/opt/lib/python/site-packages/kserve/__init__.py", line 17, in <module>
revertrisk-multilingual-1  |     from .model import Model
revertrisk-multilingual-1  |   File "/opt/lib/python/site-packages/kserve/model.py", line 27, in <module>
revertrisk-multilingual-1  |     from .errors import InvalidInput
revertrisk-multilingual-1  |   File "/opt/lib/python/site-packages/kserve/errors.py", line 17, in <module>
revertrisk-multilingual-1  |     from fastapi.responses import JSONResponse
revertrisk-multilingual-1  |   File "/opt/lib/python/site-packages/fastapi/__init__.py", line 7, in <module>
revertrisk-multilingual-1  |     from .applications import FastAPI as FastAPI
revertrisk-multilingual-1  |   File "/opt/lib/python/site-packages/fastapi/applications.py", line 16, in <module>
revertrisk-multilingual-1  |     from fastapi import routing
revertrisk-multilingual-1  |   File "/opt/lib/python/site-packages/fastapi/routing.py", line 27, in <module>
revertrisk-multilingual-1  |     from fastapi import params, temp_pydantic_v1_params
revertrisk-multilingual-1  |   File "/opt/lib/python/site-packages/fastapi/params.py", line 5, in <module>
revertrisk-multilingual-1  |     from fastapi.openapi.models import Example
revertrisk-multilingual-1  |   File "/opt/lib/python/site-packages/fastapi/openapi/models.py", line 4, in <module>
revertrisk-multilingual-1  |     from fastapi._compat import (
revertrisk-multilingual-1  |   File "/opt/lib/python/site-packages/fastapi/_compat/__init__.py", line 1, in <module>
revertrisk-multilingual-1  |     from .main import BaseConfig as BaseConfig
revertrisk-multilingual-1  |   File "/opt/lib/python/site-packages/fastapi/_compat/main.py", line 12, in <module>
revertrisk-multilingual-1  |     from fastapi._compat import may_v1
revertrisk-multilingual-1  |   File "/opt/lib/python/site-packages/fastapi/_compat/may_v1.py", line 4, in <module>
revertrisk-multilingual-1  |     from fastapi.types import ModelNameMap
revertrisk-multilingual-1  |   File "/opt/lib/python/site-packages/fastapi/types.py", line 5, in <module>
revertrisk-multilingual-1  |     from pydantic import BaseModel
revertrisk-multilingual-1  |   File "/opt/lib/python/site-packages/pydantic/__init__.py", line 5, in <module>
revertrisk-multilingual-1  |     from ._migration import getattr_migration
revertrisk-multilingual-1  |   File "/opt/lib/python/site-packages/pydantic/_migration.py", line 4, in <module>
revertrisk-multilingual-1  |     from pydantic.warnings import PydanticDeprecatedSince20
revertrisk-multilingual-1  |   File "/opt/lib/python/site-packages/pydantic/warnings.py", line 5, in <module>
revertrisk-multilingual-1  |     from .version import version_short
revertrisk-multilingual-1  |   File "/opt/lib/python/site-packages/pydantic/version.py", line 7, in <module>
revertrisk-multilingual-1  |     from pydantic_core import __version__ as __pydantic_core_version__
revertrisk-multilingual-1  |   File "/opt/lib/python/site-packages/pydantic_core/__init__.py", line 6, in <module>
revertrisk-multilingual-1  |     from typing_extensions import Sentinel
revertrisk-multilingual-1  | ImportError: cannot import name 'Sentinel' from 'typing_extensions' (/opt/lib/python/site-packages/typing_extensions.py)
revertrisk-multilingual-1 exited with code 1

Details

Related Changes in Gerrit:

	Subject	Repo	Branch	Lines +/-
	revertrisk: Add CPU version for revertrisk-multilingual.	machinelearning/liftwing/inference-services	main	+94 -0

Customize query in gerrit

Event Timeline

isarantopoulos created this task.Oct 23 2025, 8:17 AM

Restricted Application added a subscriber: Aklapper. · View Herald TranscriptOct 23 2025, 8:17 AM

isarantopoulos assigned this task to BWojtowicz-WMF.Oct 23 2025, 8:17 AM

This is issue was raised by @jsn.sherman on slack.
The current model in production is https://analytics.wikimedia.org/published/wmf-ml-models/revertrisk/multilingual/20230810110019/.
I'm not sure if this is the one that was tested as there is also a newer model available in https://analytics.wikimedia.org/published/wmf-ml-models/revertrisk/multilingual/20250605145811/
Ideally the local setup should work with both.

Looking into it! I can reproduce this issue on my machine. I’ve also confirmed that we luckily don’t encounter this issue on LiftWing, which is interesting.

The error seems to originate from the typing_extensions being too old for our version of pydantic-core, however we don’t specify typing_extensions dependency anywhere.

I've tried pinning the version in our requirements file to the newest typing_extensions==4.15.0, which contains the Sentinel class, but this didn't solve the problem. Looking into docker build logs, I can see the following:

Successfully installed [...] typing_extensions-4.15.0 [...]
WARNING: Target directory /opt/lib/python/site-packages/typing_extensions.py already exists. Specify --upgrade to force replacement.

This might suggest that pip believes to install the newest version, however there might be an old version hiding in /opt/lib/python/site-packages/typing_extensions.py.

Investigating further...

Change #1198282 had a related patch set uploaded (by Bartosz Wójtowicz; author: Bartosz Wójtowicz):

[machinelearning/liftwing/inference-services@main] revertrisk: Ensure new 'typing_extensions' version is used in docker.

https://gerrit.wikimedia.org/r/1198282

gerritbot added a project: Patch-For-Review.Oct 23 2025, 8:38 AM

I think I found the culprit - the issue stems from our base docker image, which contains the old version of typing_extensions preinstalled in /opt/lib/python/site-packages/typing_extensions.py. However, just adding the pin to typing_extensions==4.15.0 in requirements.txt does not solve the issue as I shared in https://phabricator.wikimedia.org/T408068#11301601.

What solves this for me is this change: https://gerrit.wikimedia.org/r/c/machinelearning/liftwing/inference-services/+/1198282/.
However, it's not very pretty. Will see what else can we do.

It's still unclear to me as to why we don't experience it on LiftWing.

Thanks for looking into this. I'm now seeing what looks like another dependency error:

Attaching to revertrisk-multilingual-1
revertrisk-multilingual-1  | + source common_settings.sh
revertrisk-multilingual-1  | +++ /usr/bin/python3 -c 'from python.resource_utils import get_cpu_count; print(get_cpu_count())'
revertrisk-multilingual-1  | CPU count detected from get_cpu_count: 20
revertrisk-multilingual-1  | OMP_NUM_THREADS set to: 20
revertrisk-multilingual-1  | ++ CPU_COUNT=20
revertrisk-multilingual-1  | ++ echo 'CPU count detected from get_cpu_count: 20'
revertrisk-multilingual-1  | ++ export OMP_NUM_THREADS=20
revertrisk-multilingual-1  | ++ OMP_NUM_THREADS=20
revertrisk-multilingual-1  | ++ echo 'OMP_NUM_THREADS set to: 20'
revertrisk-multilingual-1  | + MODEL_SERVER_PATH=model_server/model.py
revertrisk-multilingual-1  | + exec /usr/bin/python3 model_server/model.py
revertrisk-multilingual-1  | Traceback (most recent call last):
revertrisk-multilingual-1  |   File "/srv/revert_risk_model/model_server/model.py", line 40, in <module>
revertrisk-multilingual-1  |     model = RevisionRevertRiskModel(
revertrisk-multilingual-1  |             ^^^^^^^^^^^^^^^^^^^^^^^^
revertrisk-multilingual-1  |   File "/srv/revert_risk_model/model_server/base_model.py", line 49, in __init__
revertrisk-multilingual-1  |     self.load()
revertrisk-multilingual-1  |   File "/srv/revert_risk_model/model_server/base_model.py", line 134, in load
revertrisk-multilingual-1  |     self.model = self.ModelLoader.load_model(self.model_path)
revertrisk-multilingual-1  |                  ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
revertrisk-multilingual-1  |   File "/opt/lib/python/site-packages/knowledge_integrity/models/revertrisk_multilingual/model.py", line 345, in load_model
revertrisk-multilingual-1  |     model: MultilingualRevertRiskModel = joblib.load(f)
revertrisk-multilingual-1  |                                          ^^^^^^^^^^^^^^
revertrisk-multilingual-1  |   File "/opt/lib/python/site-packages/joblib/numpy_pickle.py", line 733, in load
revertrisk-multilingual-1  |     obj = _unpickle(fobj, ensure_native_byte_order=ensure_native_byte_order)
revertrisk-multilingual-1  |           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
revertrisk-multilingual-1  |   File "/opt/lib/python/site-packages/joblib/numpy_pickle.py", line 626, in _unpickle
revertrisk-multilingual-1  |     obj = unpickler.load()
revertrisk-multilingual-1  |           ^^^^^^^^^^^^^^^^
revertrisk-multilingual-1  |   File "/usr/lib/python3.11/pickle.py", line 1213, in load
revertrisk-multilingual-1  |     dispatch[key[0]](self)
revertrisk-multilingual-1  |   File "/usr/lib/python3.11/pickle.py", line 1538, in load_stack_global
revertrisk-multilingual-1  |     self.append(self.find_class(module, name))
revertrisk-multilingual-1  |                 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
revertrisk-multilingual-1  |   File "/usr/lib/python3.11/pickle.py", line 1580, in find_class
revertrisk-multilingual-1  |     __import__(module, level=0)
revertrisk-multilingual-1  |   File "/opt/lib/python/site-packages/knowledge_integrity/models/revertrisk.py", line 11, in <module>
revertrisk-multilingual-1  |     import xgboost as xgb  # type: ignore
revertrisk-multilingual-1  |     ^^^^^^^^^^^^^^^^^^^^^
revertrisk-multilingual-1  | ModuleNotFoundError: No module named 'xgboost'
revertrisk-multilingual-1 exited with code 1

(note that I'm on an x86 linux machine)

@jsn.sherman
Hmm this is very interesting, I could not reproduce it on my Mac machine yet. Can you share the exact commands that you are running?

I see that you're running the revertrisk-multilingual service. In this gerrit patch (https://gerrit.wikimedia.org/r/c/machinelearning/liftwing/inference-services/+/1198282), I've added a new revertrisk-multilingual-cpu option, which should be suited to be run locally on non-GPU machines. Could you verify if the revertrisk-multilingual-cpu version does work on your machine?

Running it via the commands below works on my Mac machine, assuming you have the model.pkl file stored in ./models/ directory:

docker compose build revertrisk-multilingual-cpu

PATH_TO_REVERTRISK_MULTILINGUAL_MODEL="./models/" docker compose up revertrisk-multilingual-cpu

Next, I query it as follows:

curl localhost:8080/v1/models/revertrisk-multilingual:predict -i -X POST -d '{"lang": "en", "rev_id": 12345}'

In the meantime, I'll try reproducing the issue you've shared for revertrisk-multilingual.

Yep, I have the model in the same place as you. I get same output for revertrisk-multilingual-cpu:

$ docker compose --env-file .env up --build revertrisk-multilingual-cpu
[...]
revertrisk-multilingual-cpu-1  | + source common_settings.sh
revertrisk-multilingual-cpu-1  | +++ /usr/bin/python3 -c 'from python.resource_utils import get_cpu_count; print(get_cpu_count())'
revertrisk-multilingual-cpu-1  | CPU count detected from get_cpu_count: 20
revertrisk-multilingual-cpu-1  | OMP_NUM_THREADS set to: 20
revertrisk-multilingual-cpu-1  | ++ CPU_COUNT=20
revertrisk-multilingual-cpu-1  | ++ echo 'CPU count detected from get_cpu_count: 20'
revertrisk-multilingual-cpu-1  | ++ export OMP_NUM_THREADS=20
revertrisk-multilingual-cpu-1  | ++ OMP_NUM_THREADS=20
revertrisk-multilingual-cpu-1  | ++ echo 'OMP_NUM_THREADS set to: 20'
revertrisk-multilingual-cpu-1  | + MODEL_SERVER_PATH=model_server/model.py
revertrisk-multilingual-cpu-1  | + exec /usr/bin/python3 model_server/model.py
revertrisk-multilingual-cpu-1  | Traceback (most recent call last):
revertrisk-multilingual-cpu-1  |   File "/srv/revert_risk_model/model_server/model.py", line 40, in <module>
revertrisk-multilingual-cpu-1  |     model = RevisionRevertRiskModel(
revertrisk-multilingual-cpu-1  |             ^^^^^^^^^^^^^^^^^^^^^^^^
revertrisk-multilingual-cpu-1  |   File "/srv/revert_risk_model/model_server/base_model.py", line 49, in __init__
revertrisk-multilingual-cpu-1  |     self.load()
revertrisk-multilingual-cpu-1  |   File "/srv/revert_risk_model/model_server/base_model.py", line 134, in load
revertrisk-multilingual-cpu-1  |     self.model = self.ModelLoader.load_model(self.model_path)
revertrisk-multilingual-cpu-1  |                  ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
revertrisk-multilingual-cpu-1  |   File "/opt/lib/venv/lib/python3.11/site-packages/knowledge_integrity/models/revertrisk_multilingual/model.py", line 345, in load_model
revertrisk-multilingual-cpu-1  |     model: MultilingualRevertRiskModel = joblib.load(f)
revertrisk-multilingual-cpu-1  |                                          ^^^^^^^^^^^^^^
revertrisk-multilingual-cpu-1  |   File "/opt/lib/venv/lib/python3.11/site-packages/joblib/numpy_pickle.py", line 733, in load
revertrisk-multilingual-cpu-1  |     obj = _unpickle(fobj, ensure_native_byte_order=ensure_native_byte_order)
revertrisk-multilingual-cpu-1  |           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
revertrisk-multilingual-cpu-1  |   File "/opt/lib/venv/lib/python3.11/site-packages/joblib/numpy_pickle.py", line 626, in _unpickle
revertrisk-multilingual-cpu-1  |     obj = unpickler.load()
revertrisk-multilingual-cpu-1  |           ^^^^^^^^^^^^^^^^
revertrisk-multilingual-cpu-1  |   File "/usr/lib/python3.11/pickle.py", line 1213, in load
revertrisk-multilingual-cpu-1  |     dispatch[key[0]](self)
revertrisk-multilingual-cpu-1  |   File "/usr/lib/python3.11/pickle.py", line 1538, in load_stack_global
revertrisk-multilingual-cpu-1  |     self.append(self.find_class(module, name))
revertrisk-multilingual-cpu-1  |                 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
revertrisk-multilingual-cpu-1  |   File "/usr/lib/python3.11/pickle.py", line 1580, in find_class
revertrisk-multilingual-cpu-1  |     __import__(module, level=0)
revertrisk-multilingual-cpu-1  |   File "/opt/lib/venv/lib/python3.11/site-packages/knowledge_integrity/models/revertrisk.py", line 11, in <module>
revertrisk-multilingual-cpu-1  |     import xgboost as xgb  # type: ignore
revertrisk-multilingual-cpu-1  |     ^^^^^^^^^^^^^^^^^^^^^
revertrisk-multilingual-cpu-1  | ModuleNotFoundError: No module named 'xgboost'
revertrisk-multilingual-cpu-1 exited with code 1

For good measure, I'll go ahead and run things exactly as you have and share the output.

$ docker compose build revertrisk-multilingual-cpu
[+] Building 91.5s (26/26) FINISHED
 => [internal] load local bake definitions                                                                             0.0s
 => => reading from stdin 691B                                                                                         0.0s
 => [internal] load build definition from multilingual-cpu.yaml                                                        0.0s
 => => transferring dockerfile: 1.57kB                                                                                 0.0s
 => resolve image config for docker-image://docker-registry.wikimedia.org/repos/releng/blubber/buildkit:v1.1.0         0.4s
 => docker-image://docker-registry.wikimedia.org/repos/releng/blubber/buildkit:v1.1.0@sha256:ed64673e7c362f5f264f92ea  0.7s
 => => resolve docker-registry.wikimedia.org/repos/releng/blubber/buildkit:v1.1.0@sha256:ed64673e7c362f5f264f92ea36bf  0.0s
 => => sha256:ed64673e7c362f5f264f92ea36bf379f47c90699ee36d32ed6a96aba7de858f2 1.61kB / 1.61kB                         0.0s
 => => sha256:ce07fb73962e0ec1e9c992b046a4a6e92f24383a96a605b6b3d0e670d3417d24 672B / 672B                             0.0s
 => => sha256:e8fbf5af2b062cc063b0d3fe8e1708e4cc181f4b708051250fbafa5f6cc382c2 917B / 917B                             0.0s
 => => sha256:96ee446af73074f925ee6439526ba91e78855f72ac8577002559e54f128f04b3 13.46MB / 13.46MB                       0.5s
 => => sha256:eb456fbdecdfce8fa20aa382307f185bbf574ffa5681738c791481d66ca2603c 122.68kB / 122.68kB                     0.1s
 => => extracting sha256:96ee446af73074f925ee6439526ba91e78855f72ac8577002559e54f128f04b3                              0.1s
 => => extracting sha256:eb456fbdecdfce8fa20aa382307f185bbf574ffa5681738c791481d66ca2603c                              0.0s
 => [internal] load .dockerignore                                                                                      0.0s
 => => transferring context: 88B                                                                                       0.0s
 => [build]      🌐 docker-registry.wikimedia.org/bookworm:20251019@sha256:20067208382d3086a3e519d04735cc570e2480e9     2.2s
 => => resolve docker-registry.wikimedia.org/bookworm:20251019@sha256:20067208382d3086a3e519d04735cc570e2480e9f5abcd2  0.2s
 => => sha256:20067208382d3086a3e519d04735cc570e2480e9f5abcd2d303a1d1fa3f9b3c3 529B / 529B                             0.0s
 => => sha256:31b725383487ae4367d025581b76a3e5bdd67bd307cb20ee5e67da8ad92a763b 1.63kB / 1.63kB                         0.0s
 => => sha256:77a1eeafdb5ad24781a5bbfa6ade20d114143406385f13543e236184f6df9f3b 29.15MB / 29.15MB                       1.1s
 => => extracting sha256:77a1eeafdb5ad24781a5bbfa6ade20d114143406385f13543e236184f6df9f3b                              0.7s
 => [internal] load build context                                                                                      0.0s
 => => transferring context: 2.61MB                                                                                    0.0s
 => [build]      🖥️ # apt-get update && apt-get install -y "build-essential" "python3-pip" "python3-dev" "pytho       21.3s
 => [production] 🖥️ # apt-get update && apt-get install -y "python3" "python3-distutils" "python3-setuptools" "       10.3s
 => [production] 🖥️ # (getent group "65533" || groupadd -o -g "65533" -r "somebody") && (getent passwd "65533" |       0.2s
 => [production] 🖥️ # (getent group "900" || groupadd -o -g "900" -r "runuser") && (getent passwd "900" || usera       0.2s
 => [production] 📂 [src/models/revert_risk_model/data] -> data                                                         0.0s
 => [production] 📂 [src/models/revert_risk_model/model_server] -> model_server                                         0.0s
 => [production] 📂 [python] -> python/                                                                                 0.0s
 => [build]      🖥️ # (getent group "65533" || groupadd -o -g "65533" -r "somebody") && (getent passwd "65533" |       0.2s
 => [build]      🖥️ # (getent group "900" || groupadd -o -g "900" -r "runuser") && (getent passwd "900" || usera       0.2s
 => [build]      📂 [src/models/revert_risk_model/model_server/multilingual/requirements.txt] -> src/models/revert_     0.0s
 => [build]      📂 [python/requirements.txt] -> python/                                                                0.0s
 => [build]      🖥️ @65533 $ python3 "-m" "venv" "/opt/lib/venv"                                                       2.2s
 => [build]      🖥️ @65533 $ python3 "-m" "pip" "install" "-U" "setuptools!=60.9.0" && python3 "-m" "pip" "insta       4.3s
 => [build]      🖥️ @65533 $ python3 "-m" "pip" "install" "-r" "src/models/revert_risk_model/model_server/multi       49.4s
 => [production] 📦 {build}[/opt/lib/venv/lib/python3.11/site-packages/] -> /opt/lib/venv/lib/python3.11/site-packa     2.4s
 => [production] 📂 [model_server_entrypoint.sh] -> entrypoint.sh                                                       0.0s
 => [production] 📂 [common_settings.sh] -> common_settings.sh                                                          0.0s
 => exporting to image                                                                                                 5.5s
 => => exporting layers                                                                                                5.5s
 => => writing image sha256:bd01687270853c3a355823cd6e7da8611188c8a9f270edb6760e8d259f84b1b1                           0.0s
 => => naming to docker.io/library/rrml:cpu                                                                            0.0s
 => resolving provenance for metadata file                                                                             0.0s
[+] Building 1/1
 ✔ rrml:cpu  Built                                                                                                     0.0s

$ PATH_TO_REVERTRISK_MULTILINGUAL_MODEL="./models/" docker compose up revertrisk-multilingual-cpu
[+] Running 2/2
 ✔ Network inference-services_default                          Created                                                 0.1s
 ✔ Container inference-services-revertrisk-multilingual-cpu-1  Created                                                 0.0s
Attaching to revertrisk-multilingual-cpu-1
revertrisk-multilingual-cpu-1  | + source common_settings.sh
revertrisk-multilingual-cpu-1  | +++ /usr/bin/python3 -c 'from python.resource_utils import get_cpu_count; print(get_cpu_count())'
revertrisk-multilingual-cpu-1  | CPU count detected from get_cpu_count: 20
revertrisk-multilingual-cpu-1  | OMP_NUM_THREADS set to: 20
revertrisk-multilingual-cpu-1  | ++ CPU_COUNT=20
revertrisk-multilingual-cpu-1  | ++ echo 'CPU count detected from get_cpu_count: 20'
revertrisk-multilingual-cpu-1  | ++ export OMP_NUM_THREADS=20
revertrisk-multilingual-cpu-1  | ++ OMP_NUM_THREADS=20
revertrisk-multilingual-cpu-1  | ++ echo 'OMP_NUM_THREADS set to: 20'
revertrisk-multilingual-cpu-1  | + MODEL_SERVER_PATH=model_server/model.py
revertrisk-multilingual-cpu-1  | + exec /usr/bin/python3 model_server/model.py
revertrisk-multilingual-cpu-1  | Traceback (most recent call last):
revertrisk-multilingual-cpu-1  |   File "/srv/revert_risk_model/model_server/model.py", line 40, in <module>
revertrisk-multilingual-cpu-1  |     model = RevisionRevertRiskModel(
revertrisk-multilingual-cpu-1  |             ^^^^^^^^^^^^^^^^^^^^^^^^
revertrisk-multilingual-cpu-1  |   File "/srv/revert_risk_model/model_server/base_model.py", line 49, in __init__
revertrisk-multilingual-cpu-1  |     self.load()
revertrisk-multilingual-cpu-1  |   File "/srv/revert_risk_model/model_server/base_model.py", line 134, in load
revertrisk-multilingual-cpu-1  |     self.model = self.ModelLoader.load_model(self.model_path)
revertrisk-multilingual-cpu-1  |                  ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
revertrisk-multilingual-cpu-1  |   File "/opt/lib/venv/lib/python3.11/site-packages/knowledge_integrity/models/revertrisk_multilingual/model.py", line 345, in load_model
revertrisk-multilingual-cpu-1  |     model: MultilingualRevertRiskModel = joblib.load(f)
revertrisk-multilingual-cpu-1  |                                          ^^^^^^^^^^^^^^
revertrisk-multilingual-cpu-1  |   File "/opt/lib/venv/lib/python3.11/site-packages/joblib/numpy_pickle.py", line 733, in load
revertrisk-multilingual-cpu-1  |     obj = _unpickle(fobj, ensure_native_byte_order=ensure_native_byte_order)
revertrisk-multilingual-cpu-1  |           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
revertrisk-multilingual-cpu-1  |   File "/opt/lib/venv/lib/python3.11/site-packages/joblib/numpy_pickle.py", line 626, in _unpickle
revertrisk-multilingual-cpu-1  |     obj = unpickler.load()
revertrisk-multilingual-cpu-1  |           ^^^^^^^^^^^^^^^^
revertrisk-multilingual-cpu-1  |   File "/usr/lib/python3.11/pickle.py", line 1213, in load
revertrisk-multilingual-cpu-1  |     dispatch[key[0]](self)
revertrisk-multilingual-cpu-1  |   File "/usr/lib/python3.11/pickle.py", line 1538, in load_stack_global
revertrisk-multilingual-cpu-1  |     self.append(self.find_class(module, name))
revertrisk-multilingual-cpu-1  |                 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
revertrisk-multilingual-cpu-1  |   File "/usr/lib/python3.11/pickle.py", line 1580, in find_class
revertrisk-multilingual-cpu-1  |     __import__(module, level=0)
revertrisk-multilingual-cpu-1  |   File "/opt/lib/venv/lib/python3.11/site-packages/knowledge_integrity/models/revertrisk.py", line 11, in <module>
revertrisk-multilingual-cpu-1  |     import xgboost as xgb  # type: ignore
revertrisk-multilingual-cpu-1  |     ^^^^^^^^^^^^^^^^^^^^^
revertrisk-multilingual-cpu-1  | ModuleNotFoundError: No module named 'xgboost'
revertrisk-multilingual-cpu-1 exited with code 1

Thank you for helping and sharing all the logs!

I've pruned docker on my Mac machine, but I'm still having trouble reproducing the issue you're experiencing. I've also checked on my personal Linux machine running Ubuntu 22.04 and I did not experience this.
I can suggest one more alternative to try (besides pruning docker), which is running the model locally without docker. You should be able to do it via running make revertrisk-multilingual in the root of inference-services repository. I've verified that this works on my Linux machine as well.

If this won't work, I'd suggest to set up a pair debugging session early next week to try to get to the bottom of this. @jsn.sherman Would you be up for it?

I'll also paste my build logs below in case they might be helpful:

Mac Build:

➜  inference-services git:(revertrisk/fix-multilingual-docker-build) ✗ docker compose build revertrisk-multilingual-cpu
Compose can now delegate builds to bake for better performance.
 To do so, set COMPOSE_BAKE=true.
[+] Building 466.9s (25/25) FINISHED                                                                                                                           docker:desktop-linux
 => [revertrisk-multilingual-cpu internal] load build definition from multilingual-cpu.yaml                                                                                    0.0s
 => => transferring dockerfile: 1.61kB                                                                                                                                         0.0s
 => [revertrisk-multilingual-cpu] resolve image config for docker-image://docker-registry.wikimedia.org/repos/releng/blubber/buildkit:v1.1.0                                   0.8s
 => CACHED [revertrisk-multilingual-cpu] docker-image://docker-registry.wikimedia.org/repos/releng/blubber/buildkit:v1.1.0@sha256:ed64673e7c362f5f264f92ea36bf379f47c90699ee3  0.0s
 => => resolve docker-registry.wikimedia.org/repos/releng/blubber/buildkit:v1.1.0@sha256:ed64673e7c362f5f264f92ea36bf379f47c90699ee36d32ed6a96aba7de858f2                      0.0s
 => [revertrisk-multilingual-cpu internal] load .dockerignore                                                                                                                  0.0s
 => => transferring context: 529B                                                                                                                                              0.0s
 => [revertrisk-multilingual-cpu internal] load build context                                                                                                                  0.4s
 => => transferring context: 1.17MB                                                                                                                                            0.4ss
 => [revertrisk-multilingual-cpu build]      🌐 docker-registry.wikimedia.org/bookworm:20251019@sha256:20067208382d3086a3e519d04735cc570e2480e9f5abcd2d303a1d1fa3f9b3c3         0.8s
 => => resolve docker-registry.wikimedia.org/bookworm:20251019@sha256:20067208382d3086a3e519d04735cc570e2480e9f5abcd2d303a1d1fa3f9b3c3                                         0.8s
 => [revertrisk-multilingual-cpu production] 🖥️ # apt-get update && apt-get install -y "python3" "python3-distutils" "python3-setuptools" "python3.11-venv" && rm -rf /       32.3s
 => [revertrisk-multilingual-cpu build]      🖥️ # apt-get update && apt-get install -y "build-essential" "python3-pip" "python3-dev" "python3-setuptools" "python3.11-v       71.6s
 => [revertrisk-multilingual-cpu production] 🖥️ # (getent group "65533" || groupadd -o -g "65533" -r "somebody") && (getent passwd "65533" || useradd -l -o -m -d "/home       0.2s
 => [revertrisk-multilingual-cpu production] 🖥️ # (getent group "900" || groupadd -o -g "900" -r "runuser") && (getent passwd "900" || useradd -l -o -m -d "/home/runuse       0.2s0s
 => [revertrisk-multilingual-cpu production] 📂 [src/models/revert_risk_model/data] -> data                                                                                     0.0s
 => [revertrisk-multilingual-cpu production] 📂 [src/models/revert_risk_model/model_server] -> model_server                                                                     0.0s
 => [revertrisk-multilingual-cpu production] 📂 [python] -> python/                                                                                                             0.0s
 => [revertrisk-multilingual-cpu build]      🖥️ # (getent group "65533" || groupadd -o -g "65533" -r "somebody") && (getent passwd "65533" || useradd -l -o -m -d "/home       0.3s
 => [revertrisk-multilingual-cpu build]      🖥️ # (getent group "900" || groupadd -o -g "900" -r "runuser") && (getent passwd "900" || useradd -l -o -m -d "/home/runuse       0.2s0s
 => [revertrisk-multilingual-cpu build]      📂 [src/models/revert_risk_model/model_server/multilingual/requirements.txt] -> src/models/revert_risk_model/model_server/mult     0.0s
 => [revertrisk-multilingual-cpu build]      📂 [python/requirements.txt] -> python/                                                                                            0.0s
 => [revertrisk-multilingual-cpu build]      🖥️ @65533 $ python3 "-m" "venv" "/opt/lib/venv"                                                                                   2.0s
 => [revertrisk-multilingual-cpu build]      🖥️ @65533 $ python3 "-m" "pip" "install" "-U" "setuptools!=60.9.0" && python3 "-m" "pip" "install" "-U" "wheel" "tox" "pip       13.8s
 => [revertrisk-multilingual-cpu build]      🖥️ @65533 $ python3 "-m" "pip" "install" "-r" "src/models/revert_risk_model/model_server/multilingual/requirements.txt" "       341.7s0s
 => [revertrisk-multilingual-cpu production] 📦 {build}[/opt/lib/venv/lib/python3.11/site-packages/] -> /opt/lib/venv/lib/python3.11/site-packages/                             2.0s
 => [revertrisk-multilingual-cpu production] 📂 [model_server_entrypoint.sh] -> entrypoint.sh                                                                                   0.1s
 => [revertrisk-multilingual-cpu production] 📂 [common_settings.sh] -> common_settings.sh                                                                                      0.0s
 => [revertrisk-multilingual-cpu] exporting to image                                                                                                                          30.8s
 => => exporting layers                                                                                                                                                       30.7s
 => => exporting manifest sha256:a125400e7be167b4959937739304417de3386d2a139e30249a732d88db924e1f                                                                              0.0s
 => => exporting config sha256:db63a894b64f0f034c423a7ee5e3c6af3b8592eea3cfb2d63cb5a1e7a2a20b5e                                                                                0.0s
 => => exporting attestation manifest sha256:351e943a17d28881e3214cfa3235fd7099ed11d72be479fb5457052a0aa5ab75                                                                  0.0s
 => => exporting manifest list sha256:8f8fc26b61a4dd243e2522b747be8251b2bab172f627a8e91f337b4a31a0714e                                                                         0.0s
 => => naming to docker.io/library/rrml:cpu                                                                                                                                    0.0s
 => [revertrisk-multilingual-cpu] resolving provenance for metadata file                                                                                                       0.0s
[+] Building 1/1
 ✔ revertrisk-multilingual-cpu  Built                                                                                                                                          0.0s
➜  inference-services git:(revertrisk/fix-multilingual-docker-build) ✗ PATH_TO_REVERTRISK_MULTILINGUAL_MODEL="./models/revertrisk/multilingual/20230810110019/" docker compose up revertrisk-multilingual-cpu
[+] Running 2/2
 ✔ Network inference-services_default                          Created                                                                                                         0.0s
 ✔ Container inference-services-revertrisk-multilingual-cpu-1  Created                                                                                                         6.1s
Attaching to revertrisk-multilingual-cpu-1
revertrisk-multilingual-cpu-1  | + source common_settings.sh
revertrisk-multilingual-cpu-1  | +++ /usr/bin/python3 -c 'from python.resource_utils import get_cpu_count; print(get_cpu_count())'
revertrisk-multilingual-cpu-1  | ++ CPU_COUNT=16
revertrisk-multilingual-cpu-1  | ++ echo 'CPU count detected from get_cpu_count: 16'
revertrisk-multilingual-cpu-1  | CPU count detected from get_cpu_count: 16
revertrisk-multilingual-cpu-1  | ++ export OMP_NUM_THREADS=16
revertrisk-multilingual-cpu-1  | ++ OMP_NUM_THREADS=16
revertrisk-multilingual-cpu-1  | ++ echo 'OMP_NUM_THREADS set to: 16'
revertrisk-multilingual-cpu-1  | OMP_NUM_THREADS set to: 16
revertrisk-multilingual-cpu-1  | + MODEL_SERVER_PATH=model_server/model.py
revertrisk-multilingual-cpu-1  | + exec /usr/bin/python3 model_server/model.py
revertrisk-multilingual-cpu-1  | INFO:root:Successfully loaded 342 canonical wiki languages.
revertrisk-multilingual-cpu-1  | 2025-10-24 08:41:41.046 1 kserve INFO [model_server.py:register_model():402] Registering model: revertrisk-multilingual
revertrisk-multilingual-cpu-1  | 2025-10-24 08:41:41.046 1 kserve INFO [model_server.py:setup_event_loop():282] Setting max asyncio worker threads as 20
revertrisk-multilingual-cpu-1  | 2025-10-24 08:41:41.096 1 kserve INFO [server.py:_register_endpoints():110] OpenAI endpoints not registered
revertrisk-multilingual-cpu-1  | 2025-10-24 08:41:41.096 1 kserve INFO [server.py:start():161] Starting uvicorn with 1 workers
revertrisk-multilingual-cpu-1  | 2025-10-24 08:41:41.105 1 uvicorn.error INFO:     Started server process [1]
revertrisk-multilingual-cpu-1  | 2025-10-24 08:41:41.105 1 uvicorn.error INFO:     Waiting for application startup.
revertrisk-multilingual-cpu-1  | 2025-10-24 08:41:41.131 1 kserve INFO [server.py:start():70] Starting gRPC server with 4 workers
revertrisk-multilingual-cpu-1  | 2025-10-24 08:41:41.131 1 kserve INFO [server.py:start():71] Starting gRPC server on [::]:8081
revertrisk-multilingual-cpu-1  | 2025-10-24 08:41:41.133 1 uvicorn.error INFO:     Application startup complete.
revertrisk-multilingual-cpu-1  | 2025-10-24 08:41:41.134 1 uvicorn.error INFO:     Uvicorn running on http://0.0.0.0:8080 (Press CTRL+C to quit)

Linux build:

bart@bart:~/projs/inference-services$ docker compose build revertrisk-multilingual-cpu
Compose now can delegate build to bake for better performances
Just set COMPOSE_BAKE=true
[+] Building 363.4s (25/25) FINISHED                                                                                                                         docker:default
 => [revertrisk-multilingual-cpu internal] load build definition from multilingual-cpu.yaml                                                                            0.0s
 => => transferring dockerfile: 1.57kB                                                                                                                                 0.0s
 => [revertrisk-multilingual-cpu] resolve image config for docker-image://docker-registry.wikimedia.org/repos/releng/blubber/buildkit:v1.1.0                           2.6s
 => [revertrisk-multilingual-cpu] docker-image://docker-registry.wikimedia.org/repos/releng/blubber/buildkit:v1.1.0@sha256:ed64673e7c362f5f264f92ea36bf379f47c90699ee  3.2s
 => => resolve docker-registry.wikimedia.org/repos/releng/blubber/buildkit:v1.1.0@sha256:ed64673e7c362f5f264f92ea36bf379f47c90699ee36d32ed6a96aba7de858f2              0.0s
 => => sha256:e8fbf5af2b062cc063b0d3fe8e1708e4cc181f4b708051250fbafa5f6cc382c2 917B / 917B                                                                             0.0s
 => => sha256:96ee446af73074f925ee6439526ba91e78855f72ac8577002559e54f128f04b3 13.46MB / 13.46MB                                                                       3.0s
 => => sha256:eb456fbdecdfce8fa20aa382307f185bbf574ffa5681738c791481d66ca2603c 122.68kB / 122.68kB                                                                     0.9s
 => => sha256:ed64673e7c362f5f264f92ea36bf379f47c90699ee36d32ed6a96aba7de858f2 1.61kB / 1.61kB                                                                         0.0s
 => => sha256:ce07fb73962e0ec1e9c992b046a4a6e92f24383a96a605b6b3d0e670d3417d24 672B / 672B                                                                             0.0ss
 => => extracting sha256:96ee446af73074f925ee6439526ba91e78855f72ac8577002559e54f128f04b3                                                                              0.1ss
 => => extracting sha256:eb456fbdecdfce8fa20aa382307f185bbf574ffa5681738c791481d66ca2603c                                                                              0.0ss
 => [revertrisk-multilingual-cpu internal] load .dockerignore                                                                                                          0.0ss
 => => transferring context: 88B                                                                                                                                       0.0ss
 => [revertrisk-multilingual-cpu internal] load build context                                                                                                         23.0ss
 => => transferring context: 2.65GB                                                                                                                                   22.8ss
 => [revertrisk-multilingual-cpu build]      🌐 docker-registry.wikimedia.org/bookworm:20251019@sha256:20067208382d3086a3e519d04735cc570e2480e9f5abcd2d303a1d1fa3f     21.3s
 => => resolve docker-registry.wikimedia.org/bookworm:20251019@sha256:20067208382d3086a3e519d04735cc570e2480e9f5abcd2d303a1d1fa3f9b3c3                                 2.1s
 => => sha256:77a1eeafdb5ad24781a5bbfa6ade20d114143406385f13543e236184f6df9f3b 29.15MB / 29.15MB                                                                       8.9s ^ => => sha256:20067208382d3086a3e519d04735cc570e2480e9f5abcd2d303a1d1fa3f9b3c3 529B / 529B                                                                             0.0ss
 => => sha256:31b725383487ae4367d025581b76a3e5bdd67bd307cb20ee5e67da8ad92a763b 1.63kB / 1.63kB                                                                         0.0ss
 => => extracting sha256:77a1eeafdb5ad24781a5bbfa6ade20d114143406385f13543e236184f6df9f3b                                                                              5.1ss
 => [revertrisk-multilingual-cpu production] 🖥️ # apt-get update && apt-get install -y "python3" "python3-distutils" "python3-setuptools" "python3.11-venv" &&        56.2s0s
 => [revertrisk-multilingual-cpu build]      🖥️ # apt-get update && apt-get install -y "build-essential" "python3-pip" "python3-dev" "python3-setuptools" "pyth       88.5s0s
 => [revertrisk-multilingual-cpu production] 🖥️ # (getent group "65533" || groupadd -o -g "65533" -r "somebody") && (getent passwd "65533" || useradd -l -o -m -       0.2s0s
 => [revertrisk-multilingual-cpu production] 🖥️ # (getent group "900" || groupadd -o -g "900" -r "runuser") && (getent passwd "900" || useradd -l -o -m -d "/hom       0.3s0s
 => [revertrisk-multilingual-cpu production] 📂 [src/models/revert_risk_model/data] -> data                                                                             0.1s
 => [revertrisk-multilingual-cpu production] 📂 [src/models/revert_risk_model/model_server] -> model_server                                                             0.0s
 => [revertrisk-multilingual-cpu production] 📂 [python] -> python/                                                                                                     0.0s
 => [revertrisk-multilingual-cpu build]      🖥️ # (getent group "65533" || groupadd -o -g "65533" -r "somebody") && (getent passwd "65533" || useradd -l -o -m -       1.8s1s
 => [revertrisk-multilingual-cpu build]      🖥️ # (getent group "900" || groupadd -o -g "900" -r "runuser") && (getent passwd "900" || useradd -l -o -m -d "/hom       0.3s9s
 => [revertrisk-multilingual-cpu build]      📂 [src/models/revert_risk_model/model_server/multilingual/requirements.txt] -> src/models/revert_risk_model/model_ser     0.0s
 => [revertrisk-multilingual-cpu build]      📂 [python/requirements.txt] -> python/                                                                                    0.0s
 => [revertrisk-multilingual-cpu build]      🖥️ @65533 $ python3 "-m" "venv" "/opt/lib/venv"                                                                           3.5sss
 => [revertrisk-multilingual-cpu build]      🖥️ @65533 $ python3 "-m" "pip" "install" "-U" "setuptools!=60.9.0" && python3 "-m" "pip" "install" "-U" "wheel" "to       8.5sss
 => [revertrisk-multilingual-cpu build]      🖥️ @65533 $ python3 "-m" "pip" "install" "-r" "src/models/revert_risk_model/model_server/multilingual/requirement       182.6sss
 => [revertrisk-multilingual-cpu production] 📦 {build}[/opt/lib/venv/lib/python3.11/site-packages/] -> /opt/lib/venv/lib/python3.11/site-packages/                    27.1s
 => [revertrisk-multilingual-cpu production] 📂 [model_server_entrypoint.sh] -> entrypoint.sh                                                                           0.9s
 => [revertrisk-multilingual-cpu production] 📂 [common_settings.sh] -> common_settings.sh                                                                              1.0s
 => [revertrisk-multilingual-cpu] exporting to image                                                                                                                  10.8s
 => => exporting layers                                                                                                                                               10.4s
 => => writing image sha256:88e7871403047cbfc7367641f5cadadc264e0b39a72a89065ce500c1a215fb96                                                                           0.0s
 => => naming to docker.io/library/rrml:cpu                                                                                                                            0.1s
 => [revertrisk-multilingual-cpu] resolving provenance for metadata file                                                                                               0.1s
[+] Building 1/1
 ✔ revertrisk-multilingual-cpu  Built                                                                                                                                  0.0s
bart@bart:~/projs/inference-services$ PATH_TO_REVERTRISK_MULTILINGUAL_MODEL="./models/revertrisk/multilingual/20230810110019" docker compose up revertrisk-multilingual-cpu
[+] Running 2/2
 ✔ Network inference-services_default                          Created                                                                                                 0.0s
 ✔ Container inference-services-revertrisk-multilingual-cpu-1  Created                                                                                                 0.1s
Attaching to revertrisk-multilingual-cpu-1
revertrisk-multilingual-cpu-1  | + source common_settings.sh
revertrisk-multilingual-cpu-1  | +++ /usr/bin/python3 -c 'from python.resource_utils import get_cpu_count; print(get_cpu_count())'
revertrisk-multilingual-cpu-1  | ++ CPU_COUNT=8
revertrisk-multilingual-cpu-1  | ++ echo 'CPU count detected from get_cpu_count: 8'
revertrisk-multilingual-cpu-1  | CPU count detected from get_cpu_count: 8
revertrisk-multilingual-cpu-1  | OMP_NUM_THREADS set to: 8
revertrisk-multilingual-cpu-1  | ++ export OMP_NUM_THREADS=8
revertrisk-multilingual-cpu-1  | ++ OMP_NUM_THREADS=8
revertrisk-multilingual-cpu-1  | ++ echo 'OMP_NUM_THREADS set to: 8'
revertrisk-multilingual-cpu-1  | + MODEL_SERVER_PATH=model_server/model.py
revertrisk-multilingual-cpu-1  | + exec /usr/bin/python3 model_server/model.py
revertrisk-multilingual-cpu-1  | INFO:root:Successfully loaded 342 canonical wiki languages.
revertrisk-multilingual-cpu-1  | 2025-10-24 09:31:59.635 1 kserve INFO [model_server.py:register_model():402] Registering model: revertrisk-multilingual
revertrisk-multilingual-cpu-1  | 2025-10-24 09:31:59.635 1 kserve INFO [model_server.py:setup_event_loop():282] Setting max asyncio worker threads as 12
revertrisk-multilingual-cpu-1  | 2025-10-24 09:31:59.743 1 kserve INFO [server.py:_register_endpoints():110] OpenAI endpoints not registered
revertrisk-multilingual-cpu-1  | 2025-10-24 09:31:59.744 1 kserve INFO [server.py:start():161] Starting uvicorn with 1 workers
revertrisk-multilingual-cpu-1  | 2025-10-24 09:31:59.764 1 uvicorn.error INFO:     Started server process [1]
revertrisk-multilingual-cpu-1  | 2025-10-24 09:31:59.764 1 uvicorn.error INFO:     Waiting for application startup.
revertrisk-multilingual-cpu-1  | 2025-10-24 09:31:59.785 1 kserve INFO [server.py:start():70] Starting gRPC server with 4 workers
revertrisk-multilingual-cpu-1  | 2025-10-24 09:31:59.785 1 kserve INFO [server.py:start():71] Starting gRPC server on [::]:8081
revertrisk-multilingual-cpu-1  | 2025-10-24 09:31:59.785 1 uvicorn.error INFO:     Application startup complete.
revertrisk-multilingual-cpu-1  | 2025-10-24 09:31:59.786 1 uvicorn.error INFO:     Uvicorn running on http://0.0.0.0:8080 (Press CTRL+C to quit)

Sorry for the slow turn response; I followed the updated readme in the attached patch and I'm able to stand up the model server 🎉; I was able to point it to my local dev mediawiki install with a little bit of adjustment to the port mapping:
from my docker-compose.override,yml (note that in this case I have repo sitting beside my mediawiki repo)

inference:
  extends:
    file: ../inference-services/docker-compose.yml
    service: revertrisk-language-agnostic
  env_file:
    - '.env'
  ports: !override
    - "8000:8080"

and this in my mediawiki .env

PATH_TO_REVERTRISK_LANGUAGE_AGNOSTIC_MODEL=../inference-services/models
WIKI_URL=http://mediawiki-web:8080

Now I can work on my local mediawiki code that wants rev scores and actually get real scores for my local content. Helpful!

BWojtowicz-WMF closed this task as Resolved.Feb 19 2026, 5:04 PM

BWojtowicz-WMF moved this task from In Progress to 2025-2026 Q2 Done on the Machine-Learning-Team board.Feb 25 2026, 11:51 AM

Revertrisk multilingual fails locally when ran with docker composeClosed, ResolvedPublicBUG REPORTActions

Description

Details

Event Timeline

Revertrisk multilingual fails locally when ran with docker compose
Closed, ResolvedPublicBUG REPORT
Actions