Since the purpose of this work is to update any bullseye images that are used in production since bullseye is reaching EOL soon. I would suggest to focus on on the revertisk language agnostic model for the following reasons:

• the wikidata one is not used and is not production ready
• the multilingual one is already using a bookworm based image (the pytorch one)

Testing revertrisk on bookworm image

I used the following process for testing the image:

ssh ml-testing.machine-learning.eqiad1.wikimedia.cloud

I created a Dockerfile using the bookworm base image.

echo "FROM docker-registry.wikimedia.org/bookworm:20250720 
RUN apt-get update && apt-get install -y build-essential python3-pip python3-dev python3-setuptools python3-venv git g++ wget vim curl
" > Dockerfile

Build and run image

docker build -t revrisk:bookworm .
docker run -it --entrypoint=/bin/bash revrisk:bookworm

Clone inference services inside the container

git clone inference-services
cd inference-services/

Run make command and start the model server

make revertrisk-language-agnostic

## Console Output
INFO:root:Successfully loaded 342 canonical wiki languages.
2025-07-23 13:25:48.362 101 kserve INFO [model_server.py:register_model():402] Registering model: revertrisk-language-agnostic
2025-07-23 13:25:48.363 101 kserve INFO [model_server.py:setup_event_loop():282] Setting max asyncio worker threads as 12
2025-07-23 13:25:48.416 101 kserve INFO [server.py:_register_endpoints():110] OpenAI endpoints not registered
2025-07-23 13:25:48.417 101 kserve INFO [server.py:start():161] Starting uvicorn with 1 workers
2025-07-23 13:25:48.626 101 uvicorn.error INFO:     Started server process [101]
2025-07-23 13:25:48.627 101 uvicorn.error INFO:     Waiting for application startup.
2025-07-23 13:25:48.636 101 kserve INFO [server.py:start():70] Starting gRPC server with 4 workers
2025-07-23 13:25:48.636 101 kserve INFO [server.py:start():71] Starting gRPC server on [::]:8081
2025-07-23 13:25:48.637 101 uvicorn.error INFO:     Application startup complete.
2025-07-23 13:25:48.638 101 uvicorn.error INFO:     Uvicorn running on http://0.0.0.0:8080

In another terminal

ssh ml-testing.machine-learning.eqiad1.wikimedia.cloud

Execute the same docker container which runs the model server on localhost

docker exec -it <revrisk:bookworm container id> /bin/bash

Hit the endpoint on localhost

root@77e528d41015:~# curl localhost:8080/v1/models/revertrisk-language-agnostic:predict -i -X POST -d '{"lang": "en", "rev_id": 12345}'

HTTP/1.1 200 OK
date: Wed, 23 Jul 2025 13:16:18 GMT
server: uvicorn
content-length: 206
content-type: application/json

{"model_name":"revertrisk-language-agnostic","model_version":"3","wiki_db":"enwiki","revision_id":12345,"output":{"prediction":false,"probabilities":{"true":0.17687281966209412,"false":0.8231271803379059}}}

Change #1172297 had a related patch set uploaded (by Gkyziridis; author: Gkyziridis):

[machinelearning/liftwing/inference-services@main] revertrisk-model: Update base image from bullseye to the latest bookworm image.

https://gerrit.wikimedia.org/r/1172297

Update Blubber Syntax

I tried to update the blubber syntax to the latest version of buildkit: buildkit:v1.3.0.
After various changes in the .pipeline/revertrisk/revertrisk.yaml file according to the new schema based on the latest version, I am having the following error on CI which I cannot overcome:

#35 [test]  📦 {build}[/opt/lib/python/site-packages] -> /opt/lib/python/site-packages
#35 sha256:1b559f343cf4c059216b2507cf14e6e720ccb6a4f17db956932d3bcd733a17d2
#35 ERROR: failed to walk /var/lib/docker/tmp/buildkit-mount955142797/opt/lib/python: lstat /var/lib/docker/tmp/buildkit-mount955142797/opt/lib/python: no such file or directory
------
 > [test]  📦 {build}[/opt/lib/python/site-packages] -> /opt/lib/python/site-packages:
------
failed to compute cache key: failed to walk /var/lib/docker/tmp/buildkit-mount955142797/opt/lib/python: lstat /var/lib/docker/tmp/buildkit-mount955142797/opt/lib/python: no such file or directory

I think it would be better to tackle the update of the buildkit syntax in a dedicated task in order to avoid adding extra scope to the current one.
I will open a new task for updating the blubber syntax.

@gkyziridis o/ I think that the problem may be due to setting use-system-site-packages to false, since we explicitly copy from /opt/lib/python/site-packages in the production variant (from the build one).

In T400266#11033793, @elukey wrote:

@gkyziridis o/ I think that the problem may be due to setting use-system-site-packages to false, since we explicitly copy from /opt/lib/python/site-packages in the production variant (from the build one).

Thank you for your fast response, much appreciated.
I used the use-system-site-packages: false because the current blubber had the: use-system-flag: false using the old schema.
I opened a new task for tackling that because we want to proceed with the bookworm updates in the rest of the models as well: https://phabricator.wikimedia.org/T400446

Change #1172297 had a related patch set uploaded (by Gkyziridis; author: Gkyziridis):

[machinelearning/liftwing/inference-services@main] revertrisk-model: Update base image from bullseye to the latest bookworm image.

https://gerrit.wikimedia.org/r/1172297

Change #1172297 merged by jenkins-bot:

[machinelearning/liftwing/inference-services@main] revertrisk-model: Update base image from bullseye to the latest bookworm image.

https://gerrit.wikimedia.org/r/1172297

Change #1172622 had a related patch set uploaded (by Gkyziridis; author: Gkyziridis):

[operations/deployment-charts@master] ml-services: Deploy revertrisk-language-agnostic latest published image

https://gerrit.wikimedia.org/r/1172622

In T400266#11033886, @gkyziridis wrote:

In T400266#11033793, @elukey wrote:

@gkyziridis o/ I think that the problem may be due to setting use-system-site-packages to false, since we explicitly copy from /opt/lib/python/site-packages in the production variant (from the build one).

Thank you for your fast response, much appreciated.
I used the use-system-site-packages: false because the current blubber had the: use-system-flag: false using the old schema.
I opened a new task for tackling that because we want to proceed with the bookworm updates in the rest of the models as well: https://phabricator.wikimedia.org/T400446

Yes yes it makes sense! In Bookworm the Python convention changed, namely the "system" path is by default reserved for Python Debian package, and pip-related installs should go in a venv. I am not 100% sure what is the guideline from Releng, but it may be ok to just create a venv in the blubber file and pip install packages on it (and eventually copy that from the build variant to the prod one).

In T400266#11034580, @elukey wrote:

In T400266#11033886, @gkyziridis wrote:

In T400266#11033793, @elukey wrote:

@gkyziridis o/ I think that the problem may be due to setting use-system-site-packages to false, since we explicitly copy from /opt/lib/python/site-packages in the production variant (from the build one).

Thank you for your fast response, much appreciated.
I used the use-system-site-packages: false because the current blubber had the: use-system-flag: false using the old schema.
I opened a new task for tackling that because we want to proceed with the bookworm updates in the rest of the models as well: https://phabricator.wikimedia.org/T400446

Yes yes it makes sense! In Bookworm the Python convention changed, namely the "system" path is by default reserved for Python Debian package, and pip-related installs should go in a venv. I am not 100% sure what is the guideline from Releng, but it may be ok to just create a venv in the blubber file and pip install packages on it (and eventually copy that from the build variant to the prod one).

I copied this comment to the dedicated ticket: https://phabricator.wikimedia.org/T400446

Change #1172622 merged by jenkins-bot:

[operations/deployment-charts@master] ml-services: Deploy revertrisk-language-agnostic latest published image

https://gerrit.wikimedia.org/r/1172622

Change #1175888 had a related patch set uploaded (by Gkyziridis; author: Gkyziridis):

[operations/deployment-charts@master] ml-services: Deploy revertrisk-language-agnostic latest published image on prod.

https://gerrit.wikimedia.org/r/1175888

Change #1175888 merged by jenkins-bot:

[operations/deployment-charts@master] ml-services: Deploy revertrisk-language-agnostic latest published image on prod.

https://gerrit.wikimedia.org/r/1175888

Change #1176199 had a related patch set uploaded (by Gkyziridis; author: Gkyziridis):

[operations/deployment-charts@master] ml-services: Deploy revertrisk-language-agnostic latest published image on prod.

https://gerrit.wikimedia.org/r/1176199

Change #1176199 merged by jenkins-bot:

[operations/deployment-charts@master] ml-services: Deploy revertrisk-language-agnostic latest published image on prod.

https://gerrit.wikimedia.org/r/1176199