Use the latest bookworm image and update reability server.
Description
Description
Details
Details
Related Changes in Gerrit:
Customize query in gerrit
Related Changes in GitLab:
| Title | Reference | Author | Source Branch | Dest Branch | |
|---|---|---|---|---|---|
| Update python to 3.11 | repos/machine-learning/readability-liftwing!1 | isaranto | use-ml-fork | main |
| Status | Subtype | Assigned | Task | ||
|---|---|---|---|---|---|
| Open | None | T398948 Q1 FY2025-26 Goal: Operational Excellence - LiftWing Platform Updates & Improvements | |||
| Resolved | isarantopoulos | T400144 Upgrade remaining model servers from debian bullseye to bookworm | |||
| Resolved | achou | T400352 Upgrade readability model server from debian bullseye to bookworm |
Event Timeline
Comment Actions
Change #1174648 had a related patch set uploaded (by Bartosz Wójtowicz; author: Bartosz Wójtowicz):
[machinelearning/liftwing/inference-services@main] readability: Update base image to latest bookworm.
Comment Actions
As discussed with @BWojtowicz-WMF , old catboost version does not work after the upgrades.
Therefore, we will need to train a model with a newer versions of catboost first.
@BWojtowicz-WMF ,
Should we move this task to blocked until we have a new model?
Comment Actions
@OKarakaya-WMF
I agree, let's put this in blocked until we update the catboost version in the upstream repository.
Comment Actions
isaranto updated https://gitlab.wikimedia.org/repos/machine-learning/readability-liftwing/-/merge_requests/1
Update python to 3.11
Comment Actions
In readability we are using a package that is installed directly from another repository (readability-liftwing) which does the same for a package defined in readability-experiments repo.
I've created forks of these repos under repos/machine-learning on gitlab so that we can have full ownership over the deployed service.
https://gitlab.wikimedia.org/repos/machine-learning/readability-liftwing
https://gitlab.wikimedia.org/repos/machine-learning/readability-experiments
To upgrade the service we'd have to do the following
- merge the MR in readability-experiments which updates python version to 3.11
- after the merge of ^^ update a similar MR in readability-liftwing
- update the patch to use the new fork https://gitlab.wikimedia.org/repos/machine-learning/readability-liftwing
Comment Actions
aikochou merged https://gitlab.wikimedia.org/repos/machine-learning/readability-liftwing/-/merge_requests/1
Update python to 3.11
Comment Actions
Change #1174648 merged by jenkins-bot:
[machinelearning/liftwing/inference-services@main] readability: Update base image to latest bookworm.
Comment Actions
Change #1180082 had a related patch set uploaded (by AikoChou; author: AikoChou):
[operations/deployment-charts@master] ml-services: update image for readability model on staging
Comment Actions
Change #1180082 merged by jenkins-bot:
[operations/deployment-charts@master] ml-services: update image for readability model on staging
Comment Actions
Change #1180098 had a related patch set uploaded (by AikoChou; author: AikoChou):
[operations/deployment-charts@master] ml-services: update image for readability model on prod
Comment Actions
Change #1181109 had a related patch set uploaded (by AikoChou; author: AikoChou):
[machinelearning/liftwing/inference-services@main] locust: update test result for readability model
Comment Actions
Here are the load test results:
- Staging (bookworm)
Load test results are within the threshold
[2025-08-22 10:50:13,177] stat1008/INFO/locust.main: Shutting down (exit code 0)
Type Name # reqs # fails | Avg Min Max Med | req/s failures/s
--------|----------------------------------------------------------------------------|-------|-------------|-------|-------|-------|-------|--------|-----------
POST /v1/models/readability:predict 20 0(0.00%) | 8623 6101 11790 8200 | 0.17 0.00
--------|----------------------------------------------------------------------------|-------|-------------|-------|-------|-------|-------|--------|-----------
Aggregated 20 0(0.00%) | 8623 6101 11790 8200 | 0.17 0.00
Response time percentiles (approximated)
Type Name 50% 66% 75% 80% 90% 95% 98% 99% 99.9% 99.99% 100% # reqs
--------|--------------------------------------------------------------------------------|--------|------|------|------|------|------|------|------|------|------|------|------
POST /v1/models/readability:predict 8400 9500 10000 10000 10000 12000 12000 12000 12000 12000 12000 20
--------|--------------------------------------------------------------------------------|--------|------|------|------|------|------|------|------|------|------|------|------
Aggregated 8400 9500 10000 10000 10000 12000 12000 12000 12000 12000 12000 20Comment Actions
Change #1180098 merged by jenkins-bot:
[operations/deployment-charts@master] ml-services: update image for readability model on prod
Comment Actions
The new readability model-server that supports bookworm has been deployed in prod. Pods in eqiad and codfw are up and running:
kevinbazira@deploy1003:~$ kube_env readability ml-serve-eqiad kevinbazira@deploy1003:~$ kubectl get pods NAME READY STATUS RESTARTS AGE readability-predictor-default-00010-deployment-64d668b54-zfj2j 3/3 Running 0 2m10s kevinbazira@deploy1003:~$ kube_env readability ml-serve-codfw kevinbazira@deploy1003:~$ kubectl get pods NAME READY STATUS RESTARTS AGE readability-predictor-default-00011-deployment-85466dd464-xzjkh 3/3 Running 0 3m17s
Comment Actions
Change #1181109 merged by jenkins-bot:
[machinelearning/liftwing/inference-services@main] locust: update test result for readability model