Use the latest bookworm image and update revscoring server.
Description
Description
Details
Details
Related Changes in Gerrit:
Customize query in gerrit
| Status | Subtype | Assigned | Task | ||
|---|---|---|---|---|---|
| Open | None | T398948 Q1 FY2025-26 Goal: Operational Excellence - LiftWing Platform Updates & Improvements | |||
| Resolved | isarantopoulos | T400144 Upgrade remaining model servers from debian bullseye to bookworm | |||
| Resolved | gkyziridis | T400266 Upgrade revertrisk model server from the debian bullseye base image to bookworm. | |||
| Resolved | gkyziridis | T400347 Upgrade langid model server from debian bullseye to bookworm | |||
| Resolved | gkyziridis | T400348 Upgrade ores-legacy from debian bullseye to bookworm | |||
| Resolved | gkyziridis | T400349 Upgrade articletopic-outlink model servers from debian bullseye to bookworm | |||
| Resolved | kevinbazira | T400350 Upgrade revscoring model servers from debian bullseye to bookworm | |||
| Resolved | BWojtowicz-WMF | T400351 Upgrade article-descriptions model servers from debian bullseye to bookworm | |||
| Resolved | achou | T400352 Upgrade readability model server from debian bullseye to bookworm |
Event Timeline
Comment Actions
I've upgraded revscoring and I'm getting following error.
I think we will need to re-train our models first with a newer version of sklearn
I'll share another post/branch explaining how I reached to this step :)
model = RevscoringModel(inference_name, model_type)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
return Model.load(f)
^^^^^^^^^^^^^
model = pickle.load(f.buffer)
^^^^^^^^^^^^^^^^^^^^^
ModuleNotFoundError: No module named 'sklearn.ensemble._gb_losses'Comment Actions
I've upgraded revscoring to bookworm and tried with revscoring-damaging-enwiki.
I get errors while pickling the model as I had to upgrade the version of sklearn.
Therefore, I think we need to retrain the model with the same version of sklearn first.
I'll check if we can find a version that works with both as a last check.
(myenv_revscoring) ozge@wmf3658 inference-services % docker compose up damaging
WARN[0000] The "DYLD_FALLBACK_LIBRARY_PATH" variable is not set. Defaulting to a blank string.
WARN[0000] The "DYLD_FALLBACK_LIBRARY_PATH" variable is not set. Defaulting to a blank string.
[+] Running 1/1
✔ Container inference-services-damaging-1 Recreated 0.1s
Attaching to damaging-1
damaging-1 | /home/somebody/.local/lib/python3.11/site-packages/requests/__init__.py:102: RequestsDependencyWarning: urllib3 (1.26.8) or chardet (5.2.0)/charset_normalizer (2.0.9) doesn't match a supported version!
damaging-1 | warnings.warn("urllib3 ({}) or chardet ({})/charset_normalizer ({}) doesn't match a supported "
damaging-1 | /home/somebody/.local/lib/python3.11/site-packages/revscoring/__init__.py:140: UserWarning: pkg_resources is deprecated as an API. See https://setuptools.pypa.io/en/latest/pkg_resources.html. The pkg_resources package is slated for removal as early as 2025-11-30. Refrain from using this package or pin to Setuptools<81.
damaging-1 | from pkg_resources import VersionConflict
damaging-1 | /home/somebody/.local/lib/python3.11/site-packages/sklearn/base.py:442: InconsistentVersionWarning: Trying to unpickle estimator RobustScaler from version 0.22.1 when using version 1.7.1. This might lead to breaking code or invalid results. Use at your own risk. For more info please refer to:
damaging-1 | https://scikit-learn.org/stable/model_persistence.html#security-maintainability-limitations
damaging-1 | warnings.warn(
damaging-1 | Traceback (most recent call last):
damaging-1 | File "/srv/rev/revscoring_model/model.py", line 27, in <module>
damaging-1 | model = RevscoringModel(inference_name, model_type)
damaging-1 | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
damaging-1 | File "/srv/rev/revscoring_model/model_servers/model_servers.py", line 76, in __init__
damaging-1 | self.model = self.load()
damaging-1 | ^^^^^^^^^^^
damaging-1 | File "/srv/rev/revscoring_model/model_servers/model_servers.py", line 130, in load
damaging-1 | return Model.load(f)
damaging-1 | ^^^^^^^^^^^^^
damaging-1 | File "/home/somebody/.local/lib/python3.11/site-packages/revscoring/scoring/models/model.py", line 102, in load
damaging-1 | model = pickle.load(f.buffer)
damaging-1 | ^^^^^^^^^^^^^^^^^^^^^
damaging-1 | ModuleNotFoundError: No module named 'sklearn.ensemble._gb_losses'Comment Actions
I could build the project minimum with scikit-learn==1.2.0
However, this version raises following error in inference time.
raw_predictions = self._loss.get_init_raw_predictions(X, self.init_).astype(
^^^^^^^^^^
AttributeError: 'GradientBoostingClassifier' object has no attribute '_loss'Comment Actions
I discussed this solution with @isarantopoulos, and we found that it might not be sustainable in the long term since python3.9 is reaching EOL in October 2025. Going to investigate alternative approaches before we settle on a final solution.
I have worked on the blubberfile shown below that updates the revscoring model-server's OS from bullseye to bookworm. Bookworm defaults to python3.11, which introduces incompatibilities in revscoring dependencies and leads to the dependency issues that Özge reported above. The solution that currently works is to compile and use python3.9 within the image, since this version is fully supported by revscoring.
I discussed this solution with @isarantopoulos, and we found that it might not be sustainable in the long term since python3.9 is reaching EOL in October 2025. Going to investigate alternative approaches before we settle on a final solution.
Comment Actions
This approach still resolves the incompatibilities of revscoring with bookworm's default python3.11, while extending the EOL to October 2026. This gives us over a year to phase out revscoring models.
Following the EOL concerns with python3.9 discussed in T400350#11120693, I have worked on the updated revscoring model-server blubberfile shown below. It builds and installs python3.10 within bookworm.
This approach still resolves the incompatibilities of revscoring with bookworm's default python3.11, while extending the EOL to October 2026. This gives us over a year to phase out revscoring models.
Comment Actions
Change #1182495 had a related patch set uploaded (by Kevin Bazira; author: Kevin Bazira):
[machinelearning/liftwing/inference-services@main] revscoring: upgrade model-server from bullseye to bookworm
Comment Actions
Change #1182495 merged by jenkins-bot:
[machinelearning/liftwing/inference-services@main] revscoring: upgrade model-server from bullseye to bookworm
Comment Actions
Change #1182506 had a related patch set uploaded (by Kevin Bazira; author: Kevin Bazira):
[operations/deployment-charts@master] ml-services: update revscoring staging image
Comment Actions
Change #1182506 merged by jenkins-bot:
[operations/deployment-charts@master] ml-services: update revscoring staging image
Comment Actions
articlequality isvc running in staging:
kevinbazira@deploy1003:~$ kube_env revscoring-articlequality ml-staging-codfw
kevinbazira@deploy1003:~$ kubectl get pods
NAME READY STATUS RESTARTS AGE
enwiki-articlequality-predictor-default-00031-deployment-8nb5lc 3/3 Running 0 2m17s
wikidatawiki-itemquality-predictor-default-00025-deploymenm88vm 3/3 Running 0 2m16s
kevinbazira@deploy1003:~$
kevinbazira@deploy1003:~$
kevinbazira@deploy1003:~$ curl "https://inference-staging.svc.codfw.wmnet:30443/v1/models/enwiki-articlequality:predict" -X POST -d '{"rev_id": 12345}' -H "Host: enwiki-articlequality.revscoring-articlequality.wikimedia.org" -H "Content-Type: application/json" --http1.1
{"enwiki":{"models":{"articlequality":{"version":"0.9.2"}},"scores":{"12345":{"articlequality":{"score":{"prediction":"Stub","probability":{"B":0.04109666150848109,"C":0.02060738177009099,"FA":0.0029400688910391592,"GA":0.004970401857774162,"Start":0.17362493306327959,"Stub":0.7567605529093352}}}}}}}Comment Actions
articletopic isvc running in staging:
kevinbazira@deploy1003:~$ kube_env revscoring-articletopic ml-staging-codfw
kevinbazira@deploy1003:~$ kubectl get pods
NAME READY STATUS RESTARTS AGE
enwiki-articletopic-predictor-default-00024-deployment-6fc9fwfn 3/3 Running 0 57s
kevinbazira@deploy1003:~$
kevinbazira@deploy1003:~$
kevinbazira@deploy1003:~$ curl "https://inference-staging.svc.codfw.wmnet:30443/v1/models/enwiki-articletopic:predict" -X POST -d '{"rev_id": 12345}' -H "Host: enwiki-articletopic.revscoring-articletopic.wikimedia.org" -H "Content-Type: application/json" --http1.1
{"enwiki":{"models":{"articletopic":{"version":"1.3.0"}},"scores":{"12345":{"articletopic":{"score":{"prediction":["STEM.STEM*"],"probability":{"Culture.Biography.Biography*":0.0037221493203970753,"Culture.Biography.Women":0.0016274082204131065,"Culture.Food and drink":0.003869384114515742,"Culture.Internet culture":0.0027448342044452587,"Culture.Linguistics":0.0004704196841241876,"Culture.Literature":0.00290361288753546,"Culture.Media.Books":0.000742678990345212,"Culture.Media.Entertainment":0.0989755577969651,"Culture.Media.Films":0.0031771755584005376,"Culture.Media.Media*":0.21201751150971165,"Culture.Media.Music":0.0009582260980479466,"Culture.Media.Radio":6.200606120337714e-05,"Culture.Media.Software":0.006069833295704291,"Culture.Media.Television":0.0015207772089172447,"Culture.Media.Video games":0.0005615093206567188,"Culture.Performing arts":0.0016136571753909768,"Culture.Philosophy and religion":0.0051620014754150895,"Culture.Sports":0.006003766168026103,"Culture.Visual arts.Architecture":0.02932200170357891,"Culture.Visual arts.Comics and Anime":0.000367749742655391,"Culture.Visual arts.Fashion":0.0073053897003234405,"Culture.Visual arts.Visual arts*":0.045704381965926126,"Geography.Geographical":0.0035315512289540206,"Geography.Regions.Africa.Africa*":0.00483994316149668,"Geography.Regions.Africa.Central Africa":0.0004058831032942602,"Geography.Regions.Africa.Eastern Africa":8.202888604849657e-05,"Geography.Regions.Africa.Northern Africa":0.00014172395987364393,"Geography.Regions.Africa.Southern Africa":0.0008028449272043562,"Geography.Regions.Africa.Western Africa":0.0002728161473653015,"Geography.Regions.Americas.Central America":0.00023176185775056676,"Geography.Regions.Americas.North America":0.0031703002857622485,"Geography.Regions.Americas.South America":0.000188786651678672,"Geography.Regions.Asia.Asia*":0.005622487337808649,"Geography.Regions.Asia.Central Asia":0.00035000790929257494,"Geography.Regions.Asia.East Asia":0.0033450611384150454,"Geography.Regions.Asia.North Asia":0.0004353985298481421,"Geography.Regions.Asia.South Asia":0.00020663669814920222,"Geography.Regions.Asia.Southeast Asia":0.00032506406741737574,"Geography.Regions.Asia.West Asia":0.0010116859690400576,"Geography.Regions.Europe.Eastern Europe":0.0032436576651452653,"Geography.Regions.Europe.Europe*":0.004678418783680385,"Geography.Regions.Europe.Northern Europe":0.0003478676128337494,"Geography.Regions.Europe.Southern Europe":0.0023790437464152685,"Geography.Regions.Europe.Western Europe":0.001866188163881686,"Geography.Regions.Oceania":0.020499546152686617,"History and Society.Business and economics":0.006809005223678666,"History and Society.Education":0.0004626665847193515,"History and Society.History":0.003483860446308053,"History and Society.Military and warfare":0.0007483799946402395,"History and Society.Politics and government":0.04064941446684307,"History and Society.Society":0.003989646630130237,"History and Society.Transportation":0.0021103679224344224,"STEM.Biology":0.0063757590084982845,"STEM.Chemistry":0.002592859482757056,"STEM.Computing":0.025968220954150065,"STEM.Earth and environment":0.004485463793376359,"STEM.Engineering":0.0007893097092528008,"STEM.Libraries & Information":0.0005542340017675382,"STEM.Mathematics":0.39109499414712584,"STEM.Medicine & Health":0.08689195662186826,"STEM.Physics":0.004884195584942447,"STEM.STEM*":0.9049354239597514,"STEM.Space":0.0002139344401835148,"STEM.Technology":0.07492699172495908}}}}}}}Comment Actions
draftquality isvc running in staging:
kevinbazira@deploy1003:~$ kube_env revscoring-draftquality ml-staging-codfw
kevinbazira@deploy1003:~$ kubectl get pods
NAME READY STATUS RESTARTS AGE
enwiki-draftquality-predictor-default-00023-deployment-bb5d8x2j 3/3 Running 0 57s
kevinbazira@deploy1003:~$
kevinbazira@deploy1003:~$
kevinbazira@deploy1003:~$ curl "https://inference-staging.svc.codfw.wmnet:30443/v1/models/enwiki-draftquality:predict" -X POST -d '{"rev_id": 12345}' -H "Host: enwiki-draftquality.revscoring-draftquality.wikimedia.org" -H "Content-Type: application/json" --http1.1
{"enwiki":{"models":{"draftquality":{"version":"0.2.1"}},"scores":{"12345":{"draftquality":{"score":{"prediction":"spam","probability":{"OK":0.20587071389614808,"attack":0.08356921917837871,"spam":0.39768296325893615,"vandalism":0.31287710366653704}}}}}}}Comment Actions
drafttopic isvc running in staging:
kevinbazira@deploy1003:~$ kube_env revscoring-drafttopic ml-staging-codfw
kevinbazira@deploy1003:~$ kubectl get pods
NAME READY STATUS RESTARTS AGE
enwiki-drafttopic-predictor-default-00026-deployment-775bf7sxxj 3/3 Running 0 68s
kevinbazira@deploy1003:~$
kevinbazira@deploy1003:~$
kevinbazira@deploy1003:~$ curl "https://inference-staging.svc.codfw.wmnet:30443/v1/models/enwiki-drafttopic:predict" -X POST -d '{"rev_id": 12345}' -H "Host: enwiki-drafttopic.revscoring-drafttopic.wikimedia.org" -H "Content-Type: application/json" --http1.1
{"enwiki":{"models":{"drafttopic":{"version":"1.3.0"}},"scores":{"12345":{"drafttopic":{"score":{"prediction":["STEM.STEM*"],"probability":{"Culture.Biography.Biography*":0.022902630539395685,"Culture.Biography.Women":0.005067486633806841,"Culture.Food and drink":0.011552909215159848,"Culture.Internet culture":0.0018836356548609864,"Culture.Linguistics":0.000987244438369833,"Culture.Literature":0.0050423059095638205,"Culture.Media.Books":0.0008228825281107042,"Culture.Media.Entertainment":0.027835523391703726,"Culture.Media.Films":0.0021233422398979594,"Culture.Media.Media*":0.10369183346752998,"Culture.Media.Music":0.005345021766065545,"Culture.Media.Radio":4.71813720695922e-05,"Culture.Media.Software":0.010991013112889197,"Culture.Media.Television":0.003745806852443267,"Culture.Media.Video games":0.0012726195754468805,"Culture.Performing arts":0.0011692106165364648,"Culture.Philosophy and religion":0.012211366011815949,"Culture.Sports":0.008617259930690756,"Culture.Visual arts.Architecture":0.015175202598187623,"Culture.Visual arts.Comics and Anime":0.0004580529412803368,"Culture.Visual arts.Fashion":0.0032463316804348247,"Culture.Visual arts.Visual arts*":0.04966240731500535,"Geography.Geographical":0.006501819685165024,"Geography.Regions.Africa.Africa*":0.004278923507426861,"Geography.Regions.Africa.Central Africa":0.0003367596839992931,"Geography.Regions.Africa.Eastern Africa":3.09235115912245e-05,"Geography.Regions.Africa.Northern Africa":0.0001802204967553657,"Geography.Regions.Africa.Southern Africa":0.0018079996950138254,"Geography.Regions.Africa.Western Africa":0.0007251088049127753,"Geography.Regions.Americas.Central America":0.0004991928760763337,"Geography.Regions.Americas.North America":0.006380070962953918,"Geography.Regions.Americas.South America":0.0002306844502170147,"Geography.Regions.Asia.Asia*":0.017313494735674808,"Geography.Regions.Asia.Central Asia":0.0008072328532154094,"Geography.Regions.Asia.East Asia":0.0023203835158971666,"Geography.Regions.Asia.North Asia":0.00045870078954648444,"Geography.Regions.Asia.South Asia":0.0009922199817278582,"Geography.Regions.Asia.Southeast Asia":0.0002738236138644238,"Geography.Regions.Asia.West Asia":0.004635726599288032,"Geography.Regions.Europe.Eastern Europe":0.001716235984380361,"Geography.Regions.Europe.Europe*":0.0077942173009087345,"Geography.Regions.Europe.Northern Europe":0.0007906624707708472,"Geography.Regions.Europe.Southern Europe":0.0013950233741366756,"Geography.Regions.Europe.Western Europe":0.0020105862970372567,"Geography.Regions.Oceania":0.005480678538779058,"History and Society.Business and economics":0.010849117305548793,"History and Society.Education":0.0006190153665937547,"History and Society.History":0.0075245897065247735,"History and Society.Military and warfare":0.0016535860240798408,"History and Society.Politics and government":0.04792153752537525,"History and Society.Society":0.006233592753280993,"History and Society.Transportation":0.0006034027160506236,"STEM.Biology":0.008912249418159286,"STEM.Chemistry":0.006101030851625054,"STEM.Computing":0.09725027056674931,"STEM.Earth and environment":0.01375490559994272,"STEM.Engineering":0.0007519463706161267,"STEM.Libraries & Information":0.0026560386120175325,"STEM.Mathematics":0.037996709648705204,"STEM.Medicine & Health":0.12580483337522344,"STEM.Physics":0.004410180324962277,"STEM.STEM*":0.9465090911647787,"STEM.Space":0.003443873426925551,"STEM.Technology":0.06698348905685471}}}}}}}Comment Actions
damaging isvc running in staging:
kevinbazira@deploy1003:~$ kube_env revscoring-editquality-damaging ml-staging-codfw
kevinbazira@deploy1003:~$ kubectl get pods
NAME READY STATUS RESTARTS AGE
enwiki-damaging-predictor-00007-deployment-558fd4bb66-jv75q 3/3 Running 0 63s
wikidatawiki-damaging-predictor-00006-deployment-6bb645f7-ff2kf 3/3 Running 0 61s
kevinbazira@deploy1003:~$
kevinbazira@deploy1003:~$
kevinbazira@deploy1003:~$
kevinbazira@deploy1003:~$ curl "https://inference-staging.svc.codfw.wmnet:30443/v1/models/enwiki-damaging:predict" -X POST -d '{"rev_id": 12345}' -H "Host: enwiki-damaging.revscoring-editquality-damaging.wikimedia.org" -H "Content-Type: application/json" --http1.1
{"enwiki":{"models":{"damaging":{"version":"0.5.1"}},"scores":{"12345":{"damaging":{"score":{"prediction":false,"probability":{"false":0.744276020191829,"true":0.255723979808171}}}}}}}Comment Actions
goodfaith isvc running in staging:
kevinbazira@deploy1003:~$ kube_env revscoring-editquality-goodfaith ml-staging-codfw
kevinbazira@deploy1003:~$ kubectl get pods
NAME READY STATUS RESTARTS AGE
zhwiki-goodfaith-predictor-default-00029-deployment-79c9fdvdnl4 3/3 Running 0 70s
kevinbazira@deploy1003:~$
kevinbazira@deploy1003:~$
kevinbazira@deploy1003:~$ curl "https://inference-staging.svc.codfw.wmnet:30443/v1/models/zhwiki-goodfaith:predict" -X POST -d '{"rev_id": 12345}' -H "Host: zhwiki-goodfaith.revscoring-editquality-goodfaith.wikimedia.org" -H "Content-Type: application/json" --http1.1
{"zhwiki":{"models":{"goodfaith":{"version":"0.5.0"}},"scores":{"12345":{"goodfaith":{"score":{"prediction":true,"probability":{"false":0.0113558341209401,"true":0.9886441658790599}}}}}}}Comment Actions
reverted isvc running in staging:
kevinbazira@deploy1003:~$ kube_env revscoring-editquality-reverted ml-staging-codfw
kevinbazira@deploy1003:~$ kubectl get pods
NAME READY STATUS RESTARTS AGE
viwiki-reverted-predictor-default-00027-deployment-8864db7dq5hj 3/3 Running 0 60s
kevinbazira@deploy1003:~$
kevinbazira@deploy1003:~$
kevinbazira@deploy1003:~$ curl "https://inference-staging.svc.codfw.wmnet:30443/v1/models/viwiki-reverted:predict" -X POST -d '{"rev_id": 12345}' -H "Host: viwiki-reverted.revscoring-editquality-reverted.wikimedia.org" -H "Content-Type: application/json" --http1.1
{"viwiki":{"models":{"reverted":{"version":"0.5.0"}},"scores":{"12345":{"reverted":{"score":{"prediction":false,"probability":{"false":0.9342944442219189,"true":0.06570555577808117}}}}}}}Comment Actions
Change #1182770 had a related patch set uploaded (by Kevin Bazira; author: Kevin Bazira):
[operations/deployment-charts@master] ml-services: update revscoring production image
Comment Actions
enwiki-damaging load test results are passing, but enwiki-goodfaith is failing because this isvc is not deployed in staging, but zhwiki-goodfaith is as shown in T400350#11127687.
[2025-08-28 09:28:10,086] stat1010/INFO/locust.main: Shutting down (exit code 1)
Type Name # reqs # fails | Avg Min Max Med | req/s failures/s
--------|----------------------------------------------------------------------------|-------|-------------|-------|-------|-------|-------|--------|-----------
POST /v1/models/enwiki-damaging:predict 32 0(0.00%) | 628 183 4317 310 | 0.27 0.00
POST /v1/models/enwiki-goodfaith:predict 38 38(100.00%) | 174 167 186 170 | 0.32 0.32
--------|----------------------------------------------------------------------------|-------|-------------|-------|-------|-------|-------|--------|-----------
Aggregated 70 38(54.29%) | 382 167 4317 180 | 0.59 0.32
Response time percentiles (approximated)
Type Name 50% 66% 75% 80% 90% 95% 98% 99% 99.9% 99.99% 100% # reqs
--------|--------------------------------------------------------------------------------|--------|------|------|------|------|------|------|------|------|------|------|------
POST /v1/models/enwiki-damaging:predict 370 430 530 620 920 3500 4300 4300 4300 4300 4300 32
POST /v1/models/enwiki-goodfaith:predict 170 180 180 180 180 180 190 190 190 190 190 38
--------|--------------------------------------------------------------------------------|--------|------|------|------|------|------|------|------|------|------|------|------
Aggregated 180 270 310 420 620 920 3500 4300 4300 4300 4300 70
Error report
# occurrences Error
------------------|---------------------------------------------------------------------------------------------------------------------------------------------
38 POST /v1/models/enwiki-goodfaith:predict: BadStatusCode('https://inference-staging.svc.codfw.wmnet:30443/v1/models/enwiki-goodfaith:predict', code=404)
------------------|---------------------------------------------------------------------------------------------------------------------------------------------Comment Actions
@kevinbazira it seems that are locust load test config doesn't match the deployed models on staging. After you deploy the new images in production, could you either fix the load tests or open a task about it so that the next person on rotation can tackle it? Thanks!
Comment Actions
@isarantopoulos, yep, after deploying in prod, I will open a task to fix the revscoring goodfaith locust load tests.
The patch for prod deployment is ready: https://gerrit.wikimedia.org/r/1182770
Please review it whenever you get a minute. Thanks!
Comment Actions
Change #1182770 merged by jenkins-bot:
[operations/deployment-charts@master] ml-services: update revscoring production image
Comment Actions
All revscoring isvcs have been successfully upgraded to the new image, which supports bookworm and python3.10. They are up and running in prod.
I have also opened a follow-up task to T403236: Fix revscoring load tests to match staging deployments