Page MenuHomePhabricator
Create Task
Maniphest T415892

Add Multilingual RevertRisk predictions to mediawiki.page_revert_risk_prediction_change
Closed, ResolvedPublic
Actions

Description

Slack thread: https://wikimedia.slack.com/archives/C05F8ERE2CV/p1769524602443549

Use Case

Query multilingual revert risk scores for a month of revisions via SQL instead of multiple API calls, needed for analysis in T374698.

Currently, the event.mediawiki_page_revert_risk_prediction_change_v1 table contains only RRLA (revertrisk-language-agnostic) predictions, not revertrisk-multilingual predictions.

Proposed Solution

Produce Multilingual RevertRisk predictions to the mediawiki.page_revert_risk_prediction_change stream, following the RRLA implementation approach. The Multilingual model is slower and more resource-intensive than RRLA, so model optimization may be needed first.

Related Work

Details

Related Changes in Gerrit:
SubjectAuthorRepoBranchLines +/-
changeprop: Configure all wikis for revertrisk-multilingual events.Gkyziridisoperations/deployment-chartsmaster+1 -1
expand_event_sanitized_analytics_allowlist: Add revertrisk-multilingual predictions to allowlist.Gkyziridisanalytics/refinerymaster+1 -0
ml-services: Deploy the latest version of revertrisk-multilingual model on prod.Gkyziridisoperations/deployment-chartsmaster+9 -5
eventstreams: Configure new stream for revertrisk-multilingual model.Gkyziridisoperations/deployment-chartsmaster+1 -0
changeprop: Configure RevertRisk multilingual model on changeprop.Gkyziridisoperations/deployment-chartsmaster+18 -0
ml-services: Deploy the latest version of rr-multilingual model server on prod.Gkyziridisoperations/deployment-chartsmaster+13 -8
ml-services: Use concurrency knative metric for rr-multilingual model on staging.Gkyziridisoperations/deployment-chartsmaster+1 -1
ml-services: Deploy rr-multilingual latest model version on staging.Gkyziridisoperations/deployment-chartsmaster+4 -4
ml-services: Deploy rr_multilingual latest version on experimental staging.Gkyziridisoperations/deployment-chartsmaster+2 -2
revertrisk-multilingual: Remove await from predict.Gkyziridismachinelearning/liftwing/inference-servicesmain+1 -4
changeprop: Remove rr-multilingual model from changeprop.Gkyziridisoperations/deployment-chartsmaster+0 -18
ml-services: Deploy new version of revertrisk-multilingual model on experimental staging.Gkyziridisoperations/deployment-chartsmaster+6 -6
revertrisk-multilingual: Optimize rr-multilingual model for asynchronous predictions.Gkyziridismachinelearning/liftwing/inference-servicesmain+4 -1
ml-services: Deploy rr-multilingual and langugage-agnostic in experimental staging.Gkyziridisoperations/deployment-chartsmaster+12 -5
ml-services: Deploy rr-multilingual model on experimental with autoscaling enabled.Gkyziridisoperations/deployment-chartsmaster+33 -0
ml-services: Deploy autoscaling for rr-mulgilingual on staging.Gkyziridisoperations/deployment-chartsmaster+2 -2
ml-services: Configure autoscaling for rr-multilingual on staging.Gkyziridisoperations/deployment-chartsmaster+9 -8
ml-services: Deploy rr-multilingual gpu model and eventstream in prod.Gkyziridisoperations/deployment-chartsmaster+28 -4
EventStreamConfig: Add rr-multilingual prediction_change streamGkyziridisoperations/mediawiki-configmaster+4 -0
ml-services: Deploy the newest version of rr-multulingual model on staging.Gkyziridisoperations/deployment-chartsmaster+1 -1
rr-multilingual: Fix typo for using gpu.Gkyziridismachinelearning/liftwing/inference-servicesmain+1 -1
changeprop: Add revertrisk-multilingual model to changeprop staging configuration.Gkyziridisoperations/deployment-chartsmaster+40 -3
changeprop: Add revertrisk-multilingual model ti changeprop staging configuration.Gkyziridisoperations/deployment-chartsmaster+36 -1
Revertrisk-multilingual: Add predictions to events stream.Gkyziridismachinelearning/liftwing/inference-servicesmain+157 -13
Show related patches Customize query in gerrit

Related Objects
Search...

Event Timeline

There are a very large number of changes, so older changes are hidden. Show Older Changes
Ottomata added a comment.Apr 2 2026, 5:34 PM

changeprop errors

Weird! These indeed look like some logging related error. Trying to access a field (stream?) that doesn't exist for logging purposes. If this is happening for you it is probably happening for all events.

isarantopoulos edited projects, added: Machine-Learning-Team (Q1 FY2026-27); removed: Machine-Learning-Team.Apr 7 2026, 12:22 PM
isarantopoulos moved this task from Unsorted/Needs Triage to In Progress on the Machine-Learning-Team (Q1 FY2026-27) board.Apr 7 2026, 12:27 PM
gerritbot added a comment.Apr 8 2026, 8:56 AM

Change #1268903 had a related patch set uploaded (by Gkyziridis; author: Gkyziridis):

[operations/deployment-charts@master] ml-services: Configure autoscaling for rr-multilingual on staging.

https://gerrit.wikimedia.org/r/1268903

gerritbot added a project: Patch-For-Review.Apr 8 2026, 8:56 AM
gerritbot added a comment.Apr 8 2026, 9:02 AM

Change #1268903 merged by jenkins-bot:

[operations/deployment-charts@master] ml-services: Configure autoscaling for rr-multilingual on staging.

https://gerrit.wikimedia.org/r/1268903

Maintenance_bot removed a project: Patch-For-Review.Apr 8 2026, 9:30 AM
gerritbot added a comment.Apr 8 2026, 11:30 AM

Change #1268923 had a related patch set uploaded (by Gkyziridis; author: Gkyziridis):

[operations/deployment-charts@master] ml-services: Deploy autoscaling for rr-mulgilingual on staging.

https://gerrit.wikimedia.org/r/1268923

gerritbot added a project: Patch-For-Review.Apr 8 2026, 11:30 AM
gerritbot added a comment.Apr 8 2026, 11:35 AM

Change #1268923 merged by jenkins-bot:

[operations/deployment-charts@master] ml-services: Deploy autoscaling for rr-mulgilingual on staging.

https://gerrit.wikimedia.org/r/1268923

gerritbot added a comment.Apr 8 2026, 11:46 AM

Change #1268927 had a related patch set uploaded (by Gkyziridis; author: Gkyziridis):

[operations/deployment-charts@master] ml-services: Deploy rr-multilingual model on experimental with autoscaling enabled.

https://gerrit.wikimedia.org/r/1268927

gerritbot added a comment.Apr 8 2026, 11:51 AM

Change #1268927 merged by jenkins-bot:

[operations/deployment-charts@master] ml-services: Deploy rr-multilingual model on experimental with autoscaling enabled.

https://gerrit.wikimedia.org/r/1268927

Maintenance_bot removed a project: Patch-For-Review.Apr 8 2026, 12:31 PM
gerritbot added a comment.Apr 8 2026, 1:41 PM

Change #1268963 had a related patch set uploaded (by Gkyziridis; author: Gkyziridis):

[operations/deployment-charts@master] ml-services: Deploy rr-multilingual and langugage-agnostic in experimental staging.

https://gerrit.wikimedia.org/r/1268963

gerritbot added a project: Patch-For-Review.Apr 8 2026, 1:41 PM
gerritbot added a comment.Apr 8 2026, 1:44 PM

Change #1268963 merged by jenkins-bot:

[operations/deployment-charts@master] ml-services: Deploy rr-multilingual and langugage-agnostic in experimental staging.

https://gerrit.wikimedia.org/r/1268963

Maintenance_bot removed a project: Patch-For-Review.Apr 8 2026, 2:32 PM
gerritbot added a comment.Apr 20 2026, 9:48 AM

Change #1275362 had a related patch set uploaded (by Gkyziridis; author: Gkyziridis):

[machinelearning/liftwing/inference-services@main] revertrisk-multilingual: Optimize rr-multilingual model for asynchronous predictions.

https://gerrit.wikimedia.org/r/1275362

gerritbot added a project: Patch-For-Review.Apr 20 2026, 9:48 AM
gerritbot added a comment.Apr 21 2026, 7:10 AM

Change #1275362 merged by jenkins-bot:

[machinelearning/liftwing/inference-services@main] revertrisk-multilingual: Optimize rr-multilingual model for asynchronous predictions.

https://gerrit.wikimedia.org/r/1275362

jenkins-bot mentioned this in rMLIS240468c03be3: revertrisk-multilingual: Optimize rr-multilingual model for asynchronous….Apr 21 2026, 7:11 AM
Maintenance_bot removed a project: Patch-For-Review.Apr 21 2026, 7:29 AM
gerritbot added a comment.Apr 21 2026, 8:42 AM

Change #1275810 had a related patch set uploaded (by Gkyziridis; author: Gkyziridis):

[operations/deployment-charts@master] ml-services: Deploy new version of revertrisk-multilingual model on experimental staging.

https://gerrit.wikimedia.org/r/1275810

gerritbot added a project: Patch-For-Review.Apr 21 2026, 8:42 AM
gerritbot added a comment.Apr 21 2026, 8:50 AM

Change #1275810 merged by jenkins-bot:

[operations/deployment-charts@master] ml-services: Deploy new version of revertrisk-multilingual model on experimental staging.

https://gerrit.wikimedia.org/r/1275810

Maintenance_bot removed a project: Patch-For-Review.Apr 21 2026, 9:32 AM
gerritbot added a comment.Apr 21 2026, 9:56 AM

Change #1275832 had a related patch set uploaded (by Gkyziridis; author: Gkyziridis):

[operations/deployment-charts@master] changeprop: Remove rr-multilingual model from changeprop.

https://gerrit.wikimedia.org/r/1275832

gerritbot added a project: Patch-For-Review.Apr 21 2026, 9:56 AM
gerritbot added a comment.Apr 21 2026, 10:31 AM

Change #1275832 merged by jenkins-bot:

[operations/deployment-charts@master] changeprop: Remove rr-multilingual model from changeprop.

https://gerrit.wikimedia.org/r/1275832

Maintenance_bot removed a project: Patch-For-Review.Apr 21 2026, 10:31 AM
gkyziridis added a comment.Apr 21 2026, 3:25 PM

I am pasting here some results from loading tests.
Locust Test results:

locust RevertriskMultilingual --headless   --users 35   --spawn-rate 5   --run-time 120s   --only-summary
[2026-04-21 15:17:11,442] stat1010/INFO/locust.main: Run time limit set to 120 seconds
[2026-04-21 15:17:11,442] stat1010/INFO/locust.runners: Ramping to 35 users at a rate of 5.00 per second
[2026-04-21 15:17:17,447] stat1010/INFO/locust.runners: All users spawned: {"RevertriskMultilingual": 35} (35 total users)
[2026-04-21 15:19:10,981] stat1010/INFO/locust.main: Shutting down (exit code 1)
Type     Name                                                                          # reqs      # fails |    Avg     Min     Max    Med |   req/s  failures/s
--------|----------------------------------------------------------------------------|-------|-------------|-------|-------|-------|-------|--------|-----------
POST     /v1/models/revertrisk-multilingual:predict                                       931     7(0.75%) |   1351      65   24093    430 |    7.81        0.06
--------|----------------------------------------------------------------------------|-------|-------------|-------|-------|-------|-------|--------|-----------
         Aggregated                                                                       931     7(0.75%) |   1351      65   24093    430 |    7.81        0.06

Response time percentiles (approximated)
Type     Name                                                                                  50%    66%    75%    80%    90%    95%    98%    99%  99.9% 99.99%   100% # reqs
--------|--------------------------------------------------------------------------------|--------|------|------|------|------|------|------|------|------|------|------|------
POST     /v1/models/revertrisk-multilingual:predict                                            430    760   1200   1600   3700   6900   9700  12000  24000  24000  24000    931
--------|--------------------------------------------------------------------------------|--------|------|------|------|------|------|------|------|------|------|------|------
         Aggregated                                                                            430    760   1200   1600   3700   6900   9700  12000  24000  24000  24000    931

Error report
# occurrences      Error                                                                                               
------------------|---------------------------------------------------------------------------------------------------------------------------------------------
6                  POST /v1/models/revertrisk-multilingual:predict: LocustBadStatusCode(code=422)                      
1                  POST /v1/models/revertrisk-multilingual:predict: LocustBadStatusCode(code=502)                      
------------------|---------------------------------------------------------------------------------------------------------------------------------------------

The current deployment on experimental is:

revertrisk-multilingual:
    annotations:
      autoscaling.knative.dev/target: "5"
    predictor:
      image: "machinelearning-liftwing-inference-services-revertrisk-multilingual"
      image_version: "2026-04-21-071104-publish"
      config:
        minReplicas: 3
        maxReplicas: 6
      custom_env:
        - name: MODEL_NAME
          value: "revertrisk-multilingual"
        - name: STORAGE_URI
          value: "s3://wmf-ml-models/revertrisk/multilingual/20230810110019/"
        - name: WIKI_URL
          value: "http://mw-api-int-ro.discovery.wmnet:4680"
        - name: FORCE_HTTP
          value: "true"
        - name: ALLOW_REVISION_JSON_INPUT
          value: "true"
        - name: EVENTGATE_URL
          value: "http://eventgate-main.discovery.wmnet:4480/v1/events"
        - name: EVENTGATE_STREAM
          value: "mediawiki.page_prediction_change.rc0"
      container:
        resources:
          limits:
            cpu: "6"
            memory: 10Gi
          requests:
            cpu: "6"
            memory: 10Gi
gerritbot added a comment.Apr 28 2026, 12:54 PM

Change #1278443 had a related patch set uploaded (by Gkyziridis; author: Gkyziridis):

[machinelearning/liftwing/inference-services@main] revertrisk-multilingual: Remove await from predict.

https://gerrit.wikimedia.org/r/1278443

gerritbot added a project: Patch-For-Review.Apr 28 2026, 12:54 PM
gkyziridis added a comment.EditedApr 28 2026, 1:06 PM

Update

After some changes in the model server regarding to async calling, the latest locust results look like this:

$ (.venv) gkyziridis@stat1010:~/inference-services/test/locust$ locust RevertriskMultilingual --headless   --users 35   --spawn-rate 5   --run-time 120s   --only-summary
[2026-04-28 16:06:55,923] stat1010/INFO/locust.main: Starting Locust 2.33.2
[2026-04-28 16:06:55,923] stat1010/WARNING/locust.main: Python 3.9 support is deprecated and will be removed soon
[2026-04-28 16:06:55,923] stat1010/INFO/locust.main: Run time limit set to 120 seconds
[2026-04-28 16:06:55,924] stat1010/INFO/locust.runners: Ramping to 35 users at a rate of 5.00 per second
[2026-04-28 16:08:55,432] stat1010/INFO/locust.main: Shutting down (exit code 1)
Type     Name                                                                          # reqs      # fails |    Avg     Min     Max    Med |   req/s  failures/s
--------|----------------------------------------------------------------------------|-------|-------------|-------|-------|-------|-------|--------|-----------
POST     /v1/models/revertrisk-multilingual:predict                                      1743    22(1.26%) |   1324      65   10053    820 |   14.59        0.18
--------|----------------------------------------------------------------------------|-------|-------------|-------|-------|-------|-------|--------|-----------
         Aggregated                                                                      1743    22(1.26%) |   1324      65   10053    820 |   14.59        0.18

Response time percentiles (approximated)
Type     Name                                                                                  50%    66%    75%    80%    90%    95%    98%    99%  99.9% 99.99%   100% # reqs
--------|--------------------------------------------------------------------------------|--------|------|------|------|------|------|------|------|------|------|------|------
POST     /v1/models/revertrisk-multilingual:predict                                            820   1300   1800   2200   3200   4200   5400   5900   8200  10000  10000   1743
--------|--------------------------------------------------------------------------------|--------|------|------|------|------|------|------|------|------|------|------|------
         Aggregated                                                                            820   1300   1800   2200   3200   4200   5400   5900   8200  10000  10000   1743

Error report
# occurrences      Error                                                                                               
------------------|---------------------------------------------------------------------------------------------------------------------------------------------
22                 POST /v1/models/revertrisk-multilingual:predict: LocustBadStatusCode(code=422)                      
------------------|---------------------------------------------------------------------------------------------------------------------------------------------

Experimental deployment tested:

revertrisk-multilingual:
  annotations:
    autoscaling.knative.dev/target: "5"
  predictor:
    image: "machinelearning-liftwing-inference-services-revertrisk-multilingual"
    image_version: "2026-04-28-153227-publish"
    config:
      minReplicas: 5
      maxReplicas: 10
    custom_env:
      - name: MODEL_NAME
        value: "revertrisk-multilingual"
      - name: STORAGE_URI
        value: "s3://wmf-ml-models/revertrisk/multilingual/20230810110019/"
      - name: WIKI_URL
        value: "http://mw-api-int-ro.discovery.wmnet:4680"
      - name: FORCE_HTTP
        value: "true"
      - name: ALLOW_REVISION_JSON_INPUT
        value: "true"
      - name: EVENTGATE_URL
        value: "http://eventgate-main.discovery.wmnet:4480/v1/events"
      - name: EVENTGATE_STREAM
        value: "mediawiki.page_prediction_change.rc0"
    container:
      resources:
        limits:
          cpu: "6"
          memory: 10Gi
        requests:
          cpu: "6"
          memory: 10Gi

I will create the patches for the deployment.

gerritbot added a comment.Apr 28 2026, 3:32 PM

Change #1278443 merged by jenkins-bot:

[machinelearning/liftwing/inference-services@main] revertrisk-multilingual: Remove await from predict.

https://gerrit.wikimedia.org/r/1278443

gkyziridis mentioned this in rMLISbc7649035a20: revertrisk-multilingual: Remove await from predict..Apr 28 2026, 3:34 PM
gerritbot added a comment.Apr 28 2026, 3:45 PM

Change #1278500 had a related patch set uploaded (by Gkyziridis; author: Gkyziridis):

[operations/deployment-charts@master] ml-services: Deploy rr_multilingual latest version on experimental staging.

https://gerrit.wikimedia.org/r/1278500

gerritbot added a comment.Apr 28 2026, 3:50 PM

Change #1278500 merged by jenkins-bot:

[operations/deployment-charts@master] ml-services: Deploy rr_multilingual latest version on experimental staging.

https://gerrit.wikimedia.org/r/1278500

Maintenance_bot removed a project: Patch-For-Review.Apr 28 2026, 4:32 PM
gkyziridis added a comment.EditedApr 29 2026, 8:49 AM

Hey @achou, I think that this change is adding some extra latency as well.

I also ran a heavier locust test to test autoscaling and results look like this:

$ locust RevertriskMultilingual --headless   --users 55   --spawn-rate 5   --run-time 180s   --only-summary
[2026-04-29 08:36:36,105] stat1010/INFO/locust.main: Starting Locust 2.33.2
[2026-04-29 08:36:36,106] stat1010/WARNING/locust.main: Python 3.9 support is deprecated and will be removed soon
[2026-04-29 08:36:36,106] stat1010/INFO/locust.main: Run time limit set to 180 seconds
[2026-04-29 08:36:36,107] stat1010/INFO/locust.runners: Ramping to 55 users at a rate of 5.00 per second
[2026-04-29 08:39:35,632] stat1010/INFO/locust.main: Shutting down (exit code 1)
Type     Name                                                                          # reqs      # fails |    Avg     Min     Max    Med |   req/s  failures/s
--------|----------------------------------------------------------------------------|-------|-------------|-------|-------|-------|-------|--------|-----------
POST     /v1/models/revertrisk-multilingual:predict                                      4107    36(0.88%) |   1337      63    9100    800 |   22.88        0.20
--------|----------------------------------------------------------------------------|-------|-------------|-------|-------|-------|-------|--------|-----------
         Aggregated                                                                      4107    36(0.88%) |   1337      63    9100    800 |   22.88        0.20

Response time percentiles (approximated)
Type     Name                                                                                  50%    66%    75%    80%    90%    95%    98%    99%  99.9% 99.99%   100% # reqs
--------|--------------------------------------------------------------------------------|--------|------|------|------|------|------|------|------|------|------|------|------
POST     /v1/models/revertrisk-multilingual:predict                                            800   1300   1800   2200   3300   4200   5300   6400   8100   9100   9100   4107
--------|--------------------------------------------------------------------------------|--------|------|------|------|------|------|------|------|------|------|------|------
         Aggregated                                                                            800   1300   1800   2200   3300   4200   5300   6400   8100   9100   9100   4107

Error report
# occurrences      Error                                                                                               
------------------|---------------------------------------------------------------------------------------------------------------------------------------------
36                 POST /v1/models/revertrisk-multilingual:predict: LocustBadStatusCode(code=422)                      
------------------|---------------------------------------------------------------------------------------------------------------------------------------------

✅ Load tests seem to be fine
✅ Events and streams are tested: https://phabricator.wikimedia.org/T415892#11782975

Should I proceed to deploy this?

gerritbot added a comment.Apr 29 2026, 9:10 AM

Change #1279244 had a related patch set uploaded (by Gkyziridis; author: Gkyziridis):

[operations/deployment-charts@master] ml-services: Deploy rr-multilingual latest model version on staging.

https://gerrit.wikimedia.org/r/1279244

gerritbot added a project: Patch-For-Review.Apr 29 2026, 9:10 AM
gerritbot added a comment.Apr 29 2026, 12:49 PM

Change #1279244 merged by jenkins-bot:

[operations/deployment-charts@master] ml-services: Deploy rr-multilingual latest model version on staging.

https://gerrit.wikimedia.org/r/1279244

achou added a comment.Apr 29 2026, 1:01 PM

I think that this change is adding some extra latency as well.

That change only moved the torch import inside the rr-multilingual branch to avoid import errors for the RRLA model (which doesn't require torch). It doesn't change the logic for rr-multilingual, so in theory it wouldn't affect latency.

36 POST /v1/models/revertrisk-multilingual:predict: LocustBadStatusCode(code=422)

Regarding the load test results, have you looked at these 422 errors? Although it's not a lot, it would be great to figure out why those are popping up

Maintenance_bot removed a project: Patch-For-Review.Apr 29 2026, 1:33 PM
gkyziridis added a comment.EditedApr 29 2026, 2:05 PM
In T415892#11871148, @achou wrote:

36 POST /v1/models/revertrisk-multilingual:predict: LocustBadStatusCode(code=422)

Regarding the load test results, have you looked at these 422 errors? Although it's not a lot, it would be great to figure out why those are popping up

I think that those errors are due to missing revisions which are randomly selected from the "data/revisions_lang_and_id.tsv" during the locust test.
Basically is a specific revision_id that is missing, check the logs:

INFO:root:Model Server: RevertRiskMultilingualGPU
INFO:root:Successfully loaded 342 canonical wiki languages.
WARNING:root:CUDA is not available or PyTorch is CPU-only; using CPU instead.
2026-04-29 14:47:07.078 1 kserve INFO [model_server.py:register_model():402] Registering model: revertrisk-multilingual
2026-04-29 14:47:07.079 1 kserve INFO [model_server.py:setup_event_loop():282] Setting max asyncio worker threads as 32
2026-04-29 14:47:07.130 1 kserve INFO [server.py:_register_endpoints():110] OpenAI endpoints not registered
2026-04-29 14:47:07.130 1 kserve INFO [server.py:start():161] Starting uvicorn with 1 workers
2026-04-29 14:47:07.144 1 uvicorn.error INFO:     Started server process [1]
2026-04-29 14:47:07.144 1 uvicorn.error INFO:     Waiting for application startup.
2026-04-29 14:47:07.148 1 kserve INFO [server.py:start():70] Starting gRPC server with 4 workers
2026-04-29 14:47:07.149 1 kserve INFO [server.py:start():71] Starting gRPC server on [::]:8081
2026-04-29 14:47:07.149 1 uvicorn.error INFO:     Application startup complete.
2026-04-29 14:47:07.149 1 uvicorn.error INFO:     Uvicorn running on http://0.0.0.0:8080 (Press CTRL+C to quit)
INFO:root:Opening a new Asyncio session for mwapi.
INFO:root:revision 1096365864 (en): revision_missing
INFO:root:revision 1096365864 (en): revision_missing
gerritbot added a comment.Apr 29 2026, 2:18 PM

Change #1279373 had a related patch set uploaded (by Gkyziridis; author: Gkyziridis):

[operations/deployment-charts@master] ml-services: Use concurrency knative metric for rr-multilingual model on staging.

https://gerrit.wikimedia.org/r/1279373

gerritbot added a project: Patch-For-Review.Apr 29 2026, 2:18 PM
gerritbot added a comment.Apr 29 2026, 2:24 PM

Change #1279373 merged by jenkins-bot:

[operations/deployment-charts@master] ml-services: Use concurrency knative metric for rr-multilingual model on staging.

https://gerrit.wikimedia.org/r/1279373

Maintenance_bot removed a project: Patch-For-Review.Apr 29 2026, 2:31 PM
gerritbot added a comment.Apr 29 2026, 2:51 PM

Change #1279385 had a related patch set uploaded (by Gkyziridis; author: Gkyziridis):

[operations/deployment-charts@master] changeprop: Configure RevertRisk multilingual model on changeprop.

https://gerrit.wikimedia.org/r/1279385

gerritbot added a project: Patch-For-Review.Apr 29 2026, 2:51 PM
gerritbot added a comment.Apr 29 2026, 3:12 PM

Change #1279388 had a related patch set uploaded (by Gkyziridis; author: Gkyziridis):

[operations/deployment-charts@master] ml-services: Deploy the latest version of rr-multilingual model server on prod.

https://gerrit.wikimedia.org/r/1279388

gerritbot added a comment.Apr 30 2026, 12:43 PM

Change #1279388 merged by jenkins-bot:

[operations/deployment-charts@master] ml-services: Deploy the latest version of rr-multilingual model server on prod.

https://gerrit.wikimedia.org/r/1279388

gerritbot added a comment.Apr 30 2026, 12:59 PM

Change #1279385 merged by Dpogorzelski:

[operations/deployment-charts@master] changeprop: Configure RevertRisk multilingual model on changeprop.

https://gerrit.wikimedia.org/r/1279385

Maintenance_bot removed a project: Patch-For-Review.Apr 30 2026, 1:31 PM
Ottomata added a comment.Apr 30 2026, 3:00 PM

changeprop: Configure RevertRisk multilingual model on changeprop. (1279385)

Huh! I am not fresh on changeprop rule configs. Does changeprop hit LiftWing endpoint multiple times for the same input page_change event? I'm not sure if I knew that!

If it is doing that...I wonder: Instead, could each page_change event cause a single HTTP call to LiftWing, and LiftWing could handle the logic for what models should score and what events should be produced?

Or, is the isolation of each individual matched event -> LiftWing inference + event produce better?

isarantopoulos subscribed.Apr 30 2026, 3:09 PM

individual matched event -> LiftWing inference + event produce better?

@Ottomata This indeed makes more sense. LiftWing doesn't have a central gateway for routing requests so matched events go to a specific microservice (revertrisk language agnostic, articletopic etc).

gkyziridis added a comment.Apr 30 2026, 4:06 PM

I am having errors on the current deployment, please check these logs in the paste:

P92097 Errors from RevertRisk-Multilingual
1ERROR:root:Connection error while sending an event to EventGate: Cannot connect to host eventgate-main.discovery.wmnet:4480 ssl:<ssl.SSLContext object at 0x7f5630172180> [Temporary failure in name resolution]
22026-04-30 15:27:23.530 1 kserve ERROR [errors.py:generic_exception_handler():143] Exception:
3Traceback (most recent call last):
4 File "/opt/lib/venv/lib/python3.11/site-packages/aiohttp/connector.py", line 1562, in _create_direct_connection
5 hosts = await self._resolve_host(host, port, traces=traces)
6 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
7 File "/opt/lib/venv/lib/python3.11/site-packages/aiohttp/connector.py", line 1153, in _resolve_host
8 await future
9 File "/opt/lib/venv/lib/python3.11/site-packages/aiohttp/connector.py", line 1209, in _resolve_host_with_throttle
10 addrs = await self._resolver.resolve(host, port, family=self._family)
11 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
12 File "/opt/lib/venv/lib/python3.11/site-packages/aiohttp/resolver.py", line 40, in resolve
13 infos = await self._loop.getaddrinfo(
14 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
15 File "/usr/lib/python3.11/asyncio/base_events.py", line 867, in getaddrinfo
16 return await self.run_in_executor(
17 ^^^^^^^^^^^^^^^^^^^^^^^^^^^
18 File "/usr/lib/python3.11/concurrent/futures/thread.py", line 58, in run
19 result = self.fn(*self.args, **self.kwargs)
20 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
21 File "/usr/lib/python3.11/socket.py", line 962, in getaddrinfo
22 for res in _socket.getaddrinfo(host, port, family, type, proto, flags):
23 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
24socket.gaierror: [Errno -3] Temporary failure in name resolution
25
26The above exception was the direct cause of the following exception:
27
28Traceback (most recent call last):
29 File "/srv/revert_risk_model/python/events.py", line 228, in send_event
30 async with aio_http_client.post(
31 File "/opt/lib/venv/lib/python3.11/site-packages/aiohttp/client.py", line 1510, in __aenter__
32 self._resp: _RetType = await self._coro
33 ^^^^^^^^^^^^^^^^
34 File "/opt/lib/venv/lib/python3.11/site-packages/aiohttp/client.py", line 779, in _request
35 resp = await handler(req)
36 ^^^^^^^^^^^^^^^^^^
37 File "/opt/lib/venv/lib/python3.11/site-packages/aiohttp/client.py", line 734, in _connect_and_send_request
38 conn = await self._connector.connect(
39 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
40 File "/opt/lib/venv/lib/python3.11/site-packages/aiohttp/connector.py", line 672, in connect
41 proto = await self._create_connection(req, traces, timeout)
42 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
43 File "/opt/lib/venv/lib/python3.11/site-packages/aiohttp/connector.py", line 1239, in _create_connection
44 _, proto = await self._create_direct_connection(req, traces, timeout)
45 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
46 File "/opt/lib/venv/lib/python3.11/site-packages/aiohttp/connector.py", line 1568, in _create_direct_connection
47 raise ClientConnectorDNSError(req.connection_key, exc) from exc
48aiohttp.client_exceptions.ClientConnectorDNSError: Cannot connect to host eventgate-main.discovery.wmnet:4480 ssl:<ssl.SSLContext object at 0x7f5630172180> [Temporary failure in name resolution]
49
50During handling of the above exception, another exception occurred:
51
52Traceback (most recent call last):
53 File "/opt/lib/venv/lib/python3.11/site-packages/starlette/middleware/errors.py", line 164, in __call__
54 await self.app(scope, receive, _send)
55 File "/opt/lib/venv/lib/python3.11/site-packages/timing_asgi/middleware.py", line 70, in __call__
56 await self.app(scope, receive, send_wrapper)
57 File "/opt/lib/venv/lib/python3.11/site-packages/starlette/middleware/exceptions.py", line 63, in __call__
58 await wrap_app_handling_exceptions(self.app, conn)(scope, receive, send)
59 File "/opt/lib/venv/lib/python3.11/site-packages/starlette/_exception_handler.py", line 53, in wrapped_app
60 raise exc
61 File "/opt/lib/venv/lib/python3.11/site-packages/starlette/_exception_handler.py", line 42, in wrapped_app
62 await app(scope, receive, sender)
63 File "/opt/lib/venv/lib/python3.11/site-packages/fastapi/middleware/asyncexitstack.py", line 18, in __call__
64 await self.app(scope, receive, send)
65 File "/opt/lib/venv/lib/python3.11/site-packages/starlette/routing.py", line 660, in __call__
66 await self.middleware_stack(scope, receive, send)
67 File "/opt/lib/venv/lib/python3.11/site-packages/starlette/routing.py", line 680, in app
68 await route.handle(scope, receive, send)
69 File "/opt/lib/venv/lib/python3.11/site-packages/starlette/routing.py", line 276, in handle
70 await self.app(scope, receive, send)
71 File "/opt/lib/venv/lib/python3.11/site-packages/fastapi/routing.py", line 134, in app
72 await wrap_app_handling_exceptions(app, request)(scope, receive, send)
73 File "/opt/lib/venv/lib/python3.11/site-packages/starlette/_exception_handler.py", line 53, in wrapped_app
74 raise exc
75 File "/opt/lib/venv/lib/python3.11/site-packages/starlette/_exception_handler.py", line 42, in wrapped_app
76 await app(scope, receive, sender)
77 File "/opt/lib/venv/lib/python3.11/site-packages/fastapi/routing.py", line 120, in app
78 response = await f(request)
79 ^^^^^^^^^^^^^^^^
80 File "/opt/lib/venv/lib/python3.11/site-packages/fastapi/routing.py", line 674, in app
81 raw_response = await run_endpoint_function(
82 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
83 File "/opt/lib/venv/lib/python3.11/site-packages/fastapi/routing.py", line 328, in run_endpoint_function
84 return await dependant.call(**values)
85 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
86 File "/opt/lib/venv/lib/python3.11/site-packages/kserve/protocol/rest/v1_endpoints.py", line 84, in predict
87 response, response_headers = await self.dataplane.infer(
88 ^^^^^^^^^^^^^^^^^^^^^^^^^^^
89 File "/opt/lib/venv/lib/python3.11/site-packages/kserve/protocol/dataplane.py", line 461, in infer
90 response, res_headers = await model(request, headers=headers)
91 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
92 File "/opt/lib/venv/lib/python3.11/site-packages/kserve/model.py", line 285, in __call__
93 (await self.predict(payload, headers))
94 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
95 File "/srv/revert_risk_model/model_server/gpu_model.py", line 126, in predict
96 await self.send_event(
97 File "/srv/revert_risk_model/model_server/gpu_model.py", line 158, in send_event
98 await events.send_event(
99 File "/srv/revert_risk_model/python/events.py", line 260, in send_event
100 raise RuntimeError(
101RuntimeError: Connection error while trying to post the event to EventGate. Please contact the ML team if the issue persists.
102
103====================================================================================================
104====================================================================================================
105
106
107ERROR:root:Unexpected error while trying to send an event to EventGate:
1082026-04-30 15:30:52.362 1 kserve ERROR [errors.py:generic_exception_handler():143] Exception:
109Traceback (most recent call last):
110 File "/opt/lib/venv/lib/python3.11/site-packages/aiohttp/client.py", line 779, in _request
111 resp = await handler(req)
112 ^^^^^^^^^^^^^^^^^^
113 File "/opt/lib/venv/lib/python3.11/site-packages/aiohttp/client.py", line 734, in _connect_and_send_request
114 conn = await self._connector.connect(
115 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
116 File "/opt/lib/venv/lib/python3.11/site-packages/aiohttp/connector.py", line 672, in connect
117 proto = await self._create_connection(req, traces, timeout)
118 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
119 File "/opt/lib/venv/lib/python3.11/site-packages/aiohttp/connector.py", line 1239, in _create_connection
120 _, proto = await self._create_direct_connection(req, traces, timeout)
121 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
122 File "/opt/lib/venv/lib/python3.11/site-packages/aiohttp/connector.py", line 1562, in _create_direct_connection
123 hosts = await self._resolve_host(host, port, traces=traces)
124 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
125 File "/opt/lib/venv/lib/python3.11/site-packages/aiohttp/connector.py", line 1153, in _resolve_host
126 await future
127asyncio.exceptions.CancelledError
128
129The above exception was the direct cause of the following exception:
130
131Traceback (most recent call last):
132 File "/srv/revert_risk_model/python/events.py", line 228, in send_event
133 async with aio_http_client.post(
134 File "/opt/lib/venv/lib/python3.11/site-packages/aiohttp/client.py", line 1510, in __aenter__
135 self._resp: _RetType = await self._coro
136 ^^^^^^^^^^^^^^^^
137 File "/opt/lib/venv/lib/python3.11/site-packages/aiohttp/client.py", line 624, in _request
138 with timer:
139 File "/opt/lib/venv/lib/python3.11/site-packages/aiohttp/helpers.py", line 713, in __exit__
140 raise asyncio.TimeoutError from exc_val
141TimeoutError
142
143During handling of the above exception, another exception occurred:
144
145Traceback (most recent call last):
146 File "/opt/lib/venv/lib/python3.11/site-packages/starlette/middleware/errors.py", line 164, in __call__
147 await self.app(scope, receive, _send)
148 File "/opt/lib/venv/lib/python3.11/site-packages/timing_asgi/middleware.py", line 70, in __call__
149 await self.app(scope, receive, send_wrapper)
150 File "/opt/lib/venv/lib/python3.11/site-packages/starlette/middleware/exceptions.py", line 63, in __call__
151 await wrap_app_handling_exceptions(self.app, conn)(scope, receive, send)
152 File "/opt/lib/venv/lib/python3.11/site-packages/starlette/_exception_handler.py", line 53, in wrapped_app
153 raise exc
154 File "/opt/lib/venv/lib/python3.11/site-packages/starlette/_exception_handler.py", line 42, in wrapped_app
155 await app(scope, receive, sender)
156 File "/opt/lib/venv/lib/python3.11/site-packages/fastapi/middleware/asyncexitstack.py", line 18, in __call__
157 await self.app(scope, receive, send)
158 File "/opt/lib/venv/lib/python3.11/site-packages/starlette/routing.py", line 660, in __call__
159 await self.middleware_stack(scope, receive, send)
160 File "/opt/lib/venv/lib/python3.11/site-packages/starlette/routing.py", line 680, in app
161 await route.handle(scope, receive, send)
162 File "/opt/lib/venv/lib/python3.11/site-packages/starlette/routing.py", line 276, in handle
163 await self.app(scope, receive, send)
164 File "/opt/lib/venv/lib/python3.11/site-packages/fastapi/routing.py", line 134, in app
165 await wrap_app_handling_exceptions(app, request)(scope, receive, send)
166 File "/opt/lib/venv/lib/python3.11/site-packages/starlette/_exception_handler.py", line 53, in wrapped_app
167 raise exc
168 File "/opt/lib/venv/lib/python3.11/site-packages/starlette/_exception_handler.py", line 42, in wrapped_app
169 await app(scope, receive, sender)
170 File "/opt/lib/venv/lib/python3.11/site-packages/fastapi/routing.py", line 120, in app
171 response = await f(request)
172 ^^^^^^^^^^^^^^^^
173 File "/opt/lib/venv/lib/python3.11/site-packages/fastapi/routing.py", line 674, in app
174 raw_response = await run_endpoint_function(
175 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
176 File "/opt/lib/venv/lib/python3.11/site-packages/fastapi/routing.py", line 328, in run_endpoint_function
177 return await dependant.call(**values)
178 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
179 File "/opt/lib/venv/lib/python3.11/site-packages/kserve/protocol/rest/v1_endpoints.py", line 84, in predict
180 response, response_headers = await self.dataplane.infer(
181 ^^^^^^^^^^^^^^^^^^^^^^^^^^^
182 File "/opt/lib/venv/lib/python3.11/site-packages/kserve/protocol/dataplane.py", line 461, in infer
183 response, res_headers = await model(request, headers=headers)
184 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
185 File "/opt/lib/venv/lib/python3.11/site-packages/kserve/model.py", line 285, in __call__
186 (await self.predict(payload, headers))
187 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
188 File "/srv/revert_risk_model/model_server/gpu_model.py", line 126, in predict
189 await self.send_event(
190 File "/srv/revert_risk_model/model_server/gpu_model.py", line 158, in send_event
191 await events.send_event(
192 File "/srv/revert_risk_model/python/events.py", line 268, in send_event
193 raise RuntimeError(
194RuntimeError: Unexpected error happened while the event was posted to EventGate, there is the possibility that it never reached it. Please contact the ML team if the issue persists.

I do not see stream mediawiki.page_revert_risk_multilingual_prediction_change.v1 in: stream.wikimedia.org
I also parsed this error regarding MediaWiki API

P92101 Fetching revesion from MediaWikiAPI error
1ERROR:root:An error has occurred while fetching info for revision: 1351855891 (en). Reason: Cannot connect to host mw-api-int-ro.discovery.wmnet:4680 ssl:default [Temporary failure in name resolution]
22026-04-30 15:44:23.729 1 kserve ERROR [errors.py:inference_error_handler():136] Exception:
3Traceback (most recent call last):
4 File "/opt/lib/venv/lib/python3.11/site-packages/aiohttp/connector.py", line 1562, in _create_direct_connection
5 hosts = await self._resolve_host(host, port, traces=traces)
6 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
7 File "/opt/lib/venv/lib/python3.11/site-packages/aiohttp/connector.py", line 1178, in _resolve_host
8 return await asyncio.shield(resolved_host_task)
9 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
10 File "/opt/lib/venv/lib/python3.11/site-packages/aiohttp/connector.py", line 1562, in _create_direct_connection
11 hosts = await self._resolve_host(host, port, traces=traces)
12 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
13 File "/opt/lib/venv/lib/python3.11/site-packages/aiohttp/connector.py", line 1153, in _resolve_host
14 await future
15 File "/opt/lib/venv/lib/python3.11/site-packages/aiohttp/connector.py", line 1209, in _resolve_host_with_throttle
16 addrs = await self._resolver.resolve(host, port, family=self._family)
17 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
18 File "/opt/lib/venv/lib/python3.11/site-packages/aiohttp/resolver.py", line 40, in resolve
19 infos = await self._loop.getaddrinfo(
20 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
21 File "/usr/lib/python3.11/asyncio/base_events.py", line 867, in getaddrinfo
22 return await self.run_in_executor(
23 ^^^^^^^^^^^^^^^^^^^^^^^^^^^
24 File "/usr/lib/python3.11/concurrent/futures/thread.py", line 58, in run
25 result = self.fn(*self.args, **self.kwargs)
26 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
27 File "/usr/lib/python3.11/socket.py", line 962, in getaddrinfo
28 for res in _socket.getaddrinfo(host, port, family, type, proto, flags):
29 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
30socket.gaierror: [Errno -3] Temporary failure in name resolution
31
32The above exception was the direct cause of the following exception:
33
34Traceback (most recent call last):
35 File "/opt/lib/venv/lib/python3.11/site-packages/mwapi/async_session.py", line 81, in _request
36 async with self.session.request(method=method, url=self.api_url,
37 File "/opt/lib/venv/lib/python3.11/site-packages/aiohttp/client.py", line 1510, in __aenter__
38 self._resp: _RetType = await self._coro
39 ^^^^^^^^^^^^^^^^
40 File "/opt/lib/venv/lib/python3.11/site-packages/aiohttp/client.py", line 779, in _request
41 resp = await handler(req)
42 ^^^^^^^^^^^^^^^^^^
43 File "/opt/lib/venv/lib/python3.11/site-packages/aiohttp/client.py", line 734, in _connect_and_send_request
44 conn = await self._connector.connect(
45 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
46 File "/opt/lib/venv/lib/python3.11/site-packages/aiohttp/connector.py", line 672, in connect
47 proto = await self._create_connection(req, traces, timeout)
48 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
49 File "/opt/lib/venv/lib/python3.11/site-packages/aiohttp/connector.py", line 1239, in _create_connection
50 _, proto = await self._create_direct_connection(req, traces, timeout)
51 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
52 File "/opt/lib/venv/lib/python3.11/site-packages/aiohttp/connector.py", line 1568, in _create_direct_connection
53 raise ClientConnectorDNSError(req.connection_key, exc) from exc
54aiohttp.client_exceptions.ClientConnectorDNSError: Cannot connect to host mw-api-int-ro.discovery.wmnet:4680 ssl:default [Temporary failure in name resolution]
55
56The above exception was the direct cause of the following exception:
57
58Traceback (most recent call last):
59 File "/srv/revert_risk_model/model_server/base_model.py", line 206, in preprocess
60 rev = await get_revision(session, rev_id, lang)
61 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
62 File "/opt/lib/venv/lib/python3.11/site-packages/knowledge_integrity/mediawiki.py", line 129, in get_revision
63 page, user, parent = await asyncio.gather(
64 ^^^^^^^^^^^^^^^^^^^^^
65 File "/opt/lib/venv/lib/python3.11/site-packages/knowledge_integrity/mediawiki.py", line 74, in get_parent_revision
66 response = await session.get(
67 ^^^^^^^^^^^^^^^^^^
68 File "/opt/lib/venv/lib/python3.11/site-packages/mwapi/async_session.py", line 185, in get
69 return await self.request("GET", params=params, auth=auth,
70 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
71 File "/opt/lib/venv/lib/python3.11/site-packages/mwapi/async_session.py", line 149, in request
72 return await self._request(method, params=normal_params, auth=auth)
73 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
74 File "/opt/lib/venv/lib/python3.11/site-packages/mwapi/async_session.py", line 112, in _request
75 raise ConnectionError(str(e)) from e
76mwapi.errors.ConnectionError: Cannot connect to host mw-api-int-ro.discovery.wmnet:4680 ssl:default [Temporary failure in name resolution]
77
78During handling of the above exception, another exception occurred:
79
80Traceback (most recent call last):
81 File "/opt/lib/venv/lib/python3.11/site-packages/starlette/_exception_handler.py", line 42, in wrapped_app
82 await app(scope, receive, sender)
83 File "/opt/lib/venv/lib/python3.11/site-packages/fastapi/routing.py", line 120, in app
84 response = await f(request)
85 ^^^^^^^^^^^^^^^^
86 File "/opt/lib/venv/lib/python3.11/site-packages/fastapi/routing.py", line 674, in app
87 raw_response = await run_endpoint_function(
88 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
89 File "/opt/lib/venv/lib/python3.11/site-packages/fastapi/routing.py", line 328, in run_endpoint_function
90 return await dependant.call(**values)
91 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
92 File "/opt/lib/venv/lib/python3.11/site-packages/kserve/protocol/rest/v1_endpoints.py", line 84, in predict
93 response, response_headers = await self.dataplane.infer(
94 ^^^^^^^^^^^^^^^^^^^^^^^^^^^
95 File "/opt/lib/venv/lib/python3.11/site-packages/kserve/protocol/dataplane.py", line 461, in infer
96 response, res_headers = await model(request, headers=headers)
97 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
98 File "/opt/lib/venv/lib/python3.11/site-packages/kserve/model.py", line 259, in __call__
99 await self.preprocess(body, headers)
100 File "/srv/revert_risk_model/model_server/gpu_model.py", line 104, in preprocess
101 return await super().preprocess(inputs, headers)
102 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
103 File "/srv/revert_risk_model/model_server/base_model.py", line 212, in preprocess
104 raise InferenceError(
105kserve.errors.InferenceError: An error happened while fetching info for revision from the MediaWiki API, please contact the ML-Team if the issue persists.

Ottomata added a comment.Apr 30 2026, 8:22 PM

I do not see stream mediawiki.page_revert_risk_multilingual_prediction_change.v1 in: stream.wikimedia.org

New stream, needs new config :)

https://gerrit.wikimedia.org/r/plugins/gitiles/operations/deployment-charts/+/refs/heads/master/helmfile.d/services/eventstreams/values.yaml#93

gerritbot added a comment.May 1 2026, 9:36 AM

Change #1281431 had a related patch set uploaded (by Gkyziridis; author: Gkyziridis):

[operations/deployment-charts@master] eventstreams: Configure new stream for revertrisk-multilingual model.

https://gerrit.wikimedia.org/r/1281431

gerritbot added a project: Patch-For-Review.May 1 2026, 9:36 AM
gerritbot added a comment.May 4 2026, 7:55 AM

Change #1281431 merged by jenkins-bot:

[operations/deployment-charts@master] eventstreams: Configure new stream for revertrisk-multilingual model.

https://gerrit.wikimedia.org/r/1281431

gerritbot added a comment.May 4 2026, 8:44 AM

Change #1282278 had a related patch set uploaded (by Gkyziridis; author: Gkyziridis):

[operations/deployment-charts@master] ml-services: Deploy the latest version of revertrisk-multilingual model on prod.

https://gerrit.wikimedia.org/r/1282278

gerritbot added a comment.May 4 2026, 8:51 AM

Change #1282278 merged by jenkins-bot:

[operations/deployment-charts@master] ml-services: Deploy the latest version of revertrisk-multilingual model on prod.

https://gerrit.wikimedia.org/r/1282278

gkyziridis added a comment.EditedMay 4 2026, 10:15 AM

The new stream is deployed on Eventstreams.
I can see streams at: stream.wikimedia.org.
I am still getting some errors:

P92175 Errors RevertRisk-Multilingual
1RuntimeError: Unexpected error happened while the event was posted to EventGate, there is the possibility that it never reached it. Please contact the ML team if the issue persists.
22026-05-04 09:58:11.528 1 uvicorn.error ERROR: Exception in ASGI application
3Traceback (most recent call last):
4 File "/opt/lib/venv/lib/python3.11/site-packages/aiohttp/client.py", line 779, in _request
5 resp = await handler(req)
6 ^^^^^^^^^^^^^^^^^^
7 File "/opt/lib/venv/lib/python3.11/site-packages/aiohttp/client.py", line 734, in _connect_and_send_request
8 conn = await self._connector.connect(
9 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
10 File "/opt/lib/venv/lib/python3.11/site-packages/aiohttp/connector.py", line 672, in connect
11 proto = await self._create_connection(req, traces, timeout)
12 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
13 File "/opt/lib/venv/lib/python3.11/site-packages/aiohttp/connector.py", line 1239, in _create_connection
14 _, proto = await self._create_direct_connection(req, traces, timeout)
15 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
16 File "/opt/lib/venv/lib/python3.11/site-packages/aiohttp/connector.py", line 1562, in _create_direct_connection
17 hosts = await self._resolve_host(host, port, traces=traces)
18 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
19 File "/opt/lib/venv/lib/python3.11/site-packages/aiohttp/connector.py", line 1153, in _resolve_host
20 await future
21asyncio.exceptions.CancelledError
22
23The above exception was the direct cause of the following exception:
24
25Traceback (most recent call last):
26 File "/srv/revert_risk_model/python/events.py", line 228, in send_event
27 async with aio_http_client.post(
28 File "/opt/lib/venv/lib/python3.11/site-packages/aiohttp/client.py", line 1510, in __aenter__
29 self._resp: _RetType = await self._coro
30 ^^^^^^^^^^^^^^^^
31 File "/opt/lib/venv/lib/python3.11/site-packages/aiohttp/client.py", line 624, in _request
32 with timer:
33 File "/opt/lib/venv/lib/python3.11/site-packages/aiohttp/helpers.py", line 713, in __exit__
34 raise asyncio.TimeoutError from exc_val
35TimeoutError
36
37During handling of the above exception, another exception occurred:
38
39Traceback (most recent call last):
40 File "/opt/lib/venv/lib/python3.11/site-packages/uvicorn/protocols/http/httptools_impl.py", line 401, in run_asgi
41 result = await app( # type: ignore[func-returns-value]
42 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
43 File "/opt/lib/venv/lib/python3.11/site-packages/uvicorn/middleware/proxy_headers.py", line 70, in __call__
44 return await self.app(scope, receive, send)
45 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
46 File "/opt/lib/venv/lib/python3.11/site-packages/fastapi/applications.py", line 1159, in __call__
47 await super().__call__(scope, receive, send)
48 File "/opt/lib/venv/lib/python3.11/site-packages/starlette/applications.py", line 90, in __call__
49 await self.middleware_stack(scope, receive, send)
50 File "/opt/lib/venv/lib/python3.11/site-packages/starlette/middleware/errors.py", line 186, in __call__
51 raise exc
52 File "/opt/lib/venv/lib/python3.11/site-packages/starlette/middleware/errors.py", line 164, in __call__
53 await self.app(scope, receive, _send)
54 File "/opt/lib/venv/lib/python3.11/site-packages/timing_asgi/middleware.py", line 70, in __call__
55 await self.app(scope, receive, send_wrapper)
56 File "/opt/lib/venv/lib/python3.11/site-packages/starlette/middleware/exceptions.py", line 63, in __call__
57 await wrap_app_handling_exceptions(self.app, conn)(scope, receive, send)
58 File "/opt/lib/venv/lib/python3.11/site-packages/starlette/_exception_handler.py", line 53, in wrapped_app
59 raise exc
60 File "/opt/lib/venv/lib/python3.11/site-packages/starlette/_exception_handler.py", line 42, in wrapped_app
61 await app(scope, receive, sender)
62 File "/opt/lib/venv/lib/python3.11/site-packages/fastapi/middleware/asyncexitstack.py", line 18, in __call__
63 await self.app(scope, receive, send)
64 File "/opt/lib/venv/lib/python3.11/site-packages/starlette/routing.py", line 660, in __call__
65 await self.middleware_stack(scope, receive, send)
66 File "/opt/lib/venv/lib/python3.11/site-packages/starlette/routing.py", line 680, in app
67 await route.handle(scope, receive, send)
68 File "/opt/lib/venv/lib/python3.11/site-packages/starlette/routing.py", line 276, in handle
69 await self.app(scope, receive, send)
70 File "/opt/lib/venv/lib/python3.11/site-packages/fastapi/routing.py", line 134, in app
71 await wrap_app_handling_exceptions(app, request)(scope, receive, send)
72 File "/opt/lib/venv/lib/python3.11/site-packages/starlette/_exception_handler.py", line 53, in wrapped_app
73 raise exc
74 File "/opt/lib/venv/lib/python3.11/site-packages/starlette/_exception_handler.py", line 42, in wrapped_app
75 await app(scope, receive, sender)
76 File "/opt/lib/venv/lib/python3.11/site-packages/fastapi/routing.py", line 120, in app
77 response = await f(request)
78 ^^^^^^^^^^^^^^^^
79 File "/opt/lib/venv/lib/python3.11/site-packages/fastapi/routing.py", line 674, in app
80 raw_response = await run_endpoint_function(
81 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
82 File "/opt/lib/venv/lib/python3.11/site-packages/fastapi/routing.py", line 328, in run_endpoint_function
83 return await dependant.call(**values)
84 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
85 File "/opt/lib/venv/lib/python3.11/site-packages/kserve/protocol/rest/v1_endpoints.py", line 84, in predict
86 response, response_headers = await self.dataplane.infer(
87 ^^^^^^^^^^^^^^^^^^^^^^^^^^^
88 File "/opt/lib/venv/lib/python3.11/site-packages/kserve/protocol/dataplane.py", line 461, in infer
89 response, res_headers = await model(request, headers=headers)
90 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
91 File "/opt/lib/venv/lib/python3.11/site-packages/kserve/model.py", line 285, in __call__
92 (await self.predict(payload, headers))
93 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
94 File "/srv/revert_risk_model/model_server/gpu_model.py", line 126, in predict
95 await self.send_event(
96 File "/srv/revert_risk_model/model_server/gpu_model.py", line 158, in send_event
97 await events.send_event(
98 File "/srv/revert_risk_model/python/events.py", line 268, in send_event
99 raise RuntimeError(
100RuntimeError: Unexpected error happened while the event was posted to EventGate, there is the possibility that it never reached it. Please contact the ML team if the issue persists.
101ERROR:root:Connection error while sending an event to EventGate: Cannot connect to host eventgate-main.discovery.wmnet:4480 ssl:<ssl.SSLContext object at 0x7f8c89d934a0> [Temporary failure in name resolution]
1022026-05-04 09:58:12.816 1 kserve ERROR [errors.py:generic_exception_handler():143] Exception:
103Traceback (most recent call last):
104 File "/opt/lib/venv/lib/python3.11/site-packages/aiohttp/connector.py", line 1562, in _create_direct_connection
105 hosts = await self._resolve_host(host, port, traces=traces)
106 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
107 File "/opt/lib/venv/lib/python3.11/site-packages/aiohttp/connector.py", line 1153, in _resolve_host
108 await future
109 File "/opt/lib/venv/lib/python3.11/site-packages/aiohttp/connector.py", line 1209, in _resolve_host_with_throttle
110 addrs = await self._resolver.resolve(host, port, family=self._family)
111 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
112 File "/opt/lib/venv/lib/python3.11/site-packages/aiohttp/resolver.py", line 40, in resolve
113 infos = await self._loop.getaddrinfo(
114 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
115 File "/usr/lib/python3.11/asyncio/base_events.py", line 867, in getaddrinfo
116 return await self.run_in_executor(
117 ^^^^^^^^^^^^^^^^^^^^^^^^^^^
118 File "/usr/lib/python3.11/concurrent/futures/thread.py", line 58, in run
119 result = self.fn(*self.args, **self.kwargs)
120 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
121 File "/usr/lib/python3.11/socket.py", line 962, in getaddrinfo
122 for res in _socket.getaddrinfo(host, port, family, type, proto, flags):
123 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
124socket.gaierror: [Errno -3] Temporary failure in name resolution
125
126The above exception was the direct cause of the following exception:
127
128Traceback (most recent call last):
129 File "/srv/revert_risk_model/python/events.py", line 228, in send_event
130 async with aio_http_client.post(
131 File "/opt/lib/venv/lib/python3.11/site-packages/aiohttp/client.py", line 1510, in __aenter__
132 self._resp: _RetType = await self._coro
133 ^^^^^^^^^^^^^^^^
134 File "/opt/lib/venv/lib/python3.11/site-packages/aiohttp/client.py", line 779, in _request
135 resp = await handler(req)
136 ^^^^^^^^^^^^^^^^^^
137 File "/opt/lib/venv/lib/python3.11/site-packages/aiohttp/client.py", line 734, in _connect_and_send_request
138 conn = await self._connector.connect(
139 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
140 File "/opt/lib/venv/lib/python3.11/site-packages/aiohttp/connector.py", line 672, in connect
141 proto = await self._create_connection(req, traces, timeout)
142 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
143 File "/opt/lib/venv/lib/python3.11/site-packages/aiohttp/connector.py", line 1239, in _create_connection
144 _, proto = await self._create_direct_connection(req, traces, timeout)
145 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
146 File "/opt/lib/venv/lib/python3.11/site-packages/aiohttp/connector.py", line 1568, in _create_direct_connection
147 raise ClientConnectorDNSError(req.connection_key, exc) from exc
148aiohttp.client_exceptions.ClientConnectorDNSError: Cannot connect to host eventgate-main.discovery.wmnet:4480 ssl:<ssl.SSLContext object at 0x7f8c89d934a0> [Temporary failure in name resolution]
149
150During handling of the above exception, another exception occurred:
151
152Traceback (most recent call last):
153 File "/opt/lib/venv/lib/python3.11/site-packages/starlette/middleware/errors.py", line 164, in __call__
154 await self.app(scope, receive, _send)
155 File "/opt/lib/venv/lib/python3.11/site-packages/timing_asgi/middleware.py", line 70, in __call__
156 await self.app(scope, receive, send_wrapper)
157 File "/opt/lib/venv/lib/python3.11/site-packages/starlette/middleware/exceptions.py", line 63, in __call__
158 await wrap_app_handling_exceptions(self.app, conn)(scope, receive, send)
159 File "/opt/lib/venv/lib/python3.11/site-packages/starlette/_exception_handler.py", line 53, in wrapped_app
160 raise exc
161 File "/opt/lib/venv/lib/python3.11/site-packages/starlette/_exception_handler.py", line 42, in wrapped_app
162 await app(scope, receive, sender)
163 File "/opt/lib/venv/lib/python3.11/site-packages/fastapi/middleware/asyncexitstack.py", line 18, in __call__
164 await self.app(scope, receive, send)
165 File "/opt/lib/venv/lib/python3.11/site-packages/starlette/routing.py", line 660, in __call__
166 await self.middleware_stack(scope, receive, send)
167 File "/opt/lib/venv/lib/python3.11/site-packages/starlette/routing.py", line 680, in app
168 await route.handle(scope, receive, send)
169 File "/opt/lib/venv/lib/python3.11/site-packages/starlette/routing.py", line 276, in handle
170 await self.app(scope, receive, send)
171 File "/opt/lib/venv/lib/python3.11/site-packages/fastapi/routing.py", line 134, in app
172 await wrap_app_handling_exceptions(app, request)(scope, receive, send)
173 File "/opt/lib/venv/lib/python3.11/site-packages/starlette/_exception_handler.py", line 53, in wrapped_app
174 raise exc
175 File "/opt/lib/venv/lib/python3.11/site-packages/starlette/_exception_handler.py", line 42, in wrapped_app
176 await app(scope, receive, sender)
177 File "/opt/lib/venv/lib/python3.11/site-packages/fastapi/routing.py", line 120, in app
178 response = await f(request)
179 ^^^^^^^^^^^^^^^^
180 File "/opt/lib/venv/lib/python3.11/site-packages/fastapi/routing.py", line 674, in app
181 raw_response = await run_endpoint_function(
182 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
183 File "/opt/lib/venv/lib/python3.11/site-packages/fastapi/routing.py", line 328, in run_endpoint_function
184 return await dependant.call(**values)
185 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
186 File "/opt/lib/venv/lib/python3.11/site-packages/kserve/protocol/rest/v1_endpoints.py", line 84, in predict
187 response, response_headers = await self.dataplane.infer(
188 ^^^^^^^^^^^^^^^^^^^^^^^^^^^
189 File "/opt/lib/venv/lib/python3.11/site-packages/kserve/protocol/dataplane.py", line 461, in infer
190 response, res_headers = await model(request, headers=headers)
191 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
192 File "/opt/lib/venv/lib/python3.11/site-packages/kserve/model.py", line 285, in __call__
193 (await self.predict(payload, headers))
194 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
195 File "/srv/revert_risk_model/model_server/gpu_model.py", line 126, in predict
196 await self.send_event(
197 File "/srv/revert_risk_model/model_server/gpu_model.py", line 158, in send_event
198 await events.send_event(
199 File "/srv/revert_risk_model/python/events.py", line 260, in send_event
200 raise RuntimeError(
201RuntimeError: Connection error while trying to post the event to EventGate. Please contact the ML team if the issue persists.
2022026-05-04 09:58:12.817 1 uvicorn.error ERROR: Exception in ASGI application
203Traceback (most recent call last):
204 File "/opt/lib/venv/lib/python3.11/site-packages/aiohttp/connector.py", line 1562, in _create_direct_connection
205 hosts = await self._resolve_host(host, port, traces=traces)
206 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
207 File "/opt/lib/venv/lib/python3.11/site-packages/aiohttp/connector.py", line 1153, in _resolve_host
208 await future
209 File "/opt/lib/venv/lib/python3.11/site-packages/aiohttp/connector.py", line 1209, in _resolve_host_with_throttle
210 addrs = await self._resolver.resolve(host, port, family=self._family)
211 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
212 File "/opt/lib/venv/lib/python3.11/site-packages/aiohttp/resolver.py", line 40, in resolve
213 infos = await self._loop.getaddrinfo(
214 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
215 File "/usr/lib/python3.11/asyncio/base_events.py", line 867, in getaddrinfo
216 return await self.run_in_executor(
217 ^^^^^^^^^^^^^^^^^^^^^^^^^^^
218 File "/usr/lib/python3.11/concurrent/futures/thread.py", line 58, in run
219 result = self.fn(*self.args, **self.kwargs)
220 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
221 File "/usr/lib/python3.11/socket.py", line 962, in getaddrinfo
222 for res in _socket.getaddrinfo(host, port, family, type, proto, flags):
223 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
224socket.gaierror: [Errno -3] Temporary failure in name resolution
225
226The above exception was the direct cause of the following exception:
227
228Traceback (most recent call last):
229 File "/srv/revert_risk_model/python/events.py", line 228, in send_event
230 async with aio_http_client.post(
231 File "/opt/lib/venv/lib/python3.11/site-packages/aiohttp/client.py", line 1510, in __aenter__
232 self._resp: _RetType = await self._coro
233 ^^^^^^^^^^^^^^^^
234 File "/opt/lib/venv/lib/python3.11/site-packages/aiohttp/client.py", line 779, in _request
235 resp = await handler(req)
236 ^^^^^^^^^^^^^^^^^^
237 File "/opt/lib/venv/lib/python3.11/site-packages/aiohttp/client.py", line 734, in _connect_and_send_request
238 conn = await self._connector.connect(
239 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
240 File "/opt/lib/venv/lib/python3.11/site-packages/aiohttp/connector.py", line 672, in connect
241 proto = await self._create_connection(req, traces, timeout)
242 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
243 File "/opt/lib/venv/lib/python3.11/site-packages/aiohttp/connector.py", line 1239, in _create_connection
244 _, proto = await self._create_direct_connection(req, traces, timeout)
245 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
246 File "/opt/lib/venv/lib/python3.11/site-packages/aiohttp/connector.py", line 1568, in _create_direct_connection
247 raise ClientConnectorDNSError(req.connection_key, exc) from exc
248aiohttp.client_exceptions.ClientConnectorDNSError: Cannot connect to host eventgate-main.discovery.wmnet:4480 ssl:<ssl.SSLContext object at 0x7f8c89d934a0> [Temporary failure in name resolution]
249
250During handling of the above exception, another exception occurred:
251
252Traceback (most recent call last):
253 File "/opt/lib/venv/lib/python3.11/site-packages/uvicorn/protocols/http/httptools_impl.py", line 401, in run_asgi
254 result = await app( # type: ignore[func-returns-value]
255 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
256 File "/opt/lib/venv/lib/python3.11/site-packages/uvicorn/middleware/proxy_headers.py", line 70, in __call__
257 return await self.app(scope, receive, send)
258 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
259 File "/opt/lib/venv/lib/python3.11/site-packages/fastapi/applications.py", line 1159, in __call__
260 await super().__call__(scope, receive, send)
261 File "/opt/lib/venv/lib/python3.11/site-packages/starlette/applications.py", line 90, in __call__
262 await self.middleware_stack(scope, receive, send)
263 File "/opt/lib/venv/lib/python3.11/site-packages/starlette/middleware/errors.py", line 186, in __call__
264 raise exc
265 File "/opt/lib/venv/lib/python3.11/site-packages/starlette/middleware/errors.py", line 164, in __call__
266 await self.app(scope, receive, _send)
267 File "/opt/lib/venv/lib/python3.11/site-packages/timing_asgi/middleware.py", line 70, in __call__
268 await self.app(scope, receive, send_wrapper)
269 File "/opt/lib/venv/lib/python3.11/site-packages/starlette/middleware/exceptions.py", line 63, in __call__
270 await wrap_app_handling_exceptions(self.app, conn)(scope, receive, send)
271 File "/opt/lib/venv/lib/python3.11/site-packages/starlette/_exception_handler.py", line 53, in wrapped_app
272 raise exc
273 File "/opt/lib/venv/lib/python3.11/site-packages/starlette/_exception_handler.py", line 42, in wrapped_app
274 await app(scope, receive, sender)
275 File "/opt/lib/venv/lib/python3.11/site-packages/fastapi/middleware/asyncexitstack.py", line 18, in __call__
276 await self.app(scope, receive, send)
277 File "/opt/lib/venv/lib/python3.11/site-packages/starlette/routing.py", line 660, in __call__
278 await self.middleware_stack(scope, receive, send)
279 File "/opt/lib/venv/lib/python3.11/site-packages/starlette/routing.py", line 680, in app
280 await route.handle(scope, receive, send)
281 File "/opt/lib/venv/lib/python3.11/site-packages/starlette/routing.py", line 276, in handle
282 await self.app(scope, receive, send)
283 File "/opt/lib/venv/lib/python3.11/site-packages/fastapi/routing.py", line 134, in app
284 await wrap_app_handling_exceptions(app, request)(scope, receive, send)
285 File "/opt/lib/venv/lib/python3.11/site-packages/starlette/_exception_handler.py", line 53, in wrapped_app
286 raise exc
287 File "/opt/lib/venv/lib/python3.11/site-packages/starlette/_exception_handler.py", line 42, in wrapped_app
288 await app(scope, receive, sender)
289 File "/opt/lib/venv/lib/python3.11/site-packages/fastapi/routing.py", line 120, in app
290 response = await f(request)
291 ^^^^^^^^^^^^^^^^
292 File "/opt/lib/venv/lib/python3.11/site-packages/fastapi/routing.py", line 674, in app
293 raw_response = await run_endpoint_function(
294 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
295 File "/opt/lib/venv/lib/python3.11/site-packages/fastapi/routing.py", line 328, in run_endpoint_function
296 return await dependant.call(**values)
297 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
298 File "/opt/lib/venv/lib/python3.11/site-packages/kserve/protocol/rest/v1_endpoints.py", line 84, in predict
299 response, response_headers = await self.dataplane.infer(
300 ^^^^^^^^^^^^^^^^^^^^^^^^^^^
301 File "/opt/lib/venv/lib/python3.11/site-packages/kserve/protocol/dataplane.py", line 461, in infer
302 response, res_headers = await model(request, headers=headers)
303 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
304 File "/opt/lib/venv/lib/python3.11/site-packages/kserve/model.py", line 285, in __call__
305 (await self.predict(payload, headers))
306 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
307 File "/srv/revert_risk_model/model_server/gpu_model.py", line 126, in predict
308 await self.send_event(
309 File "/srv/revert_risk_model/model_server/gpu_model.py", line 158, in send_event
310 await events.send_event(
311 File "/srv/revert_risk_model/python/events.py", line 260, in send_event
312 raise RuntimeError(
313RuntimeError: Connection error while trying to post the event to EventGate. Please contact the ML team if the issue persists.
314ERROR:root:Connection error while sending an event to EventGate: Cannot connect to host eventgate-main.discovery.wmnet:4480 ssl:<ssl.SSLContext object at 0x7f8c4bee3ec0> [Temporary failure in name resolution]
3152026-05-04 09:58:12.819 1 kserve ERROR [errors.py:generic_exception_handler():143] Exception:
316Traceback (most recent call last):
317 File "/opt/lib/venv/lib/python3.11/site-packages/aiohttp/connector.py", line 1562, in _create_direct_connection
318 hosts = await self._resolve_host(host, port, traces=traces)
319 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
320 File "/opt/lib/venv/lib/python3.11/site-packages/aiohttp/connector.py", line 1153, in _resolve_host
321 await future
322 File "/opt/lib/venv/lib/python3.11/site-packages/aiohttp/connector.py", line 1209, in _resolve_host_with_throttle
323 addrs = await self._resolver.resolve(host, port, family=self._family)
324 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
325 File "/opt/lib/venv/lib/python3.11/site-packages/aiohttp/resolver.py", line 40, in resolve
326 infos = await self._loop.getaddrinfo(
327 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
328 File "/usr/lib/python3.11/asyncio/base_events.py", line 867, in getaddrinfo
329 return await self.run_in_executor(
330 ^^^^^^^^^^^^^^^^^^^^^^^^^^^
331 File "/usr/lib/python3.11/concurrent/futures/thread.py", line 58, in run
332 result = self.fn(*self.args, **self.kwargs)
333 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
334 File "/usr/lib/python3.11/socket.py", line 962, in getaddrinfo
335 for res in _socket.getaddrinfo(host, port, family, type, proto, flags):
336 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
337socket.gaierror: [Errno -3] Temporary failure in name resolution
338
339The above exception was the direct cause of the following exception:
340
341Traceback (most recent call last):
342 File "/srv/revert_risk_model/python/events.py", line 228, in send_event
343 async with aio_http_client.post(
344 File "/opt/lib/venv/lib/python3.11/site-packages/aiohttp/client.py", line 1510, in __aenter__
345 self._resp: _RetType = await self._coro
346 ^^^^^^^^^^^^^^^^
347 File "/opt/lib/venv/lib/python3.11/site-packages/aiohttp/client.py", line 779, in _request
348 resp = await handler(req)
349 ^^^^^^^^^^^^^^^^^^
350 File "/opt/lib/venv/lib/python3.11/site-packages/aiohttp/client.py", line 734, in _connect_and_send_request
351 conn = await self._connector.connect(
352 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
353 File "/opt/lib/venv/lib/python3.11/site-packages/aiohttp/connector.py", line 672, in connect
354 proto = await self._create_connection(req, traces, timeout)
355 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
356 File "/opt/lib/venv/lib/python3.11/site-packages/aiohttp/connector.py", line 1239, in _create_connection
357 _, proto = await self._create_direct_connection(req, traces, timeout)
358 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
359 File "/opt/lib/venv/lib/python3.11/site-packages/aiohttp/connector.py", line 1568, in _create_direct_connection
360 raise ClientConnectorDNSError(req.connection_key, exc) from exc
361aiohttp.client_exceptions.ClientConnectorDNSError: Cannot connect to host eventgate-main.discovery.wmnet:4480 ssl:<ssl.SSLContext object at 0x7f8c4bee3ec0> [Temporary failure in name resolution]
362
363During handling of the above exception, another exception occurred:
364
365Traceback (most recent call last):
366 File "/opt/lib/venv/lib/python3.11/site-packages/starlette/middleware/errors.py", line 164, in __call__
367 await self.app(scope, receive, _send)
368 File "/opt/lib/venv/lib/python3.11/site-packages/timing_asgi/middleware.py", line 70, in __call__
369 await self.app(scope, receive, send_wrapper)
370 File "/opt/lib/venv/lib/python3.11/site-packages/starlette/middleware/exceptions.py", line 63, in __call__
371 await wrap_app_handling_exceptions(self.app, conn)(scope, receive, send)
372 File "/opt/lib/venv/lib/python3.11/site-packages/starlette/_exception_handler.py", line 53, in wrapped_app
373 raise exc
374 File "/opt/lib/venv/lib/python3.11/site-packages/starlette/_exception_handler.py", line 42, in wrapped_app
375 await app(scope, receive, sender)
376 File "/opt/lib/venv/lib/python3.11/site-packages/fastapi/middleware/asyncexitstack.py", line 18, in __call__
377 await self.app(scope, receive, send)
378 File "/opt/lib/venv/lib/python3.11/site-packages/starlette/routing.py", line 660, in __call__
379 await self.middleware_stack(scope, receive, send)
380 File "/opt/lib/venv/lib/python3.11/site-packages/starlette/routing.py", line 680, in app
381 await route.handle(scope, receive, send)
382 File "/opt/lib/venv/lib/python3.11/site-packages/starlette/routing.py", line 276, in handle
383 await self.app(scope, receive, send)
384 File "/opt/lib/venv/lib/python3.11/site-packages/fastapi/routing.py", line 134, in app
385 await wrap_app_handling_exceptions(app, request)(scope, receive, send)
386 File "/opt/lib/venv/lib/python3.11/site-packages/starlette/_exception_handler.py", line 53, in wrapped_app
387 raise exc
388 File "/opt/lib/venv/lib/python3.11/site-packages/starlette/_exception_handler.py", line 42, in wrapped_app
389 await app(scope, receive, sender)
390 File "/opt/lib/venv/lib/python3.11/site-packages/fastapi/routing.py", line 120, in app
391 response = await f(request)
392 ^^^^^^^^^^^^^^^^
393 File "/opt/lib/venv/lib/python3.11/site-packages/fastapi/routing.py", line 674, in app
394 raw_response = await run_endpoint_function(
395 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
396 File "/opt/lib/venv/lib/python3.11/site-packages/fastapi/routing.py", line 328, in run_endpoint_function
397 return await dependant.call(**values)
398 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
399 File "/opt/lib/venv/lib/python3.11/site-packages/kserve/protocol/rest/v1_endpoints.py", line 84, in predict
400 response, response_headers = await self.dataplane.infer(
401 ^^^^^^^^^^^^^^^^^^^^^^^^^^^
402 File "/opt/lib/venv/lib/python3.11/site-packages/kserve/protocol/dataplane.py", line 461, in infer
403 response, res_headers = await model(request, headers=headers)
404 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
405 File "/opt/lib/venv/lib/python3.11/site-packages/kserve/model.py", line 285, in __call__
406 (await self.predict(payload, headers))
407 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
408 File "/srv/revert_risk_model/model_server/gpu_model.py", line 126, in predict
409 await self.send_event(
410 File "/srv/revert_risk_model/model_server/gpu_model.py", line 158, in send_event
411 await events.send_event(
412 File "/srv/revert_risk_model/python/events.py", line 260, in send_event
413 raise RuntimeError(
414RuntimeError: Connection error while trying to post the event to EventGate. Please contact the ML team if the issue persists.
4152026-05-04 09:58:12.820 1 uvicorn.error ERROR: Exception in ASGI application
416Traceback (most recent call last):
417 File "/opt/lib/venv/lib/python3.11/site-packages/aiohttp/connector.py", line 1562, in _create_direct_connection
418 hosts = await self._resolve_host(host, port, traces=traces)
419 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
420 File "/opt/lib/venv/lib/python3.11/site-packages/aiohttp/connector.py", line 1153, in _resolve_host
421 await future
422 File "/opt/lib/venv/lib/python3.11/site-packages/aiohttp/connector.py", line 1209, in _resolve_host_with_throttle
423 addrs = await self._resolver.resolve(host, port, family=self._family)
424 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
425 File "/opt/lib/venv/lib/python3.11/site-packages/aiohttp/resolver.py", line 40, in resolve
426 infos = await self._loop.getaddrinfo(
427 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
428 File "/usr/lib/python3.11/asyncio/base_events.py", line 867, in getaddrinfo
429 return await self.run_in_executor(
430 ^^^^^^^^^^^^^^^^^^^^^^^^^^^
431 File "/usr/lib/python3.11/concurrent/futures/thread.py", line 58, in run
432 result = self.fn(*self.args, **self.kwargs)
433 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
434 File "/usr/lib/python3.11/socket.py", line 962, in getaddrinfo
435 for res in _socket.getaddrinfo(host, port, family, type, proto, flags):
436 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
437socket.gaierror: [Errno -3] Temporary failure in name resolution
438
439The above exception was the direct cause of the following exception:
440
441Traceback (most recent call last):
442 File "/srv/revert_risk_model/python/events.py", line 228, in send_event
443 async with aio_http_client.post(
444 File "/opt/lib/venv/lib/python3.11/site-packages/aiohttp/client.py", line 1510, in __aenter__
445 self._resp: _RetType = await self._coro
446 ^^^^^^^^^^^^^^^^
447 File "/opt/lib/venv/lib/python3.11/site-packages/aiohttp/client.py", line 779, in _request
448 resp = await handler(req)
449 ^^^^^^^^^^^^^^^^^^
450 File "/opt/lib/venv/lib/python3.11/site-packages/aiohttp/client.py", line 734, in _connect_and_send_request
451 conn = await self._connector.connect(
452 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
453 File "/opt/lib/venv/lib/python3.11/site-packages/aiohttp/connector.py", line 672, in connect
454 proto = await self._create_connection(req, traces, timeout)
455 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
456 File "/opt/lib/venv/lib/python3.11/site-packages/aiohttp/connector.py", line 1239, in _create_connection
457 _, proto = await self._create_direct_connection(req, traces, timeout)
458 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
459 File "/opt/lib/venv/lib/python3.11/site-packages/aiohttp/connector.py", line 1568, in _create_direct_connection
460 raise ClientConnectorDNSError(req.connection_key, exc) from exc
461aiohttp.client_exceptions.ClientConnectorDNSError: Cannot connect to host eventgate-main.discovery.wmnet:4480 ssl:<ssl.SSLContext object at 0x7f8c4bee3ec0> [Temporary failure in name resolution]
462
463During handling of the above exception, another exception occurred:
464
465Traceback (most recent call last):
466 File "/opt/lib/venv/lib/python3.11/site-packages/uvicorn/protocols/http/httptools_impl.py", line 401, in run_asgi
467 result = await app( # type: ignore[func-returns-value]
468 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
469 File "/opt/lib/venv/lib/python3.11/site-packages/uvicorn/middleware/proxy_headers.py", line 70, in __call__
470 return await self.app(scope, receive, send)
471 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
472 File "/opt/lib/venv/lib/python3.11/site-packages/fastapi/applications.py", line 1159, in __call__
473 await super().__call__(scope, receive, send)
474 File "/opt/lib/venv/lib/python3.11/site-packages/starlette/applications.py", line 90, in __call__
475 await self.middleware_stack(scope, receive, send)
476 File "/opt/lib/venv/lib/python3.11/site-packages/starlette/middleware/errors.py", line 186, in __call__
477 raise exc
478 File "/opt/lib/venv/lib/python3.11/site-packages/starlette/middleware/errors.py", line 164, in __call__
479 await self.app(scope, receive, _send)
480 File "/opt/lib/venv/lib/python3.11/site-packages/timing_asgi/middleware.py", line 70, in __call__
481 await self.app(scope, receive, send_wrapper)
482 File "/opt/lib/venv/lib/python3.11/site-packages/starlette/middleware/exceptions.py", line 63, in __call__
483 await wrap_app_handling_exceptions(self.app, conn)(scope, receive, send)
484 File "/opt/lib/venv/lib/python3.11/site-packages/starlette/_exception_handler.py", line 53, in wrapped_app
485 raise exc
486 File "/opt/lib/venv/lib/python3.11/site-packages/starlette/_exception_handler.py", line 42, in wrapped_app
487 await app(scope, receive, sender)
488 File "/opt/lib/venv/lib/python3.11/site-packages/fastapi/middleware/asyncexitstack.py", line 18, in __call__
489 await self.app(scope, receive, send)
490 File "/opt/lib/venv/lib/python3.11/site-packages/starlette/routing.py", line 660, in __call__
491 await self.middleware_stack(scope, receive, send)
492 File "/opt/lib/venv/lib/python3.11/site-packages/starlette/routing.py", line 680, in app
493 await route.handle(scope, receive, send)
494 File "/opt/lib/venv/lib/python3.11/site-packages/starlette/routing.py", line 276, in handle
495 await self.app(scope, receive, send)
496 File "/opt/lib/venv/lib/python3.11/site-packages/fastapi/routing.py", line 134, in app
497 await wrap_app_handling_exceptions(app, request)(scope, receive, send)
498 File "/opt/lib/venv/lib/python3.11/site-packages/starlette/_exception_handler.py", line 53, in wrapped_app
499 raise exc
500 File "/opt/lib/venv/lib/python3.11/site-packages/starlette/_exception_handler.py", line 42, in wrapped_app
501 await app(scope, receive, sender)
502 File "/opt/lib/venv/lib/python3.11/site-packages/fastapi/routing.py", line 120, in app
503 response = await f(request)
504 ^^^^^^^^^^^^^^^^
505 File "/opt/lib/venv/lib/python3.11/site-packages/fastapi/routing.py", line 674, in app
506 raw_response = await run_endpoint_function(
507 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
508 File "/opt/lib/venv/lib/python3.11/site-packages/fastapi/routing.py", line 328, in run_endpoint_function
509 return await dependant.call(**values)
510 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
511 File "/opt/lib/venv/lib/python3.11/site-packages/kserve/protocol/rest/v1_endpoints.py", line 84, in predict
512 response, response_headers = await self.dataplane.infer(
513 ^^^^^^^^^^^^^^^^^^^^^^^^^^^
514 File "/opt/lib/venv/lib/python3.11/site-packages/kserve/protocol/dataplane.py", line 461, in infer
515 response, res_headers = await model(request, headers=headers)
516 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
517 File "/opt/lib/venv/lib/python3.11/site-packages/kserve/model.py", line 285, in __call__
518 (await self.predict(payload, headers))
519 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
520 File "/srv/revert_risk_model/model_server/gpu_model.py", line 126, in predict
521 await self.send_event(
522 File "/srv/revert_risk_model/model_server/gpu_model.py", line 158, in send_event
523 await events.send_event(
524 File "/srv/revert_risk_model/python/events.py", line 260, in send_event
525 raise RuntimeError(
526RuntimeError: Connection error while trying to post the event to EventGate. Please contact the ML team if the issue persists.

Am I missing anything ? Is this a infra/network issue?

isarantopoulos added a comment.May 4 2026, 11:00 AM

🎉 Indeed there are events in stream.wikimedia.org.
Maybe I'm not of much help but judging from the error above Cannot connect to host eventgate-main.discovery.wmnet:4480 and the fact that we have events in the streams could these errors be transient or handled by a retry mechanism? I'm trying to say that if the errors to eventgate were persistent we wouldn't see any events published at all.

gkyziridis added a comment.EditedMay 4 2026, 11:42 AM
In T415892#11886231, @isarantopoulos wrote:

🎉 Indeed there are events in stream.wikimedia.org.
Maybe I'm not of much help but judging from the error above Cannot connect to host eventgate-main.discovery.wmnet:4480 and the fact that we have events in the streams could these errors be transient or handled by a retry mechanism? I'm trying to say that if the errors to eventgate were persistent we wouldn't see any events published at all.

I am investigating this.

LogStash_Dashboard

Grafana - Istio

Grafana - Pods

isarantopoulos added a comment.May 5 2026, 1:56 PM

since this happens also for revertisk language agnostic iiuc the issue above are related to T346136: Outlink returns 500 when EventGate returns 503 Service Unavailable
I think we can conclude this task and investigate the 5xx separately

isarantopoulos added a comment.May 6 2026, 11:43 AM

@gkyziridis since the above errors are not related to this task we can resolve this task after we persist the predictions in the data lake like we did in T405358: Add LiftWing streams data to event_sanitized (increase data retention)

Maintenance_bot removed a project: Patch-For-Review.May 6 2026, 12:31 PM
gerritbot added a comment.May 6 2026, 12:54 PM

Change #1283749 had a related patch set uploaded (by Gkyziridis; author: Gkyziridis):

[analytics/refinery@master] expand_event_sanitized_analytics_allowlist: Add revertrisk-multilingual predictions to allowlist.

https://gerrit.wikimedia.org/r/1283749

gerritbot added a project: Patch-For-Review.May 6 2026, 12:54 PM
gerritbot added a comment.May 6 2026, 1:08 PM

Change #1283758 had a related patch set uploaded (by Gkyziridis; author: Gkyziridis):

[operations/deployment-charts@master] changeprop: Configure all wikis for revertrisk-multilingual events.

https://gerrit.wikimedia.org/r/1283758

gkyziridis added a comment.May 6 2026, 1:41 PM

We can close this task after the patchers are merged and deployed.

gkyziridis added a comment.May 11 2026, 9:50 AM

I +1 the patch event-sanitisation: https://gerrit.wikimedia.org/r/c/analytics/refinery/+/1283749
I think DE needs to deploy? I do not have permissions for +2 in the patch.

gerritbot added a comment.May 11 2026, 12:48 PM

Change #1283749 merged by Ottomata:

[analytics/refinery@master] expand_event_sanitized_analytics_allowlist: Add revertrisk-multilingual predictions to allowlist.

https://gerrit.wikimedia.org/r/1283749

Ottomata added a comment.May 11 2026, 12:50 PM

Merged and added to https://wikitech.wikimedia.org/wiki/Data_Platform_Engineering/Ops_week/Analytics_weekly_train#NEXT_TRAIN

gkyziridis added a comment.May 11 2026, 2:15 PM

Thank you very much @Ottomata !

gerritbot added a comment.May 13 2026, 7:27 AM

Change #1283758 merged by jenkins-bot:

[operations/deployment-charts@master] changeprop: Configure all wikis for revertrisk-multilingual events.

https://gerrit.wikimedia.org/r/1283758

Maintenance_bot removed a project: Patch-For-Review.May 13 2026, 7:30 AM
gkyziridis added a comment.May 13 2026, 7:46 AM

Configuration of 46 wikis + testwiki on changeprop is merged and deployed.
We can see streams coming from multiple wikis in https://stream.wikimedia.org/v2/ui/#/?streams=mediawiki.page_revert_risk_multilingual_prediction_change.v1.

gkyziridis added a comment.EditedMay 18 2026, 12:16 PM

Hey @Ottomata,

I cannot see results in`event_sanitized.mediawiki_page_revert_risk_multilingual_prediction_change_v1`:

event_sanitized query (3,408×926 px, 201 KB)

The mediawiki.page_revert_risk_multilingual_prediction_change.v1 is configured in: https://meta.wikimedia.org/w/api.php?action=streamconfigs

"mediawiki.page_revert_risk_multilingual_prediction_change.v1": {
      "topic_prefixes": [
        "eqiad.",
        "codfw."
      ],
      "canary_events_enabled": true,
      "producers": {
        "eventgate": {
          "enrich_fields_from_http_headers": {
            "meta.request_id": "x-request-id",
            "http.request_headers.user-agent": false
          }
        }
      },
      "consumers": {
        "analytics_hadoop_ingestion": {
          "job_name": "event_default",
          "enabled": true
        },
        "analytics_hive_ingestion": {
          "enabled": true
        }
      },
      "schema_title": "mediawiki/page/prediction_classification_change",
      "destination_event_service": "eventgate-main",
      "stream": "mediawiki.page_revert_risk_multilingual_prediction_change.v1",
      "topics": [
        "eqiad.mediawiki.page_revert_risk_multilingual_prediction_change.v1",
        "codfw.mediawiki.page_revert_risk_multilingual_prediction_change.v1"
      ]
    },

Do I miss anything?
Should update the airflow-dags ?

gkyziridis added a comment.May 19 2026, 3:40 PM

Update

I can query event_sanitized.mediawiki_page_revert_risk_multilingual_prediction_change_v1 and see results.
Thank you very much for your help @Ottomata !

mediawiki_page_revert_risk_multilingual_prediction_change_v1 (3,430×1,884 px, 395 KB)

We can close this task.

Ottomata added a comment.May 19 2026, 4:06 PM

Great!

Samwalton9-WMF added a comment.May 20 2026, 10:29 AM

Thanks folks!! This will make our analysis of this model much easier. Appreciate you spending the time on this.

gkyziridis closed this task as Resolved.May 20 2026, 10:58 AM
gkyziridis moved this task from Q1 FY2026-27 to 2025-2026 Q2 Done on the Machine-Learning-Team board.
gkyziridis edited projects, added: Machine-Learning-Team; removed: Machine-Learning-Team (Q1 FY2026-27).
isarantopoulos edited projects, added: Machine-Learning-Team (Q1 FY2026-27); removed: Machine-Learning-Team.May 20 2026, 11:07 AM
isarantopoulos moved this task from In Progress to Done on the Machine-Learning-Team (Q1 FY2026-27) board.
isarantopoulos moved this task from Q1 FY2026-27 to Task Archive on the Machine-Learning-Team board.Jul 6 2026, 7:49 AM
isarantopoulos edited projects, added: Machine-Learning-Team; removed: Machine-Learning-Team (Q1 FY2026-27).
isarantopoulos moved this task to Archived on the Machine-Learning-Team board.Jul 6 2026, 11:57 AM
Content licensed under Creative Commons Attribution-ShareAlike (CC BY-SA) 4.0 unless otherwise noted; code licensed under GNU General Public License (GPL) 2.0 or later and other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL · Credits