Update

Since the task: T406217 is finished we have a first version of end-to-end pipeline including all the basic steps of an ML-Lifecycle: Data Generation -> Model Training -> Export model in S3 bucket.
More info could be found here: https://phabricator.wikimedia.org/T398970

Generate Data (SparkSubmitOperator) -> Train/Validation/Test split (SparkSubmitOperator) -> Copy from HDFS to a PVC (WMFKubernetesPodOperator) -> Train model on GPU pod (WMFKubernetesPodOperator) -> Copy retrained model to S3 (PythonOperator)

Hey, I am working on this, I think that I have finished the implementation for publishing the predictions in events. I am now testing it locally.
Based on this: https://wikitech.wikimedia.org/wiki/Machine_Learning/LiftWing/Streams I think there are these steps:

1. Implementation on inference-services side (this is what I am testing).
2. Test it and deploy the new model server versions.
3. Configure Changeprop.
4. Configure the new changes in the mediawiki-config repo.

Hey @Isaac, this ticket is assigned to @klausman but he is currently on his sabbatical. He will start working on this when he is back, I think around next month (???).
I am tagging @DPogorzelski-WMF here for visibility, maybe he has something more to add.

Update

The end-to-end tone-check retraining pipeline succeeded, we solved the issues of Multy-Attach PVC.

$ s3cmd -c /etc/s3cmd/cfg.d/ml-team.cfg ls -H s3://wmf-ml-models/retrained-models/tone-check/checkpoint-21530/
2026-01-28 22:24   865   s3://wmf-ml-models/retrained-models/tone-check/checkpoint-21530/config.json
2026-01-28 22:24   678M  s3://wmf-ml-models/retrained-models/tone-check/checkpoint-21530/model.safetensors
2026-01-28 22:24  1357M  s3://wmf-ml-models/retrained-models/tone-check/checkpoint-21530/optimizer.pt
2026-01-28 22:24    13K  s3://wmf-ml-models/retrained-models/tone-check/checkpoint-21530/rng_state.pth
2026-01-28 22:24  1064   s3://wmf-ml-models/retrained-models/tone-check/checkpoint-21530/scheduler.pt
2026-01-28 22:24   695   s3://wmf-ml-models/retrained-models/tone-check/checkpoint-21530/special_tokens_map.json
2026-01-28 22:24     2M  s3://wmf-ml-models/retrained-models/tone-check/checkpoint-21530/tokenizer.json
2026-01-28 22:24  1330   s3://wmf-ml-models/retrained-models/tone-check/checkpoint-21530/tokenizer_config.json
2026-01-28 22:24     9K  s3://wmf-ml-models/retrained-models/tone-check/checkpoint-21530/trainer_state.json
2026-01-28 22:24     5K  s3://wmf-ml-models/retrained-models/tone-check/checkpoint-21530/training_args.bin
2026-01-28 22:24   972K  s3://wmf-ml-models/retrained-models/tone-check/checkpoint-21530/vocab.txt

We are currently do not store anywhere the predictions from the rr-multilingual model so we cannot export them in the same way that we are doing for the rr-language-agnostic one.
If there is this necessity, I can open a new Phabricator task in order to start developing the first step of saving the slice of the rr-multilingual predictions into the event stream, and then we can add them to the refinery and export them into the event_sanitized as we do for the rr-langugage-agnostic.

In T405358#11557401, @kostajh wrote:

@gkyziridis I'm testing this out today but only seeing revertrisk-language-agnostic for an example revision on enwiki, is that expected?

spark-sql (default)> select predicted_classification from event.mediawiki_page_revert_risk_prediction_change_v1 where revision.rev_id = 1333904928;
predicted_classification
{"model_name":"revertrisk-language-agnostic","model_version":"3","predictions":["false"],"probabilities":{"false":0.7348057627677917,"true":0.26519423723220825}}

I also checked the PVC using kubectl and I see that the PVC is "RWO": "ReadWriteOnce" I am not sure if this makes the problem:

$ kube_env airflow-ml-deploy dse-k8s-eqiad
$ kubectl get pvc airflow-ml-model-training -n airflow-dev
NAME                        STATUS   VOLUME                                     CAPACITY   ACCESS MODES   STORAGECLASS   AGE
airflow-ml-model-training   Bound    pvc-8a6a2920-8d7e-4616-8ab6-a6a70b26d116   20Gi       RWO            ceph-rbd-ssd   151d

In T406179#11510890, @kevinbazira wrote:

Weekly Update:

• The Wikimedia Enterprise team conducted load tests to simulate their traffic and shared results in T409388#11483570
• We are working on optimizing the revertrisk-wikidata inference service to achieve the Enterprise team's latency target in T414060

Things we need to keep in mind:

• Testwiki is not a canonical/normal wiki so it is excluded from canonical_wikis list
• Testwiki is not a supported wiki for the revertrisk model, so predictions will be completely inaccurate.
• We treat testwiki as enwiki on the fly in order for the revert-risk model server to accept such API hits posting {"lang"="test"}

The testwiki is not a canonical Wikipedia, it is a testing environment where articles can be written in any language, and it wasn’t part of the RR model’s training data, so we excluded from the list of canonical Wikipedias. So the, RR model doesn’t support testwiki.
We can easily parse this in the following two requests to enwiki and testwiki, respectively:

In T410663#11447291, @Isaac wrote:

Nevertheless, this combination of versions seems to fix the issue using the GPU image, so your curiosity is in a super good shape towards the correct direction :P.

Haha, always happy to be accidentally helpful :) Once it's deployed on ml-lab1002, happy to test but definitely looking promising!

In T410663#11444307, @Isaac wrote:

Thanks @gkyziridis for digging into this! Out of curiosity, why not jump to the current stable versions (2.9.1 for torch and 6.4 for AMD)? I see you commented that line out in the initial file that at least had torch at 2.9.1.

I built the image using: docker build --network=host -t torch_rocm3 .

In T409438#11428348, @Kgraessle wrote:

Model configuration and threshold configuration have been deployed.
The next step is to backport adding thwiki to the ORES dblist: https://gerrit.wikimedia.org/r/c/operations/mediawiki-config/+/1207923.
Let's discuss in engineering weekly when we would like to go ahead and do that along with the backfill script.

When we start the actual deployment:
Due to the fact that we have a huge number of wikis which are needed to be deployed, I suggest to to do it in batches. Right now, in the patch above only the thresholds are set for each wiki, that means that if this patch is merged and deployed nothing will be changed. In the next iterations, when we start to deploying the wikis we need to enable ORES model and enable the UI as well. Only then the thresholds which are configured in the patch will be functional. So, I suggest to enable ORES model in batches e.g. for 4-5 wikis per batch. This will take some time to finish all batches, but we can easily handle issues that could occur during the backport deployments

Update

I configure all the rr thresholds for all the wikis and enabled the model for all of them in this patch: https://gerrit.wikimedia.org/r/c/operations/mediawiki-config/+/1212086 .
I excluded thwiki from the above patch since you are using it for the MVP.
I also avoided to run the composer manage-dblist add {wiki_name} ores for all the wikis, which means that whenever we deploy all these wikis we need to run the composer for all of them.

I think that there is one more step which needs to be done which is to run: composer manage-dblist add {wiki_name} ores. I do not see thwiki to be added under "dblists/ores.dblist" file in this patch -> https://gerrit.wikimedia.org/r/c/operations/mediawiki-config/+/1207932

In T409388#11405117, @kevinbazira wrote:

The revertrisk-wikidata inference service production endpoint uses similar scaling configs that other revertrisk inference-services use: https://github.com/wikimedia/operations-deployment-charts/blob/8412fc655d3b1e10b38cf0c954d910b820e93a05/helmfile.d/ml-services/revertrisk/values.yaml#L145-L150

IMO the prod endpoint should scale well unless results from the WME folks say otherwise.

Progress update on the hypothesis for the week, including if something has shipped:

Hey @kevinbazira thank very much for running the loading tests for Revert-Risk wikidata.
I think we should change a little bit the configuration in order to simulate a more realistic scenario close to reality.
We also need to run heavier tests spawning more users in order to check our API's capacity and capability to handle maximum RPS.
I ran three different locust tests with heavier configuration, you can see the results in the following phab paste:

For the dewiki we had spotted an issue which is described here: T407155#11311194 regarding many english samples used for training the model in dewiki. In order to overcome this, I used translation only where the english samples exists inside the dewiki dataset.

You can try tweaking the filters in the notebook, such as loosening the diff size conditions, expanding the revert time periods, or asking the community for more signals if possible.

Update

The issue I am facing for reproducing the error is that we are logging the incoming request if it is successful (status code 200), but we do not log it if is not.
We need somehow to log it immediately after we receive it in order to reproduce it.
I will open a ticket for upgrading the logging on the model server side: https://phabricator.wikimedia.org/T409931

In T409657#11358844, @elukey wrote:

@gkyziridis I am not 100% sure if the rev-id in the task's description is the problematic one, I thought it was when checking the logs but you may need to review /home/elukey/T409657 on deploy2002 to get other testing samples :(

Optimal Thresholds Table