We also need to repeat T351057 for this model - shall I make that a separate ticket?

@Samwalton9-WMF yes, please create a separate ticket for that.

I would also like to tag @Pablo @diego: do we have regular snapshots of revert risk scores based on the multilingual model as well, or even a single snapshot for a few months?

Something similar to risk_observatory.revert_risk_predictions or one-off snapshot like /user/paragon/riskobservatory/revertrisk_20212022_anonymous_bot.parquet that I previously used.

I would also like to tag @Pablo @diego: do we have regular snapshots of revert risk scores based on the multilingual model as well, or even a single snapshot for a few months?

I don't know, maybe @fkaelin knows.

There is no pre computed dataset available. The implementation is general, e.g. by passing a multi-lingual model url it could create predictions for that model. See the risk observatory pipeline code as an example that uses this transformation end-to-end; this can be run via a notebook (pip install repo and import/execute the run method). To create a pipeline that generates predictions for the multi-lingual model via airflow dag, we would need to create research engineering request.

@KCVelaga_WMF I misread this code previously - for now the model loading/inference for the various variants of the revert risk model is not unified (we plan to do that though). There are separate loading and classify methods for the multi-lingual model. The features extraction pipeline for all models is generalized, but the inference step needs some more work. I started updating this notebook to work with the multi-lingual model but run into some torch/transfomer version issues.

@Samwalton9-WMF The predictions are available on this spreadsheet for testing. Note: for this task and reverts per day, I had to limit to the first week of August, as I ran into memory issues.

In T372747#10161126, @KCVelaga_WMF wrote:

@Samwalton9-WMF The predictions are available on this spreadsheet for testing. Note: for this task and reverts per day, I had to limit to the first week of August, as I ran into memory issues.

Thank you! I ran through ~200 edits at revert risk scores between 0.9 and 1 on enwiki. My unscientific accuracy results around different scores, with cases where I wasn't sure conservatively counted as false positives:

0.9: 38%
0.93: 56%
0.95: 75%
0.96: 84%
0.97: 80%
0.98: 81%
0.99: 91%
0.999: 100%

My initial ballpark thresholds to test, then, would be something like buckets of 0.1 starting at 0.95. @OTichonova is doing a similar runthrough for de.wiki and ru.wiki.

My initial ballpark thresholds to test, then, would be something like buckets of 0.1 starting at 0.95. @OTichonova is doing a similar runthrough for de.wiki and ru.wiki.

An important note from @diego about accuracy for the multilingual model - it varies substantially between different languages, and therefore we're unlikely to be able to pick one set of thresholds for all wikis, unlike with the language-agnostic model. This poses some challenges for our testing process and UX. We may need to let each community set a very specific (i.e. number) threshold, rather than giving them predetermined buckets.

See teal columns in top graph, or bottom-left plot: