Below are the thresholds that provide 1% FPR for each wiki:

|Wiki| Revisions| Start Date| End Date| AUC| ⭐ Threshold |Final FPR
|------------|-------------|---------------|------------|-------------|---------------|---------------|
|enwiki |159,580 |2026-02-01| 2026-02-01 |0.768| 0.8955| 0.01
| frwiki |109,134| 2026-02-01| 2026-02-05 |0.763 |0.8388| 0.01|
| ruwiki| 104,049| 2026-02-01| 2026-02-07| 0.74| 0.8884| 0.01
| ptwiki |70,798| 2026-01-25 |2026-02-07| 0.66| 0.8466 |0.01
| fawiki| 91,317 |2026-01-25| 2026-02-07| 0.569 |0.8869| 0.01
| simplewiki| 11,908| 2026-01-25| 2026-02-07 |0.708| 0.9587| 0.01
| rowiki| 37,999| 2026-01-25| 2026-02-07 |0.781| 0.8178| 0.01
| bgwiki |10,880 |2026-01-25| 2026-02-07| 0.892| 0.9535| 0.01

For more detailed results and the data collection process, please refer to here.

Also, based on the AUC scores, low AUC scores (<0.7) indicate that the model is less sensitive in fawiki and ptwiki.

@cchen can you explain how does the 0.8178 threshold on rowiki matches against the 1.7% error rate @ 0.95 on rowiki reported in [[T374698#11614851]]? Did you use a different definition of false positive?

I wonder if the fact that revisions are reverted based on the same model you're evaluating might skew the results.

@Strainu the data processing here follows the previous thresholds analysis.

After using the same criteria as in T374698#11614851

• remove bot edits
• remove article creations
• main namespace only

The updated thresholds are:

For more detailed results and the data collection process, please refer to here.

I am wondering whether we should add back the edits that are reverts here.

I think you need significantly more context here so your results and the human perception (which will prevail on deciding how good Automoderator is) are aligned. Reading the code, I see you exclude Automoderator reverts, but I can't find anything tracking the other bots, especially the ones using RR scores. Have I missed anything?

Why this is important: In T374698#11614851, false positives where defined as bot edits being reverted by another user. If you consider false positives as "RR scores > threshold but edit not reverted", you actually include a subset of the FP above (those with RR score > threshold) in the TP bucket just because they where reverted.