To understand how impactful this model would be if implemented in a revert bot I'd really appreciate some data on the following:
How many edits would be reverted if it was configured for 95% and 99% accuracy on: en.wiki, es.wiki, and uk.wiki?
Description
Description
| Status | Subtype | Assigned | Task | ||
|---|---|---|---|---|---|
| Resolved | diego | T314384 Develop a ML-based service to predict reverts on Wikipedia(s) | |||
| Resolved | diego | T336421 Revert prediction data request |
Event Timeline
Comment Actions
@diego per our chat about this model this would be a really helpful metric for me to understand. Have I phrased the question in a way that's helpful? And are 95% and 99% accuracy realistic figures to be testing for?
Comment Actions
@Pablo do you have these numbers on the Risk Observatory? (If not, I would need until Monday to compute this)
Comment Actions
Yes, with a test dataset of all revisions from January to October 2022 I evaluated the model for 100 thresholds (0.00, 0.01, 0.02, ..., 1.00) on several wikis (unfortunately not ukwiki, I would need to compute that one).
Results
 |  |
https://gitlab.wikimedia.org/paragon/knowledge-integrity-risk-index/-/blob/main/data/reverts/enwiki.csv
https://gitlab.wikimedia.org/paragon/knowledge-integrity-risk-index/-/blob/main/data/reverts/eswiki.csv
Results distinguishing IP edits and user edits
 |  | |
https://gitlab.wikimedia.org/paragon/knowledge-integrity-risk-index/-/blob/main/data/reverts_anonymous/enwiki.csv
https://gitlab.wikimedia.org/paragon/knowledge-integrity-risk-index/-/blob/main/data/reverts_anonymous/eswiki.csv
Risk Observatory
The new version of the Risk Observatory contains metrics based on the notion of high-risk revision: a revision with a revert risk score greater than or equal to the threshold that maximises accuracy for that wiki in the test dataset (filtering out IP and bot edits).
Comment Actions
@Pablo , please can you provide the absolute numbers for the 3 wikis that @Samwalton9 is asking for? let's consider January'22
Comment Actions
Sure, these are the numbers for enwiki and eswiki in January 22 with different thresholds:
risk score ≥ 0.0
| wiki_db | reverted | count | ratio |
| enwiki | false | 2148599 | 0.882 |
| enwiki | true | 288768 | 0.118 |
| eswiki | false | 299485 | 0.818 |
| eswiki | true | 66769 | 0.182 |
risk score ≥ 0.9
| wiki_db | reverted | count | ratio |
| enwiki | false | 128723 | 0.55 |
| enwiki | true | 105494 | 0.45 |
| eswiki | false | 20290 | 0.398 |
| eswiki | true | 30705 | 0.602 |
risk score ≥ 0.91
| wiki_db | reverted | count | ratio |
| enwiki | false | 109162 | 0.533 |
| enwiki | true | 95759 | 0.467 |
| eswiki | false | 17297 | 0.38 |
| eswiki | true | 28209 | 0.62 |
risk score ≥ 0.92
| wiki_db | reverted | count | ratio |
| enwiki | false | 89724 | 0.513 |
| enwiki | true | 85174 | 0.487 |
| eswiki | false | 14261 | 0.361 |
| eswiki | true | 25289 | 0.639 |
risk score ≥ 0.93
| wiki_db | reverted | count | ratio |
| enwiki | false | 71160 | 0.49 |
| enwiki | true | 73995 | 0.51 |
| eswiki | false | 11271 | 0.336 |
| eswiki | true | 22271 | 0.664 |
risk score ≥ 0.94
| wiki_db | reverted | count | ratio |
| enwiki | false | 54180 | 0.464 |
| enwiki | true | 62475 | 0.536 |
| eswiki | false | 8459 | 0.308 |
| eswiki | true | 18982 | 0.692 |
risk score ≥ 0.95
| wiki_db | reverted | count | ratio |
| enwiki | false | 39062 | 0.435 |
| enwiki | true | 50759 | 0.565 |
| eswiki | false | 5980 | 0.277 |
| eswiki | true | 15633 | 0.723 |
risk score ≥ 0.96
| wiki_db | reverted | count | ratio |
| enwiki | false | 25745 | 0.397 |
| enwiki | true | 39131 | 0.603 |
| eswiki | false | 3741 | 0.237 |
| eswiki | true | 12055 | 0.763 |
risk score ≥ 0.97
| wiki_db | reverted | count | ratio |
| enwiki | false | 13753 | 0.34 |
| enwiki | true | 26688 | 0.66 |
| eswiki | false | 1875 | 0.187 |
| eswiki | true | 8173 | 0.813 |
risk score ≥ 0.98
| wiki_db | reverted | count | ratio |
| enwiki | false | 4476 | 0.249 |
| enwiki | true | 13491 | 0.751 |
| eswiki | false | 571 | 0.119 |
| eswiki | true | 4244 | 0.881 |
risk score ≥ 0.99
| wiki_db | reverted | count | ratio |
| enwiki | false | 410 | 0.112 |
| enwiki | true | 3245 | 0.888 |
| eswiki | false | 63 | 0.049 |
| eswiki | true | 1235 | 0.951 |
Note: Results were obtained with the language-agnostic model, not the multilingual one. Revisions, that include IP edits and bot edits, were retrieved with this script https://gitlab.wikimedia.org/repos/research/knowledge_integrity/-/blob/mnz/examples/examples/notebooks/revertrisk_example.ipynb.
Comment Actions
Much appreciated! Summarising the enwiki data from above in a more readable format so I can remember how this should be interpreted:
- In Jan 2022 on English Wikipedia, 288,768 edits were reverted (12% of all edits).
- With a threshold of 0.98, the model would revert 13,491 of those (4.7% of reverted edits; 450 per day) and with a threshold of 0.99 would revert 3,245 (1.1% of reverted edits; 108 per day).
- At 0.98 it would have a false positive rate of 25%, reverting 4,476 edits which were not reverted by the community, and at 0.99 a false positive rate of 11%, reverting 410 edits which were not reverted by the community.
Worth noting, as I understand it, that this definition of reverted includes any edit which takes a page back to an earlier state. That means we're not comparing only to anti-vandalism edits.
I'm curious what kinds of edits the model is giving false positives for at 0.98/0.99, i.e. is there a pattern? Could we easily drop that number by, for example, excluding certain namespaces or user groups?
Comment Actions
We know that all these models (Revert Risk but also ORES) are biased against IP edits. Anyhow, our model is know mitigating that bias, and we are doing better on that front that former models like ORES.
The model that Pablo is reporting here is the language agnostic one, we run that one because is faster. However, we have another model based on mBert, a Large Language Model, that is more accurate, and less biased against IP edits, but slower. So, one option could be to apply the Language Agnostic (faster) on edits done by registered users, and use the slower one for the IP edits.
Comment Actions
Just to clarify, do you mean that the false positives are probably primarily good IP edits?
Can you remind me how many languages the mBert version supports?