This work probably depends on T365581: Use multilingual revert risk model in Automoderator on supported wikis so that we know which model's data to use for comparison.

Looks like we'll need to make API calls for the multilingual revert risk scores (until T415892, but I'd like to make a start straight away). Then we can do something like what was done in the previous analysis for actual reverts.

I'm not sure how long calling the API will take (will need to fetch all the revision IDs for a month, then call the LiftWing API for revert risk scores for each one). I'm not sure how well this will scale, so I'll test first on a smaller wiki. @Samwalton9-WMF Any preference on which small wiki I start with - is there a particular wiki whose data it would be interesting to see first? If not, I'll start with bg.wiki.

In T374698#11601081, @Tchanders wrote:

I'm not sure how long calling the API will take (will need to fetch all the revision IDs for a month, then call the LiftWing API for revert risk scores for each one). I'm not sure how well this will scale, so I'll test first on a smaller wiki. @Samwalton9-WMF Any preference on which small wiki I start with - is there a particular wiki whose data it would be interesting to see first? If not, I'll start with bg.wiki.

We don't necessarily need a month, we could start with a week, if time is an issue, that might still give us enough analysis to present useful data.

I think bg.wiki is a fine place to start, no strong preference from me.

Thanks @Samwalton9-WMF - that helps!

In T374698#11601081, @Tchanders wrote:

I'm not sure how long calling the API will take (will need to fetch all the revision IDs for a month, then call the LiftWing API for revert risk scores for each one).

Not really scientific, but for ~3000 changes out of which maybe 20 need more processing, obtaining the scores from LiftWing from Toolforge tages ~20 min. I actually do some more processing (getting the data from the RC API and reverting the 20 problematic edits) and also includes errors (aka article creations), so this is more of an upper limit.

Thanks @Strainu - I experienced roughly similar timings to get multilingual revert risk scores via the internal stats server (about 15 minutes for 4-5k revisions).

@Samwalton9-WMF I have a couple more questions, now that I've tested it out on bgwiki.

Are these assumptions OK when fetching the revision IDs to include in the analysis for a given wiki?

• Use the last week of January as the time period (in case we need the historical dumps for looking at reverts - not available yet for February)
• Don't include article creations
• Don't include bot edits
• Main namespace only

And thinking ahead to the larger wikis, would it be OK to look at something like the last n revisions in January (e.g. 10-20k or something), rather than a fixed time range? (enwiki has about 700k non-article-creation non-bot edits in the last week of January, so would take about 35 hours to get the revert risk scores, assuming the timings I got from bgwiki scale linearly.)

One more question! From the task description:

• How many of the existing bot's edits were subsequently reverted by a user other than the one that made the initially-reverted edit? (approximating false positive rate, which we estimate to be ~1% for Automoderator at this threshold)
  • We should perhaps also generate false positive values for the categories listed above too - i.e. the extra reverts that one of these bots might make might be at a higher false positive rate.

I understand the first bullet, but not the second - what are the categories?

@Tchanders PSS9 hasn't run since August 2025.- I believe you'll need to choose a week when the local bot run in order to do the comparison.

In T374698#11603818, @Tchanders wrote:

Thanks @Strainu - I experienced roughly similar timings to get multilingual revert risk scores via the internal stats server (about 15 minutes for 4-5k revisions).

@Samwalton9-WMF I have a couple more questions, now that I've tested it out on bgwiki.

Are these assumptions OK when fetching the revision IDs to include in the analysis for a given wiki?

• Use the last week of January as the time period (in case we need the historical dumps for looking at reverts - not available yet for February)
• Don't include article creations
• Don't include bot edits
• Main namespace only

Yep, I think that's all correct.

And thinking ahead to the larger wikis, would it be OK to look at something like the last n revisions in January (e.g. 10-20k or something), rather than a fixed time range? (enwiki has about 700k non-article-creation non-bot edits in the last week of January, so would take about 35 hours to get the revert risk scores, assuming the timings I got from bgwiki scale linearly.)

That could work! I guess the main consideration will be what number of bot reverts that gets us to, but perhaps we can start here and see what the numbers look like?

In T374698#11603860, @Tchanders wrote:

One more question! From the task description:

• How many of the existing bot's edits were subsequently reverted by a user other than the one that made the initially-reverted edit? (approximating false positive rate, which we estimate to be ~1% for Automoderator at this threshold)
  • We should perhaps also generate false positive values for the categories listed above too - i.e. the extra reverts that one of these bots might make might be at a higher false positive rate.

I understand the first bullet, but not the second - what are the categories?

Ah, I meant that we could generate overall false positive rates for a given anti-vandalism bot, but also the false positive rate of 'Reverts the existing bot would make that Automoderator wouldn't', and 'Reverts Automoderator would make that the anti-vandalism bot wouldn't'. The hypothesis being that we might say "Automoderator would revert 20% more edits!", which sounds good, and then discover that those 20% edits are, say, 80% false positives.

In T374698#11604193, @Strainu wrote:

@Tchanders PSS9 hasn't run since August 2025.- I believe you'll need to choose a week when the local bot run in order to do the comparison.

Oh, great catch, I forgot to look at these bots again this month. Perhaps we should start with simple.wiki instead then? It's currently active.

Thanks @Samwalton9-WMF and @Strainu for the pointers.

I'm sharing the results so far for simplewiki, to check it's on the right lines and ask a couple of questions. I'm still working on the final question and also working on where to share the Jupyter notebooks. The results so far are in this internal sheet.

Answers to the questions:

• Wiki: simplewiki
• Date range: 25-31 Jan 2026
• Revisions included in the analysis: 4508

Included revisions were non-bot-group, main namespace, non-page-creation revisions that returned successful results from RRML model. A further ~350 returned HTTP 422 unprocessable errors, e.g revision_info_deleted, and were not included in the analysis.

• What is the median (and mode) number of edits that the existing bot reverted and Automoderator would revert per day?**

Mean: 83.1/day
Median: 82/day
Mode: As it happens there's a modal value of 72/day (occurs on 2 days), but is this likely to be a useful metric? I imagine it will be unusual to have exactly the same number of reverts in a day.

• What percentage of edits that the existing bot reverted would Automoderator have also reverted?

87.5%

• What percentage of the reverts Automoderator would have made did the existing bot already make?

4.8%

• What percentage of reverts would only Automoderator or only the existing bot have made?

I realised I'm not quite sure what this means. I took two guesses - is either one right?!

Guess 1: The number of reverts only made by one bot / the number of reverts made by either or both bots:

• automoderator: 94.5%
• existing bot: 0.7%

Guess 2: The number of reverts made by one bot / the total number of all revisions

• 12.9%
• 0.7% (Note - this weirdly is the same % as above. I double checked and it's just a coincidence)

• How many of the existing bot's edits were subsequently reverted by a user other than the one that made the initially-reverted edit? (approximating false positive rate, which we estimate to be ~1% for Automoderator at this threshold)
  • We should perhaps also generate false positive values for the categories listed above too - i.e. the extra reverts that one of these bots might make might be at a higher false positive rate.

Not answered yet - working on this tomorrow.

• How many of the existing bot's edits were subsequently reverted by a user other than the one that made the initially-reverted edit? (approximating false positive rate, which we estimate to be ~1% for Automoderator at this threshold)

False positive rate: 6.2%

Are we removing bgwiki from the analysis, or would it be helpful to compare some data from August?

In T374698#11614466, @Tchanders wrote:

Are we removing bgwiki from the analysis, or would it be helpful to compare some data from August?

My2c here is that it would help the community gain trust in Automoderator. As I was already asked to help in the process, I'd like to ask for the comparison to be made, if it's not too much effort.

In T374698#11614466, @Tchanders wrote:

• How many of the existing bot's edits were subsequently reverted by a user other than the one that made the initially-reverted edit? (approximating false positive rate, which we estimate to be ~1% for Automoderator at this threshold)

It's maybe not the right place to raise this, so feel free to move wherever you see fit, but 1% seems unreasonable. I believe this will become apparent when you extract data for rowiki, which already supports the same model, with a 0.95 threshold. I'll not give out the numbers just yet, but they're much higher.

Thanks @Strainu. It's not too much effort to analyse bgwiki from August.

Similar question for ptwiki - it looks as though Salebot hasn't been active since 2020: https://pt.wikipedia.org/wiki/Especial:Contribui%C3%A7%C3%B5es/Salebot . @Samwalton9-WMF Is there a more recent bot for ptwiki?

In T374698#11614561, @Strainu wrote:

In T374698#11614466, @Tchanders wrote:

• How many of the existing bot's edits were subsequently reverted by a user other than the one that made the initially-reverted edit? (approximating false positive rate, which we estimate to be ~1% for Automoderator at this threshold)

It's maybe not the right place to raise this, so feel free to move wherever you see fit, but 1% seems unreasonable. I believe this will become apparent when you extract data for rowiki, which already supports the same model, with a 0.95 threshold. I'll not give out the numbers just yet, but they're much higher.

I see 1.7% error rate for rowiki, defined as % of bot reverts that were re-reverted by someone other than the original author.

Thanks @Tchanders!! I really appreciate the raw numbers in the sheet - it helps give a lot of context to the percentages! My first reaction is that ~500 reverts out of 4,500 edits seems like a lot. I ran a couple of quick queries for this date range and 509 out of 4,508 edits were reverted in total on simple.wiki. I wouldn't expect Automoderator to be correctly reverting every single possible human edit, and also don't anticipate that Simple English editors are missing significant numbers of bad edits. This highlights that the Revert Risk threshold we estimated (T385102) probably isn't accurate for simple.wiki. It was based on id.wiki, but we already know that the required threshold is likely to be different for different wikis. I wonder if we need to repeat that analysis too, to target 1% revert rate on our various target wikis...

I realised I'm not quite sure what this means. I took two guesses - is either one right?!

I think Guess 1 is what I was going for here - figuring out the overlap of edits each tool would revert. The data in Guess 2 is interesting too though.