Summary:

• The bot usually monitors content namespaces and for every revert that was made, a message is posted to the user's talk page whose edit had been reverted.
  • While there were edits made in content namespaces that are not reverts, and reverts in non-content namespaces, their frequency is insigificant, and don't add any value to the analysis.
  • non-content reverts: 0.029% & content edits (excluding reverts): 0.009% of all time edits made by the bot
• From 2010 to early 2013, the bot made between 1500-2500 reverts on average per day.
• From 2013 to 2018, the average daily reverts were between 500-1000.
• From 2019 to mid-2020, the average daily reverts were between 150-250.
• Starting July 2020, the average daily reverts increased to ~500, a trend which continued until the end of 2021.
• There was a sharp drop in the daily reverts made since Jan 2022; it continued to drop until August 2022 when the bot made 65 reverts on average per day.
• From Sep 2022 to Jun 2023 (end of data), the average daily reverts were between 150-200.

Note: As the bot posts a talk page message for each revert (in most cases), the actual edit count would approximately be double of average reverts mentioned above.

Trend of average number of daily reverts (by month; content namespace)

Trend of average number of daily edits (by month; non content namespace)

• Complete data file
• Notebook

@DamianZaremba - I believe you're the current primary maintainer of ClueBot NG? If so, do you think you could provide any additional context to the data above? I'm assuming that some config changes occurred to result in such substantial changes in revert rate, so I'd love to know more about what those were if possible.

In terms of the bot, there's been no notable configuration changes made over the described time period; quite literally the configuration has not changed from the currently deployed version in 2 years (previously was the same, but 2 years ago was the last build in CI and is the as-deploy artefacts x-ref https://github.com/cluebotng/utilities/blob/main/fabfile.py / https://github.com/cluebotng/core/releases/tag/v1.0.2).

Looking at the internal vandalism database on reverted and non-reverted (i.e. beaten/failed) edits I don't really see a massive difference in terms of % reverted, but the overall volume has dropped at least at the monthly stats;

Looking at the beaten stats (logged when vandalism is not reverted), it doesn't look horrific either;

There are only ~2 days of logs stored due to toolforge wishes (T284966, T272435), but to give you a general idea of how "successful" the bot is at reverting;

Default reverted successfully;
18: 155
19: 118

Revert beaten by another user;
18: 20
19: 8

Reverted before;
18: 22
19: 14

User has edit count;
18: 12
19: 13

User is creator;
18: 1
19: 0

Checking git, the main change in March 22 (a long time ago, but around the time of the volume drop) was the migration from running on bigbrother to kubernetes.

Given the forking model of ClueBot NG and the relatively strict quotas on k8s it wouldn't surprise me if there is an artificial limit after that date (I certainly experienced this while experimenting with moving to a coroutine model in terms of database access).

The pods are regularly restarted, either due to health check failures (no recent edits) or quotas

$ kubectl get pods
NAME                         READY   STATUS             RESTARTS            AGE
cbng-7c85d8b75f-9m5rc        3/4     CrashLoopBackOff   17751 (3m46s ago)   96d

$ kubectl get quota tool-cluebotng
NAME             AGE      REQUEST                                                                                                                                                                                                                                                                          LIMIT
tool-cluebotng   3y246d   configmaps: 2/10, count/cronjobs.batch: 0/50, count/deployments.apps: 2/3, count/jobs.batch: 0/15, persistentvolumeclaims: 0/3, pods: 2/10, replicationcontrollers: 0/1, requests.cpu: 1/2, requests.memory: 5220Mi/6Gi, secrets: 1/10, services: 1/1, services.nodeports: 0/0   limits.cpu: 1700m/2, limits.memory: 5732Mi/8Gi

I've bumped the cpu quota from 0.2 to 0.5 for the bot, which leaves us at

NAME             AGE      REQUEST                                                                                                                                                                                                                                                                          LIMIT
tool-cluebotng   3y246d   configmaps: 2/10, count/cronjobs.batch: 0/50, count/deployments.apps: 2/3, count/jobs.batch: 0/15, persistentvolumeclaims: 0/3, pods: 2/10, replicationcontrollers: 0/1, requests.cpu: 1/2, requests.memory: 5220Mi/6Gi, secrets: 1/10, services: 1/1, services.nodeports: 0/0   limits.cpu: 1700m/2, limits.memory: 5732Mi/8Gi

That appears to be the upper limit, even-though there is some quota left, though now there is nothing free to any ad-hoc jobs.... perhaps it will improve the throughput given the current architecture.

Thanks for the thorough reply @DamianZaremba! I guess we should give this a few weeks and check back in to see if anything changed in terms of the revert volume.

It's certainly still curious that the number of daily reverts has gradually decreased over time. I wonder if this is more indicative of an increasing quality of edits, or simply that other anti-abuse mechanisms are being used more (e.g. AbuseFilter, page protection), or something else.

@DamianZaremba

I was looking a bit further into the percentage of the reverts made by ClueBot NG that were reverted back. Although all of these are not necessarily false positives, it's interesting to see that the percentage of reverts reverted back has come down by about 50% over time (~16% at the beginning to ~8% in June 2023). I am wondering if you have any insight into what's causing this (it can be that the model is re-learning consistently, or the decrease in overall revert rate might be a factor, or something else).

In T343952#9124449, @KCVelaga_WMF wrote:

@DamianZaremba

I was looking a bit further into the percentage of the reverts made by ClueBot NG that were reverted back. Although all of these are not necessarily false positives, it's interesting to see that the percentage of reverts reverted back has come down by about 50% over time (~16% at the beginning to ~8% in June 2023). I am wondering if you have any insight into what's causing this (it can be that the model is re-learning consistently, or the decrease in overall revert rate might be a factor, or something else).

Not really - the data files (model) for core have no changed since the approval as far as I know (i.e. no re-training has happened); the current files are @ https://github.com/cluebotng/core/tree/main/data.
Indeed the configs related to e.g. thresholds (https://github.com/cluebotng/core/tree/main/conf) have not changed either. There have been minor updates over time for e.g. API changes, but nothing substantial that I can think of right now.

Reported false positives (including those human reviewed as not valid, not yet human reviewed etc) look like this (I've excluded 2020-07 at 13917, which makes the rest of the graph useless);

I might be able to do some investigation into if there is currently anything "off", but for history it will be quite difficult. I'm also rather busy through until December at this point, so unfortunately have limited time for CB.