While we wait to hear back from KC :)

I am back at work today, I will re-run and update back before 3 Dec EOD.

Done. There are some edits which were of high risk and weren't reverted by Automoderator. I added a new sheet to the previous workbook itself.

Also, this notebook can be used to re-run the analysis in future if needed, for a different month or a different wiki. It needs to be run on one of the stat machines, and please change as required under the "Data-Gathering" section.

Amazing, thank you!

Here are some diffs where it's not clear to me why Automoderator wouldn't have reverted:

• https://tr.wikipedia.org/wiki/Special:Diff/34121590
• https://tr.wikipedia.org/wiki/Special:Diff/33985153
• https://tr.wikipedia.org/wiki/Special:Diff/33929703
• https://tr.wikipedia.org/wiki/Special:Diff/34045629

Let's start with those and I can identify more if it would be helpful.

https://tr.wikipedia.org/w/index.php?diff=34121590

• Self-revert so this will not pass our revision check logic.

https://tr.wikipedia.org/w/index.php?title=Donald_Trump&diff=next&oldid=33985153

• Reverted minutes after which is indicative of an edit conflict that caused this revert not to happen.

https://tr.wikipedia.org/w/index.php?title=Kadir_M%C4%B1s%C4%B1ro%C4%9Flu&diff=next&oldid=33929703

• Also reverted minutes after-indicative of an edit conflict that caused this revert not to happen.

After looking through a lot of the reverts in the spreadsheet, most of them were reverted within minutes of the edit occurring which tracks with an edit conflict.

However, there are some questionable reverts:
There are no error logs for any of these which makes me think that the hook was never fired for these revisions:
https://tr.wikipedia.org/w/index.php?diff=34017497
https://tr.wikipedia.org/w/index.php?diff=34029085
https://tr.wikipedia.org/w/index.php?diff=33998212
https://tr.wikipedia.org/w/index.php?diff=33934858
https://tr.wikipedia.org/w/index.php?diff=34045117
https://tr.wikipedia.org/w/index.php?diff=33991974
https://tr.wikipedia.org/w/index.php?diff=34128673
https://tr.wikipedia.org/w/index.php?diff=34010612

Most of the edits that were not reverted have two things in common:

• the next edit occurs within the same hour or two as the original edit
• the next edit is an undo, revert, or rollback (no content added)

I am not fast enough to trigger it locally, however it's similar to our other issue T381280: Handle edits that could not be rolled back, but with a fun new twist!

1. Someone edits the page and it is an edit that should be reverted by AutoModerator.
2. Job gets queued.
3. Someone else edits the page before our queued job is complete and it is an undo
   • The getUndoContent logic in RevisionCheck grabs the current revision of the page rather than the original revision passed into the job which would be the revision that was a revert. (see here.)
   • The edit was an undo so there is null content on the revision record (Is this true?)
   • Job succeeds and revert does not happen

We can remove the getUndoContent function entirely since it was cribbed from EditPage.php specifically for undos and we're not using those anymore.

In T374440#10411188, @Kgraessle wrote:

Most of the edits that were not reverted have two things in common:

• the next edit occurs within the same hour or two as the original edit
• the next edit is an undo, revert, or rollback (no content added)

I am not fast enough to trigger it locally, however it's similar to our other issue T381280: Handle edits that could not be rolled back, but with a fun new twist!

You can add a delay into the job at insertion time if you want to slow things down

1. Someone edits the page and it is an edit that should be reverted by AutoModerator.
2. Job gets queued.
3. Someone else edits the page before our queued job is complete and it is an undo
   • The getUndoContent logic in RevisionCheck grabs the current revision of the page rather than the original revision passed into the job which would be the revision that was a revert. (see here.)

getUndoContent does a 3-way merge. It grabs the latest revision, the revision that triggered the job, and the parent of the revision that triggered the job.

• The edit was an undo so there is null content on the revision record (Is this true?)

I don't think so? You can check your local revision table after doing an undo if you'd like to check what's happening here. I have a nasty habit of calling some things null revision even though they aren't, so I may have added to confusion here.
I'd expect $this->contentHandler->getUndoContent() to be false and I'd expect this to have an originalRevId turn up in a revision search like that run by EditPage if we were running it in the job after the followup edit instead of feeding the originalRevId value at job insert time.

• Job succeeds and revert does not happen

If this is the case, I'd expect maybeRollback() to return something like [ 0 => 'error' ]; I'd have to do more spelunking to know how that does or doesn't bubble up to the job status. I think that's worth checking.

We can remove the getUndoContent function entirely since it was cribbed from EditPage.php specifically for undos and we're not using those anymore.

Agreed; this (early exit) is the behavior I'd expect for an undo that's already been done, but I don't see how it's helping us with rollbacks. We could probably just start here and see if the issue stops occurring.

The edit was an undo so there is null content on the revision record (Is this true?)

I think what is also happening is that undo, reverts and rollbacks are skipped on the pre-check. https://gerrit.wikimedia.org/r/plugins/gitiles/mediawiki/extensions/AutoModerator/+/refs/heads/master/src/RevisionCheck.php#204

In T374440#10411294, @jsn.sherman wrote:

In T374440#10411188, @Kgraessle wrote:

Most of the edits that were not reverted have two things in common:

• the next edit occurs within the same hour or two as the original edit
• the next edit is an undo, revert, or rollback (no content added)

I am not fast enough to trigger it locally, however it's similar to our other issue T381280: Handle edits that could not be rolled back, but with a fun new twist!

You can add a delay into the job at insertion time if you want to slow things down

1. Someone edits the page and it is an edit that should be reverted by AutoModerator.
2. Job gets queued.
3. Someone else edits the page before our queued job is complete and it is an undo
   • The getUndoContent logic in RevisionCheck grabs the current revision of the page rather than the original revision passed into the job which would be the revision that was a revert. (see here.)

getUndoContent does a 3-way merge. It grabs the latest revision, the revision that triggered the job, and the parent of the revision that triggered the job.

• The edit was an undo so there is null content on the revision record (Is this true?)

I don't think so? You can check your local revision table after doing an undo if you'd like to check what's happening here. I have a nasty habit of calling some things null revision even though they aren't, so I may have added to confusion here.
I'd expect $this->contentHandler->getUndoContent() to be false and I'd expect this to have an originalRevId turn up in a revision search like that run by EditPage if we were running it in the job after the followup edit instead of feeding the originalRevId value at job insert time.

Ok, so I tested this locally and you are correct; looks like this theory is out.

• Job succeeds and revert does not happen

If this is the case, I'd expect maybeRollback() to return something like [ 0 => 'error' ]; I'd have to do more spelunking to know how that does or doesn't bubble up to the job status. I think that's worth checking.

I'll take a look at this.

We can remove the getUndoContent function entirely since it was cribbed from EditPage.php specifically for undos and we're not using those anymore.

Agreed; this (early exit) is the behavior I'd expect for an undo that's already been done, but I don't see how it's helping us with rollbacks. We could probably just start here and see if the issue stops occurring.

Agree.

In T374440#10411323, @Scardenasmolinar wrote:

The edit was an undo so there is null content on the revision record (Is this true?)

I think what is also happening is that undo, reverts and rollbacks are skipped on the pre-check. https://gerrit.wikimedia.org/r/plugins/gitiles/mediawiki/extensions/AutoModerator/+/refs/heads/master/src/RevisionCheck.php#204

--It looks like currently the hook is not firing on reverts unless someone has changed content as far as I can tell.-- (Ignore this)

Moderator tools engineers looked at this together today and determined that this is in fact an issue with flagged revs; when the parent rev hasn't been approved, getParentId() will return the closest approved ancestor in the revision graph, leading to this issue. Moving to done column, we'll resolve this after we backlog a task to deal with the issue.

In T374440#10413319, @jsn.sherman wrote:

Moderator tools engineers looked at this together today and determined that this is in fact an issue with flagged revs; when the parent rev hasn't been approved, getParentId() will return the closest approved ancestor in the revision graph, leading to this issue. Moving to done column, we'll resolve this after we backlog a task to deal with the issue.

Ok, I was writing up a ticket for this and realized I was looking at the logs for AutoModerator reverts not the editor's reverts. This may not be the issue after all. Will move back to in progress for further investigation.

Change #1105450 had a related patch set uploaded (by Kgraessle; author: Kgraessle):

[mediawiki/extensions/AutoModerator@master] Investigate why some tr.wiki edits with Revert Risk scores greater than 0.99 were not reverted by Automoderator

https://gerrit.wikimedia.org/r/1105450

I have a small PR to refactor the RevisionCheck class to remove two early returns:

• When content is falsey (this was only relevant when we were doing undos)
• When the edit is a revert and the parent is AutoModerator (I don't think we want this right?)

This may have been the cause of some of the missing reverts.
I think I will re-run the analysis (and test out my access to stats servers) to get data around more recent edits, as a lot of code changes have also happened since these October edits.

I ended up running the analysis for November, because December is not released yet.

61 edits were not reverted by AutoModerator that should have been. This is down from the 74 we had in October.
Of the 61 edits, 59 were reverted within 5 hours of the initial edit which could indicate an edit conflict.
We'll handle those cases when we do T381280: Handle edits that could not be rolled back.

There were two outliers:
https://tr.wikipedia.org/w/index.php?diff=34202786
https://tr.wikipedia.org/w/index.php?diff=34215585

More to come, just wanted to post what updates I had.

Of the 61 edits, 59 were reverted within 5 hours of the initial edit which could indicate an edit conflict.

I'm curious about this - according to our dashboard it currently takes Automoderator an average of 15 seconds to revert an edit. Is it sometimes taking a long time? It's surprising to me that it might sometimes take hours to revert the edit, which is what it sounds like this implies?

In T374440#10486751, @Kgraessle wrote:

[...]

Since there's not enough data for me to understand why this is happening, I suggest we move this back to ready and stall it until we can do one of the following:

Agree; we can create a new task for one or both of these as we decide.

1. Logging out early returns in logstash (not all of them, but maybe the ones we most suspect as being the cause?).

I thought about this approach some more. We could add extra logging behind a config flag so that we can enable just on a debug host:
https://wikitech.wikimedia.org/wiki/Debugging_in_production

I've never done this, so we'd need to find out a little bit more about how we should go about it (eg. should the new logs be warning level along with a log level change for the debug host? should they be error level even though they are diagnostic rather than errors?)

@Scardenasmolinar can you weigh in as well?

Let's create a task for logging the early returns and maybe create another spike to investigate how to do proper debugging in production.

@KCVelaga_WMF

I was attempting to re-run this analysis for the month of December and I'm seeing the following error:

AssertionError: for the specified snapshot: 2024-12 and wiki_db: trwiki, there are no records in revert risk predictions

Do I need to wait longer for the snapshot to be released?

@KCVelaga_WMF not sure if you missed this.

Change #1105450 merged by jenkins-bot:

[mediawiki/extensions/AutoModerator@master] Investigate why some tr.wiki edits with Revert Risk scores greater than 0.99 were not reverted by Automoderator

https://gerrit.wikimedia.org/r/1105450

@KCVelaga_WMF

It looks like there are still no revert risk predictions for 2024-12 and wiki_db: trwiki.
Do you happen to know when the revert risk prediction dumps will be available to re-run this analysis?

Stalled on T388453: Make the revert risk predictions datasets available for analysis

@Kgraessle From the Slack conversation, we now know that the pipeline hasn't been active for the past couple of months. But Muniza generated the scores manually, so I can modify the code (a very minor one) to read from the parquet, instead of the Hive table. However, I have hit a permissions issue, and I pinged Muniza. I'll share the modified code once I get the access.

The assertion is intentional, to ensure the snapshots are available before running the rest of the code.

Also, for future reference, feel free to book a Consultation Hour with the Product Analytics team -- we can resolve these on a call, and it will be faster :)

Here is the notebook with the modified code to read from the parquet (cells: In [19], In [21], In [26], In [32]). This is a branch of the actual one, which can used later if the pipeline is active again.

@Kgraessle

@KCVelaga_WMF is it possible to get the same data for January?

Shared on Slack.

After some more investigation with more up to date data, I believe this is actually a non-issue.

I went back through the December edits that were problematic and scored them via the Liftwing API.
None of the edits except for 34496346 came back above the very-cautious threshold.

I also went back through the January edits and none of the problematic edits scored above the very-cautious threshold.

@Samwalton9-WMF if you want me to still look into the single edit from December that scored above the threshold we can, though a lot of code has changed since then so it may not be that useful of an exercise.

If so, it would be coincidental. When we initially specced this out, I think we only cared about precision to the 3rd decimal place? We should probably actually make sure we match what's available! It's absolutely possible that there is a rounding/precision issue here. If I recall, floats in php can be kind of hinky (which was actually part of the secure poll pain).

In T374440#10648946, @Kgraessle wrote:

After some more investigation with more up to date data, I believe this is actually a non-issue.

I went back through the December edits that were problematic and scored them via the Liftwing API.
None of the edits except for 34496346 came back above the very-cautious threshold.

I also went back through the January edits and none of the problematic edits scored above the very-cautious threshold.

@Samwalton9-WMF if you want me to still look into the single edit from December that scored above the threshold we can, though a lot of code has changed since then so it may not be that useful of an exercise.

Hm. I'm confused :)

Why would the scores have changed for these edits? In the spreadsheet KC produced, this edit, for example, is listed as having a score of 0.9967. When I use Jason's script to get the score on-wiki, it now returns 0.94.

In T374440#10651387, @Samwalton9-WMF wrote:

In T374440#10648946, @Kgraessle wrote:

After some more investigation with more up to date data, I believe this is actually a non-issue.

I went back through the December edits that were problematic and scored them via the Liftwing API.
None of the edits except for 34496346 came back above the very-cautious threshold.

I also went back through the January edits and none of the problematic edits scored above the very-cautious threshold.

@Samwalton9-WMF if you want me to still look into the single edit from December that scored above the threshold we can, though a lot of code has changed since then so it may not be that useful of an exercise.

Hm. I'm confused :)

Why would the scores have changed for these edits? In the spreadsheet KC produced, this edit, for example, is listed as having a score of 0.9967. When I use Jason's script to get the score on-wiki, it now returns 0.94.

I can think of three possibilities:

• I made a mistake somewhere
• KC made a mistake somewhere
• the scores returned changed between when KC ran his analysis and now

I just did a check:

curl https://api.wikimedia.org/service/lw/inference/v1/models/revertrisk-language-agnostic:predict -X POST -d '{"rev_id": "33513733", "lang": "tr"}' -H "Content-type: application/json"

and got the following result:

{"model_name":"revertrisk-language-agnostic","model_version":"3","wiki_db":"trwiki","revision_id":"33513733","output":{"prediction":true,"probabilities":{"true":0.9379007816314697,"false":0.06209921836853027}}}

Which lines up with what you see in the user script, which means it should be working fine; that would also align with AutoModerator not reverting the edit.

thinking about how the model works; it checks features like edit count. I wonder if subsequent edits from an existing user could lower the risk score of previous edits; eg. checking months later when the user has more of a history might result in a lower revert risk score?

Hmm, it is indeed confusing. I did some QA and the results are surprising. For all the edits, the output the API is different from what we have in the revert risk dataset snapshots. I added a new spreadsheet 2024-12 qa.

This is the code I used (at the end of the notebook).

We should check with the Research team why this is happening.

I'll get a conversation started; thanks @KCVelaga_WMF!

Am I right in thinking that here we concluded that the issue is that Revert Risk API scores are computed at the current time rather than the time of edit, so scores drift, and there's no Automoderator issue?