I think I may have figured what's happening. First, given there are duplicate log messages, that must mean there are duplicate AutoModeratorFetchRevScoreJobs, since that's the only place that creates a log message. There are two hook handlers that create this job, one for after revisions are saved, and one for after ORES scores are saved. So the immediate suspicion is both hooks are getting called, and both jobs end up making a log message (which doesn't happen with logging turned off, since it will error out gracefully if a revert already happened).

I can see in the RevisionFromEditCompleteHook handler that we are checking if the applicable ORES model is enabled, which in this case is revertriskmultilingual. If and only if that is enabled, do we return early and abort that hook handler. Util::getORESLanguageAgnosticModelName() also has the comment:

TODO: This hasn't been added to ORES; check if the name matches the model name when it is

Next I had a look at the itwiki site configuration, and indeed there is no value for $wgOresModels with the key revertriskmultilingual, rather I only see revertrisklanguageagnostic which is disabled.

This isn't really conclusive as I had to do a lot of hackery and mocking of return values just to get the code paths to flow as I suspect they would in production, but I was eventually able to observe multiple log messages for the same rev ID.

As for a solution, it seems instead of checking if revertriskmultilingual is enabled in RevisionFromEditCompleteHookHandler as the means to short-circuit, perhaps we should check if revertrisklanguageagnostic is disabled?

I'd like to defer to those who are more familiar with the extension and ORES/LiftWing, so I'm going to move this to the "Eng review" column. I could very easily be lacking some fundamental understanding of how it all ties together.

Also:

We're also noticing that logs display in users' Watchlists if they are watching a page which Automoderator would revert an edit on. This is not desirable - we do not want these logs to show up in the Watchlist. Adding this to the same ticket as it may be related.

I can confirm this is an unrelated issue. I have not verified, but I assume it is possible to hide these log entries from showing up in the Watchlist, but that should probably be filed as a separate task.

In T426401#12003187, @MusikAnimal wrote:

Also:

We're also noticing that logs display in users' Watchlists if they are watching a page which Automoderator would revert an edit on. This is not desirable - we do not want these logs to show up in the Watchlist. Adding this to the same ticket as it may be related.

I can confirm this is an unrelated issue. I have not verified, but I assume it is possible to hide these log entries from showing up in the Watchlist, but that should probably be filed as a separate task.

Thanks! Split out to T428717.