Failed executing job: AutoModeratorFetchRevScoreJob
Closed, ResolvedPublicPRODUCTION ERROR
Actions

Assigned To

Authored By

	jsn.sherman
	Jun 12 2024, 5:15 PM

Description

We're seeing a high number of "revision not found" errors now that trwiki has enabled automoderaotr (T362622); this isn't destructive in any way, as AutoModerator simply doesn't make any changes when this error occurs, but it means that the extension isn't working for these revisions.

Error

mwversion: 1.43.0-wmf.8
reqId: c425cca2-c514-4061-aaf6-3feec5361d51
Find reqId in Logstash

normalized_message

Failed executing job: AutoModeratorFetchRevScoreJob Rıdvan_Uz wikiPageId=3567508 revId=33119075 originalRevId= userId=233143 userName=Fatih Demirci tags=[] namespace=0 title=Rıdvan_Uz requestId=c425cca2-c514-4061-aaf6-3feec5361d51

exception.trace

Impact

Notes

Details

Request URL: https://mw-jobrunner.discovery.wmnet/rpc/RunSingleJob.php

Related Changes in Gerrit:

	Subject	Repo	Branch	Lines +/-
	Jobs: retry when revision not found	mediawiki/extensions/AutoModerator	master	+1 -1
	Jobs: use lazyPush	mediawiki/extensions/AutoModerator	master	+68 -25

Customize query in gerrit

Related Objects

Mentioned In: rEAMO190fc52351bc: Jobs: retry when revision not found
rEAMO9c9987439172: Jobs: use lazyPush
T362622: Enable AutoModerator on tr.wiki
Mentioned Here: T358708: Very large logspam of "Inconsistent revision ID"
T366933: AutoModerator should perform prechecks in RevisionFromEditCompleteHookHandler before queuing job
T362622: Enable AutoModerator on tr.wiki

Event Timeline

jsn.sherman created this task.Jun 12 2024, 5:15 PM

jsn.sherman triaged this task as Unbreak Now! priority.

Restricted Application added a subscriber: Aklapper. · View Herald TranscriptJun 12 2024, 5:15 PM

Dogu subscribed.Jun 12 2024, 5:17 PM

This looks like a problem with flagged revisions, which we'll need to sort out.

jsn.sherman mentioned this in T362622: Enable AutoModerator on tr.wiki.Jun 12 2024, 5:27 PM

actually, this looks a lot like the error in T366933. Our solution there is to make some changes to avoid running into it on edits that we don't actually want to score in the first place. This could be a good thing, since it makes flagged revisions a less likely suspect

@jsn.sherman could it be possible for the Automoderator to ignore flagged revisions on tr:wiki? If so, what would be the implications and potential solutions to implement this?

@Dogu we haven't diagnosed the problem yet, so it's too early to start working on solutions. The other task I referenced is on test wiki which isn't using flagged revisions. That's a good thing, because it means we can try to work out the problem somewhere other than trwiki.

Trizek-WMF subscribed.Jun 13 2024, 10:18 AM

jsn.sherman claimed this task.Jun 13 2024, 12:02 PM

jsn.sherman moved this task from Ready to In Progress on the Moderator-Tools-Team (Kanban) board.

Looking for commonalities between testwiki and trwiki:

On testwiki, these errors coincide with an "Inconsistent revision ID" warning from parser cache:

In this example revid1 in parsercache matches the revision id that we are using. That also matches what's in the page revision history

And I verified the same is true with trwiki. I tried to get a better screenshot this time:

and that warning has a phabricator task T358708: Very large logspam of "Inconsistent revision ID"

kostajh subscribed.Jun 13 2024, 5:53 PM

kostajh added a project: Automoderator.Jun 13 2024, 5:56 PM

I switched over to lazy push for the job queue insert. I experimented with adding a release timestamp, but that seemed to break things in mediawiki-docker, eg. No jobs were queued at all that way.

Change #1043168 had a related patch set uploaded (by Scardenasmolinar; author: Jsn.sherman):

[mediawiki/extensions/AutoModerator@master] Jobs: use lazyPush

https://gerrit.wikimedia.org/r/1043168

gerritbot added a project: Patch-For-Review.Jun 13 2024, 11:19 PM

Change #1043168 merged by jenkins-bot:

[mediawiki/extensions/AutoModerator@master] Jobs: use lazyPush

https://gerrit.wikimedia.org/r/1043168

Scardenasmolinar mentioned this in rEAMO9c9987439172: Jobs: use lazyPush.Jun 14 2024, 12:40 AM

Merged!

ReleaseTaggerBot added a project: MW-1.43-notes (1.43.0-wmf.10; 2024-06-18).Jun 14 2024, 1:00 AM

Maintenance_bot removed a project: Patch-For-Review.Jun 14 2024, 1:30 AM

A few notes:

I didn't state it clearly in this task, but the reasoning behind trying to defer running our jobs is that we suspect there is either a caching or a db consistency issue. Running the job later is a very coarse way to check that. Switching to lazyPush is a move in the right direction, but I'm not sure if it's enough.

@kostajh suggested that this could be a moment to consider moving to RecentChangeSave. ORES uses this hook and doesn't seem to suffer from the same issue. Having said that, ORES fetch score job isn't instantiating revisions to work on them, so that's not definitive.

I'll be looking at this more tomorrow.

Change #1043347 had a related patch set uploaded (by Jsn.sherman; author: Jsn.sherman):

[mediawiki/extensions/AutoModerator@master] Jobs: retry when revision not found

https://gerrit.wikimedia.org/r/1043347

gerritbot added a project: Patch-For-Review.Jun 14 2024, 2:01 AM

The simplest change we could make besides doing lazyPush is to allow retries when we encounter this.

jsn.sherman moved this task from Done to Eng review on the Moderator-Tools-Team (Kanban) board.Jun 14 2024, 2:01 AM

Change #1043347 merged by jenkins-bot:

[mediawiki/extensions/AutoModerator@master] Jobs: retry when revision not found

https://gerrit.wikimedia.org/r/1043347

jsn.sherman mentioned this in rEAMO190fc52351bc: Jobs: retry when revision not found.Jun 14 2024, 2:26 AM

Maintenance_bot removed a project: Patch-For-Review.Jun 14 2024, 2:30 AM

jsn.sherman moved this task from Eng review to QA on the Moderator-Tools-Team (Kanban) board.Jun 14 2024, 1:03 PM

So far, it looks like there have been no more occurrences of this error on testwiki since 1.43.0-wmf.10 rolled out to it.

In T367339#9903362, @jsn.sherman wrote:

So far, it looks like there have been no more occurrences of this error on testwiki since 1.43.0-wmf.10 rolled out to it.

Does that mean we can downgrade from UBN?

Scardenasmolinar lowered the priority of this task from Unbreak Now! to High.Jun 20 2024, 4:13 PM

@Dogu we believe we have this sorted out; could you re-enable automoderator on trwiki? We'll be monitoring for errors.

In T367339#9922719, @jsn.sherman wrote:

@Dogu we believe we have this sorted out; could you re-enable automoderator on trwiki? We'll be monitoring for errors.

Done!

Kgraessle closed this task as Resolved.Jun 27 2024, 4:03 PM

Scardenasmolinar moved this task from QA to Done on the Moderator-Tools-Team (Kanban) board.Jun 11 2025, 11:11 PM

	F55286616: image.png
	Jun 13 2024, 12:51 PM

	F55286068: image.png
	Jun 13 2024, 12:32 PM

Failed executing job: AutoModeratorFetchRevScoreJobClosed, ResolvedPublicPRODUCTION ERRORActions