Add revertrisk-language-agnostic to RecentChanges filters
Open, MediumPublic
Actions

Assigned To

None

Authored By

	isarantopoulos
	Oct 6 2023, 6:18 AM

Description

As a patroller,

I'd like to have revertrisk language-agnostic predictions exposed via recent changes filters so that they can further assist my work while trying to fight vandalism.

Exposing revertrisk predictions as separate filters can act complementary to the existing damaging and goodfaith models but more importantly will allow us to enable the ORES extension in more wikis/languages for which we don't have such models at the moment.

We also need to update https://www.mediawiki.org/wiki/Help:New_filters_for_edit_review to document the addition of new filters.

Per T348298#9559718 we will start with one filter level at 0.95

Details

Subject	Repo	Branch	Lines +/-
update revertrisk-language-agnostic min & desc	mediawiki/extensions/ORES	master	+2 -2
revertrisk: Fix i18n messages	mediawiki/extensions/ORES	wmf/1.42.0-wmf.15	+2 -2
revertrisk: Fix i18n message reference	mediawiki/extensions/ORES	wmf/1.42.0-wmf.15	+3 -3
revertrisk: Fix i18n messages	mediawiki/extensions/ORES	master	+2 -2
revertrisk: Fix i18n message reference	mediawiki/extensions/ORES	master	+3 -3
enable revertrisk	mediawiki/extensions/ORES	master	+22 -19
ORES: Enable renamed revertrisklanguageagnostic model	operations/mediawiki-config	master	+4 -2
revertrisk: Rename model to use single lowercase word	mediawiki/extensions/ORES	master	+49 -40
testwiki: enable revertrisk model in ores extension	operations/mediawiki-config	master	+2 -0
Revert "Revert "[beta] ores-extension: enable revertrisk model for enwiki""	operations/mediawiki-config	master	+1 -0
Beta: Move the override of ores LW URL to CS-labs.php	operations/mediawiki-config	master	+1 -3
[beta] ores-extension: enable revertrisk model for enwiki	operations/mediawiki-config	master	+1 -0
LiftWingService: Set Content-Type header for requests to RevertRisk	mediawiki/extensions/ORES	master	+1 -1
Add revertrisk model to the list of models	mediawiki/extensions/ORES	master	+222 -3

Related Objects
Search...

Status	Subtype	Assigned	Task
Open		None	T348298 Add revertrisk-language-agnostic to RecentChanges filters
Open		None	T352217 CRS: Community rollout plan and discussion about adding revertrisk to RecentChanges filters
Open		None	T356280 Use multilingual revert risk for anonymous edits
Resolved		jsn.sherman	T356281 Exclude first revision on page from scoring
Stalled		None	T365701 Enable Revert Risk RecentChanges filter on id.wiki
Open	Spike	None	T374077 [SPIKE] Investigate how to install ORES in idwiki [8HRS]

Event Timeline

There are a very large number of changes, so older changes are hidden. Show Older Changes

Samwalton9-WMF removed a parent task: T352437: Create Automoderator's edit reverting functionality.Dec 5 2023, 2:23 PM

Change 971547 merged by jenkins-bot:

[mediawiki/extensions/ORES@master] Add revertrisk model to the list of models

https://gerrit.wikimedia.org/r/971547

ReleaseTaggerBot added a project: MW-1.42-notes (1.42.0-wmf.9; 2023-12-12).Dec 5 2023, 10:00 PM

Maintenance_bot removed a project: Patch-For-Review.Dec 5 2023, 10:10 PM

Change 980539 had a related patch set uploaded (by Kosta Harlan; author: Kosta Harlan):

[mediawiki/extensions/ORES@master] LiftWingService: Set Content-Type header for requests to RevertRisk

https://gerrit.wikimedia.org/r/980539

gerritbot added a project: Patch-For-Review.Dec 6 2023, 6:17 AM

Change 980539 merged by jenkins-bot:

[mediawiki/extensions/ORES@master] LiftWingService: Set Content-Type header for requests to RevertRisk

https://gerrit.wikimedia.org/r/980539

Maintenance_bot removed a project: Patch-For-Review.Dec 6 2023, 8:10 AM

Change 981337 had a related patch set uploaded (by Ilias Sarantopoulos; author: Ilias Sarantopoulos):

[operations/mediawiki-config@master] [beta] ores-extension: enable revertrisk model for enwiki

https://gerrit.wikimedia.org/r/981337

gerritbot added a project: Patch-For-Review.Dec 7 2023, 2:19 PM

Change 981337 merged by jenkins-bot:

[operations/mediawiki-config@master] [beta] ores-extension: enable revertrisk model for enwiki

https://gerrit.wikimedia.org/r/981337

Mentioned in SAL (#wikimedia-operations) [2023-12-07T21:13:52Z] <jdrewniak@deploy2002> Started scap: Backport for [[gerrit:980951|Enable Vector beta feature for all wikis (T351339)]], [[gerrit:981337|[beta] ores-extension: enable revertrisk model for enwiki (T348298)]], [[gerrit:976911|Enable action blocks in Serbian Wikipedia (T351873)]]

Mentioned in SAL (#wikimedia-operations) [2023-12-07T21:15:13Z] <jdrewniak@deploy2002> zoranzoki21 and isaranto and jdlrobson and jdrewniak: Backport for [[gerrit:980951|Enable Vector beta feature for all wikis (T351339)]], [[gerrit:981337|[beta] ores-extension: enable revertrisk model for enwiki (T348298)]], [[gerrit:976911|Enable action blocks in Serbian Wikipedia (T351873)]] synced to the testservers (https://wikitech.wikimedia.org/wiki/Mwdebug)

Mentioned in SAL (#wikimedia-operations) [2023-12-07T21:23:46Z] <jdrewniak@deploy2002> Finished scap: Backport for [[gerrit:980951|Enable Vector beta feature for all wikis (T351339)]], [[gerrit:981337|[beta] ores-extension: enable revertrisk model for enwiki (T348298)]], [[gerrit:976911|Enable action blocks in Serbian Wikipedia (T351873)]] (duration: 09m 54s)

Maintenance_bot removed a project: Patch-For-Review.Dec 7 2023, 9:30 PM

Can I confirm the status of this patch? I see some merged patches, but was this only for Beta?

Yes this was only for Beta, but failed. I need to debug it and resubmit a new one Will do so in the following days and post a status update here.

Change 983681 had a related patch set uploaded (by Ilias Sarantopoulos; author: Ilias Sarantopoulos):

[mediawiki/extensions/ORES@master] enable revertrisk

https://gerrit.wikimedia.org/r/983681

gerritbot added a project: Patch-For-Review.Dec 18 2023, 10:47 AM

Test wiki on Patch demo by ISarantopoulos-WMF using patch(es) linked to this task was deleted:

~~https://patchdemo.wmflabs.org/wikis/9685c9655b/w/~~

Test wiki created on Patch demo by ISarantopoulos-WMF using patch(es) linked to this task:
https://patchdemo.wmflabs.org/wikis/989d97cc42/w

Change 983724 had a related patch set uploaded (by Ladsgroup; author: Amir Sarabadani):

[operations/mediawiki-config@master] Beta: Move the override of ores LW URL to CS-labs.php

https://gerrit.wikimedia.org/r/983724

Change 983724 merged by jenkins-bot:

[operations/mediawiki-config@master] Beta: Move the override of ores LW URL to CS-labs.php

https://gerrit.wikimedia.org/r/983724

Change 983734 had a related patch set uploaded (by Ladsgroup; author: Ladsgroup):

[operations/mediawiki-config@master] Revert "Revert "[beta] ores-extension: enable revertrisk model for enwiki""

https://gerrit.wikimedia.org/r/983734

Change 983734 merged by jenkins-bot:

[operations/mediawiki-config@master] Revert "Revert "[beta] ores-extension: enable revertrisk model for enwiki""

https://gerrit.wikimedia.org/r/983734

Change 983747 had a related patch set uploaded (by Ilias Sarantopoulos; author: Ilias Sarantopoulos):

[operations/mediawiki-config@master] testwiki: enable revertris kmodel in ores extension

https://gerrit.wikimedia.org/r/983747

After getting some help from @Ladsgroup we figured out the ores extension was long broken on the beta cluster. I manually triggered a job through the terminal and results came through as expected.
Will proceed to deploy and try it out on testwiki.

Thanks Amir!

isarantopoulos added a subscriber: Ladsgroup.Dec 18 2023, 4:17 PM

I noticed the order of the filters isn't as I expected it to be (Revertrisk filter on top) so I'll look into that again. If anyone mediawiki expert knows how the order is enforce in the UI of the recent changes page let me know!

Just put it at the top of the array in OresModels, that should be enoughTM

Test wiki on Patch demo by ISarantopoulos-WMF using patch(es) linked to this task was deleted:

~~https://patchdemo.wmflabs.org/wikis/989d97cc42/w/~~

Test wiki created on Patch demo by ISarantopoulos-WMF using patch(es) linked to this task:
https://patchdemo.wmflabs.org/wikis/e1febe2e5d/w

Change 983747 merged by jenkins-bot:

[operations/mediawiki-config@master] testwiki: enable revertrisk model in ores extension

https://gerrit.wikimedia.org/r/983747

Mentioned in SAL (#wikimedia-operations) [2023-12-19T14:05:34Z] <lucaswerkmeister-wmde@deploy2002> Started scap: Backport for [[gerrit:983747|testwiki: enable revertrisk model in ores extension (T348298)]]

Mentioned in SAL (#wikimedia-operations) [2023-12-19T14:08:02Z] <lucaswerkmeister-wmde@deploy2002> isaranto and lucaswerkmeister-wmde: Backport for [[gerrit:983747|testwiki: enable revertrisk model in ores extension (T348298)]] synced to the testservers (https://wikitech.wikimedia.org/wiki/Mwdebug)

Mentioned in SAL (#wikimedia-operations) [2023-12-19T14:15:56Z] <lucaswerkmeister-wmde@deploy2002> Finished scap: Backport for [[gerrit:983747|testwiki: enable revertrisk model in ores extension (T348298)]] (duration: 10m 22s)

@isarantopoulos The filter currently doesn't work, because in DatabaseQueryBuilder.php, validateSelected returns false, so there's no WHERE clause for revert risk results added to the query. I think the reason for this is that the other models have different thresholds for their results, while revertrisk is defined as a boolean "likely to be reverted" or not. I am not sure, but it seems like the easiest fix is to update the code to align with the threshold categories used in the`goodfaith` and damaging models (e.g. "likelybad", "maybebad", "verylikelybad").

Test wiki created on Patch demo by ISarantopoulos-WMF using patch(es) linked to this task:
https://patchdemo.wmflabs.org/wikis/15efa9543f/w

Change 988048 had a related patch set uploaded (by Kosta Harlan; author: Kosta Harlan):

[mediawiki/extensions/ORES@master] WIP revertrisk: Add thresholds for maybe/likely/very likely bad

https://gerrit.wikimedia.org/r/988048

I also found a bug but I'm not sure yet if fixing it would do the trick. In this function call the model name wasn't defined correctly. I have an open patch which I was testing with patchdemo but no revisions seem to be flagged over there. I'm also trying it out locally to see if anything changes.

Change 988048 merged by jenkins-bot:

[mediawiki/extensions/ORES@master] revertrisk: Rename model to use single lowercase word

https://gerrit.wikimedia.org/r/988048

ReleaseTaggerBot edited projects, added MW-1.42-notes (1.42.0-wmf.15; 2024-01-23); removed MW-1.42-notes (1.42.0-wmf.9; 2023-12-12).Jan 22 2024, 7:00 PM

Change 992423 had a related patch set uploaded (by Kosta Harlan; author: Kosta Harlan):

[operations/mediawiki-config@master] ORES: Enable renamed revertrisklanguageagnostic model

https://gerrit.wikimedia.org/r/992423

Change 992423 merged by jenkins-bot:

[operations/mediawiki-config@master] ORES: Enable renamed revertrisklanguageagnostic model

https://gerrit.wikimedia.org/r/992423

Mentioned in SAL (#wikimedia-operations) [2024-01-23T14:47:59Z] <logmsgbot> lucaswerkmeister-wmde@deploy2002 Started scap: Backport for [[gerrit:992423|ORES: Enable renamed revertrisklanguageagnostic model (T348298)]]

Mentioned in SAL (#wikimedia-operations) [2024-01-23T14:49:28Z] <logmsgbot> lucaswerkmeister-wmde@deploy2002 lucaswerkmeister-wmde and kharlan: Backport for [[gerrit:992423|ORES: Enable renamed revertrisklanguageagnostic model (T348298)]] synced to the testservers (https://wikitech.wikimedia.org/wiki/Mwdebug)

Mentioned in SAL (#wikimedia-operations) [2024-01-23T14:59:20Z] <logmsgbot> lucaswerkmeister-wmde@deploy2002 Finished scap: Backport for [[gerrit:992423|ORES: Enable renamed revertrisklanguageagnostic model (T348298)]] (duration: 11m 20s)

Samwalton9-WMF updated the task description. (Show Details)Jan 23 2024, 6:28 PM

Change 992492 had a related patch set uploaded (by Kosta Harlan; author: Kosta Harlan):

[mediawiki/extensions/ORES@master] revertrisk: Fix i18n message

https://gerrit.wikimedia.org/r/992492

Change 992493 had a related patch set uploaded (by Kosta Harlan; author: Kosta Harlan):

[mediawiki/extensions/ORES@master] revertrisk: Fix i18n message reference

https://gerrit.wikimedia.org/r/992493

Change 983681 abandoned by Kosta Harlan:

[mediawiki/extensions/ORES@master] enable revertrisk

Reason:

See I633a7c37326202461d9de2ef20ef8a5f0940f3aa

https://gerrit.wikimedia.org/r/983681

Change 992493 merged by jenkins-bot:

[mediawiki/extensions/ORES@master] revertrisk: Fix i18n message reference

https://gerrit.wikimedia.org/r/992493

Change 992506 had a related patch set uploaded (by Kosta Harlan; author: Kosta Harlan):

[mediawiki/extensions/ORES@wmf/1.42.0-wmf.15] revertrisk: Fix i18n message reference

https://gerrit.wikimedia.org/r/992506

Change 992492 merged by jenkins-bot:

[mediawiki/extensions/ORES@master] revertrisk: Fix i18n messages

https://gerrit.wikimedia.org/r/992492

Change 992507 had a related patch set uploaded (by Kosta Harlan; author: Kosta Harlan):

[mediawiki/extensions/ORES@wmf/1.42.0-wmf.15] revertrisk: Fix i18n messages

https://gerrit.wikimedia.org/r/992507

Change 992506 merged by jenkins-bot:

[mediawiki/extensions/ORES@wmf/1.42.0-wmf.15] revertrisk: Fix i18n message reference

https://gerrit.wikimedia.org/r/992506

Change 992507 merged by jenkins-bot:

[mediawiki/extensions/ORES@wmf/1.42.0-wmf.15] revertrisk: Fix i18n messages

https://gerrit.wikimedia.org/r/992507

Mentioned in SAL (#wikimedia-operations) [2024-01-23T21:05:12Z] <kharlan@deploy2002> Started scap: Backport for [[gerrit:992506|revertrisk: Fix i18n message reference (T348298)]], [[gerrit:992507|revertrisk: Fix i18n messages (T348298)]]

Mentioned in SAL (#wikimedia-operations) [2024-01-23T21:25:59Z] <kharlan@deploy2002> kharlan: Backport for [[gerrit:992506|revertrisk: Fix i18n message reference (T348298)]], [[gerrit:992507|revertrisk: Fix i18n messages (T348298)]] synced to the testservers (https://wikitech.wikimedia.org/wiki/Mwdebug)

Maintenance_bot removed a project: Patch-For-Review.Jan 23 2024, 9:30 PM

Mentioned in SAL (#wikimedia-operations) [2024-01-23T21:36:04Z] <kharlan@deploy2002> Finished scap: Backport for [[gerrit:992506|revertrisk: Fix i18n message reference (T348298)]], [[gerrit:992507|revertrisk: Fix i18n messages (T348298)]] (duration: 30m 51s)

ReleaseTaggerBot edited projects, added MW-1.42-notes (1.42.0-wmf.16; 2024-01-30); removed MW-1.42-notes (1.42.0-wmf.15; 2024-01-23).Jan 23 2024, 10:00 PM

JTannerWMF mentioned this in T356075: Swap ORES Scores for Revert Risk Model in Edit Patrol .Jan 29 2024, 4:29 PM

My recommendation is that we start lightweight with one filter level at 0.95. This should give a decent balance between accuracy and coverage without bogging us down too much in the details.

@KStoller-WMF suggested that we start by releasing to wikis which don't already have ORES support, as there will be no issues with too many filters. How feasible is this?

mirror-kt subscribed.Feb 22 2024, 5:47 PM

I've done a first pass update of https://www.mediawiki.org/wiki/Help:New_filters_for_edit_review/Quality_and_Intent_Filters to include info about RRLA.

Samwalton9-WMF removed a parent task: T352769: Automoderator should be able to revert edits based on Revert Risk scores from the ORES extension.Feb 23 2024, 2:44 PM

In T348298#9559718, @Samwalton9-WMF wrote:

My recommendation is that we start lightweight with one filter level at 0.95. This should give a decent balance between accuracy and coverage without bogging us down too much in the details.

To clarify, would 0.5-0.95 be flagged as "May be would be reverted" and >0.95 would be "Very likely to be reverted"? Existing ORES filters use "May be", "Likely" and "Very likely"

@KStoller-WMF suggested that we start by releasing to wikis which don't already have ORES support, as there will be no issues with too many filters. How feasible is this?

Can we pick one or two wikis as a trial run, to see how that will work in practice?

Also noting that T356280: Use multilingual revert risk for anonymous edits and T356281: Exclude first revision on page from scoring are probably blockers to releasing.

In T348298#9576023, @kostajh wrote:

In T348298#9559718, @Samwalton9-WMF wrote:

My recommendation is that we start lightweight with one filter level at 0.95. This should give a decent balance between accuracy and coverage without bogging us down too much in the details.

To clarify, would 0.5-0.95 be flagged as "May be would be reverted" and >0.95 would be "Very likely to be reverted"? Existing ORES filters use "May be", "Likely" and "Very likely"

Could we just have one level at >0.95 called something like "Likely to be reverted"?

In T348298#9576023, @kostajh wrote:

@KStoller-WMF suggested that we start by releasing to wikis which don't already have ORES support, as there will be no issues with too many filters. How feasible is this?

Can we pick one or two wikis as a trial run, to see how that will work in practice?

It sounds like ideally we can pick two wikis that don't have ORES support for edit quality or article quality filters.
https://ores-support-checklist.toolforge.org/ It looks like I can use this list to determine which wikis have no coverage, "basic" coverage or "advanced" model coverage. My assumption is that wikis that aren't listed don't have any coverage, but someone should please correct me if that seems mistaken!

Let me chat with @Johan to see if we could connect with two Product Ambassadors at wikis that have no coverage, so we have a point of contact for feedback.

kostajh mentioned this in T351897: Set the thresholds Revert Risk models to be used on the Recent Changes Feed (via ORES Extension).Feb 28 2024, 7:51 PM

@kostajh , to the best of my knowledge @KStoller-WMF is leading this project. We had a meeting on January and I gave my input there. I think other teams that have done community testing process can talk more about this. Technically we could go for targeting certain precision, what would involve different thresholds per wiki. Using the Knowledge Observatory data it should be easy to compute these numbers, however, maintenance could be hard, so my understanding was the decision was to go for a single threshold for all wikis.

jsn.sherman changed the status of subtask T356281: Exclude first revision on page from scoring from Open to In Progress.Mar 26 2024, 4:46 PM

jsn.sherman updated the task description. (Show Details)Mar 28 2024, 4:02 PM

Change #1014519 had a related patch set uploaded (by Jsn.sherman; author: Jsn.sherman):

[mediawiki/extensions/ORES@master] update revertrisk-language-agnostic min & desc

https://gerrit.wikimedia.org/r/1014519

gerritbot added a project: Patch-For-Review.Mar 28 2024, 4:26 PM

Change #1014519 merged by jenkins-bot:

[mediawiki/extensions/ORES@master] update revertrisk-language-agnostic min & desc

https://gerrit.wikimedia.org/r/1014519

ReleaseTaggerBot added a project: MW-1.43-notes (1.43.0-wmf.2; 2024-04-23).Apr 17 2024, 2:00 PM

HNordeenWMF mentioned this in T360581: [SPIKE] Test MGAD Model on LiftWing.Apr 17 2024, 4:37 PM

HNordeenWMF subscribed.

KStoller-WMF moved this task from Needs Discussion to Triaged on the Growth-Team board.Apr 23 2024, 3:57 PM

Jan_Kepler subscribed.May 19 2024, 5:06 PM

Samwalton9-WMF closed subtask T356281: Exclude first revision on page from scoring as Resolved.May 21 2024, 2:36 PM

Samwalton9-WMF mentioned this in T365701: Enable Revert Risk RecentChanges filter on id.wiki.May 23 2024, 12:34 PM

Samwalton9-WMF added a subtask: T365701: Enable Revert Risk RecentChanges filter on id.wiki.

Pppery removed a project: Patch-For-Review.Jun 12 2024, 3:34 AM

isarantopoulos removed isarantopoulos as the assignee of this task.Jun 28 2024, 3:06 PM

isarantopoulos moved this task from In Progress to Watching on the Machine-Learning-Team board.

Samwalton9-WMF changed the status of subtask T365701: Enable Revert Risk RecentChanges filter on id.wiki from Open to Stalled.Sep 16 2024, 4:28 PM

Test wiki on Patch demo by ISarantopoulos-WMF using patch(es) linked to this task was deleted:

~~https://patchdemo-legacy.wmcloud.org/wikis/15efa9543f/w/~~

Test wiki on Patch demo by ISarantopoulos-WMF using patch(es) linked to this task was deleted:

~~https://patchdemo-legacy.wmcloud.org/wikis/e1febe2e5d/w/~~

	F41444311: Screenshot 2023-11-03 at 6.56.24 PM.png
	Nov 4 2023, 1:59 AM

	F42144270: image.png
	Feb 26 2024, 11:29 AM

Add revertrisk-language-agnostic to RecentChanges filtersOpen, MediumPublicActions

Description

Details

Related ObjectsSearch...

Event Timeline

Add revertrisk-language-agnostic to RecentChanges filters
Open, MediumPublic
Actions

Related Objects
Search...