We've come to realise that this task is important for both the Android team's work on patrolling and the Moderator Tools team's work on Automoderator, so I just want to make sure we're all on the same page following the discussion here, in T329071 (duplicate?), and on Slack. This is my understanding of the situation, please let me know if anything is inaccurate:

• Getting Revert Risk into the ORES extension wasn't an officially roadmapped plan, but @isarantopoulos hacked on this with @achou and needs some support finalising the patch.
• @kostajh is providing code review on the patch, and @jsn.sherman is also happy to help.
• This patch meets both Moderator Tools and Android team needs in terms of having Revert Risk scores available via the ORES extension.
  • Scores are stored for 30 days (because the extension stores for recentchanges rows).
• This patch also adds Revert Risk to the user-facing Recent Changes feed, but we don't currently have any plan/CRS support to announce this to communities.
  • There was discussion on Slack of rolling this out to a few wikis first, have any decisions been made around this?

Is the above correct? @KStoller-WMF and I are happy to PM-hat on this to help with rollout :)

A quick question I have is what score threshold is the recent changes filter using for 'likely to be reverted'? Glancing at the patch it looks like maybe 0.5, which feels low to me given that we're looking at ranges above 0.97 for Automoderator (though we need to be much more certain there, obviously).

In T348298#9362986, @Samwalton9-WMF wrote:
This patch also adds Revert Risk to the user-facing Recent Changes feed, but we don't currently have any plan/CRS support to announce this to communities.

• There was discussion on Slack of rolling this out to a few wikis first, have any decisions been made around this?

I don't think any decision was made, except that this should likely wait until early 2024.
Sounds like we should discuss with CRS, create subtasks, and post a tentative plan here.

I also changed the order of the filters in order for revertrisk to appear before the other models as requested by @Samwalton9-WMF

Change 971547 merged by jenkins-bot:

[mediawiki/extensions/ORES@master] Add revertrisk model to the list of models

https://gerrit.wikimedia.org/r/971547

Change 980539 had a related patch set uploaded (by Kosta Harlan; author: Kosta Harlan):

[mediawiki/extensions/ORES@master] LiftWingService: Set Content-Type header for requests to RevertRisk

https://gerrit.wikimedia.org/r/980539

Change 980539 merged by jenkins-bot:

[mediawiki/extensions/ORES@master] LiftWingService: Set Content-Type header for requests to RevertRisk

https://gerrit.wikimedia.org/r/980539

Change 981337 had a related patch set uploaded (by Ilias Sarantopoulos; author: Ilias Sarantopoulos):

[operations/mediawiki-config@master] [beta] ores-extension: enable revertrisk model for enwiki

https://gerrit.wikimedia.org/r/981337

Change 981337 merged by jenkins-bot:

[operations/mediawiki-config@master] [beta] ores-extension: enable revertrisk model for enwiki

https://gerrit.wikimedia.org/r/981337

Mentioned in SAL (#wikimedia-operations) [2023-12-07T21:13:52Z] <jdrewniak@deploy2002> Started scap: Backport for [[gerrit:980951|Enable Vector beta feature for all wikis (T351339)]], [[gerrit:981337|[beta] ores-extension: enable revertrisk model for enwiki (T348298)]], [[gerrit:976911|Enable action blocks in Serbian Wikipedia (T351873)]]

Mentioned in SAL (#wikimedia-operations) [2023-12-07T21:15:13Z] <jdrewniak@deploy2002> zoranzoki21 and isaranto and jdlrobson and jdrewniak: Backport for [[gerrit:980951|Enable Vector beta feature for all wikis (T351339)]], [[gerrit:981337|[beta] ores-extension: enable revertrisk model for enwiki (T348298)]], [[gerrit:976911|Enable action blocks in Serbian Wikipedia (T351873)]] synced to the testservers (https://wikitech.wikimedia.org/wiki/Mwdebug)

Mentioned in SAL (#wikimedia-operations) [2023-12-07T21:23:46Z] <jdrewniak@deploy2002> Finished scap: Backport for [[gerrit:980951|Enable Vector beta feature for all wikis (T351339)]], [[gerrit:981337|[beta] ores-extension: enable revertrisk model for enwiki (T348298)]], [[gerrit:976911|Enable action blocks in Serbian Wikipedia (T351873)]] (duration: 09m 54s)

Can I confirm the status of this patch? I see some merged patches, but was this only for Beta?

Yes this was only for Beta, but failed. I need to debug it and resubmit a new one Will do so in the following days and post a status update here.

Change 983681 had a related patch set uploaded (by Ilias Sarantopoulos; author: Ilias Sarantopoulos):

[mediawiki/extensions/ORES@master] enable revertrisk

https://gerrit.wikimedia.org/r/983681

Test wiki on Patch demo by ISarantopoulos-WMF using patch(es) linked to this task was deleted:

https://patchdemo.wmflabs.org/wikis/9685c9655b/w/

Test wiki created on Patch demo by ISarantopoulos-WMF using patch(es) linked to this task:
https://patchdemo.wmflabs.org/wikis/989d97cc42/w

Change 983724 had a related patch set uploaded (by Ladsgroup; author: Amir Sarabadani):

[operations/mediawiki-config@master] Beta: Move the override of ores LW URL to CS-labs.php

https://gerrit.wikimedia.org/r/983724

Change 983724 merged by jenkins-bot:

[operations/mediawiki-config@master] Beta: Move the override of ores LW URL to CS-labs.php

https://gerrit.wikimedia.org/r/983724

Change 983734 had a related patch set uploaded (by Ladsgroup; author: Ladsgroup):

[operations/mediawiki-config@master] Revert "Revert "[beta] ores-extension: enable revertrisk model for enwiki""

https://gerrit.wikimedia.org/r/983734

Change 983734 merged by jenkins-bot:

[operations/mediawiki-config@master] Revert "Revert "[beta] ores-extension: enable revertrisk model for enwiki""

https://gerrit.wikimedia.org/r/983734

Change 983747 had a related patch set uploaded (by Ilias Sarantopoulos; author: Ilias Sarantopoulos):

[operations/mediawiki-config@master] testwiki: enable revertris kmodel in ores extension

https://gerrit.wikimedia.org/r/983747

After getting some help from @Ladsgroup we figured out the ores extension was long broken on the beta cluster. I manually triggered a job through the terminal and results came through as expected.
Will proceed to deploy and try it out on testwiki.

Thanks Amir!

I noticed the order of the filters isn't as I expected it to be (Revertrisk filter on top) so I'll look into that again. If anyone mediawiki expert knows how the order is enforce in the UI of the recent changes page let me know!

Just put it at the top of the array in OresModels, that should be enoughTM

Test wiki on Patch demo by ISarantopoulos-WMF using patch(es) linked to this task was deleted:

https://patchdemo.wmflabs.org/wikis/989d97cc42/w/

Test wiki created on Patch demo by ISarantopoulos-WMF using patch(es) linked to this task:
https://patchdemo.wmflabs.org/wikis/e1febe2e5d/w

Change 983747 merged by jenkins-bot:

[operations/mediawiki-config@master] testwiki: enable revertrisk model in ores extension

https://gerrit.wikimedia.org/r/983747

Mentioned in SAL (#wikimedia-operations) [2023-12-19T14:05:34Z] <lucaswerkmeister-wmde@deploy2002> Started scap: Backport for [[gerrit:983747|testwiki: enable revertrisk model in ores extension (T348298)]]

Mentioned in SAL (#wikimedia-operations) [2023-12-19T14:08:02Z] <lucaswerkmeister-wmde@deploy2002> isaranto and lucaswerkmeister-wmde: Backport for [[gerrit:983747|testwiki: enable revertrisk model in ores extension (T348298)]] synced to the testservers (https://wikitech.wikimedia.org/wiki/Mwdebug)

Mentioned in SAL (#wikimedia-operations) [2023-12-19T14:15:56Z] <lucaswerkmeister-wmde@deploy2002> Finished scap: Backport for [[gerrit:983747|testwiki: enable revertrisk model in ores extension (T348298)]] (duration: 10m 22s)

@isarantopoulos The filter currently doesn't work, because in DatabaseQueryBuilder.php, validateSelected returns false, so there's no WHERE clause for revert risk results added to the query. I think the reason for this is that the other models have different thresholds for their results, while revertrisk is defined as a boolean "likely to be reverted" or not. I am not sure, but it seems like the easiest fix is to update the code to align with the threshold categories used in the`goodfaith` and damaging models (e.g. "likelybad", "maybebad", "verylikelybad").

Test wiki created on Patch demo by ISarantopoulos-WMF using patch(es) linked to this task:
https://patchdemo.wmflabs.org/wikis/15efa9543f/w

Change 988048 had a related patch set uploaded (by Kosta Harlan; author: Kosta Harlan):

[mediawiki/extensions/ORES@master] WIP revertrisk: Add thresholds for maybe/likely/very likely bad

https://gerrit.wikimedia.org/r/988048

I also found a bug but I'm not sure yet if fixing it would do the trick. In this function call the model name wasn't defined correctly. I have an open patch which I was testing with patchdemo but no revisions seem to be flagged over there. I'm also trying it out locally to see if anything changes.

Change 988048 merged by jenkins-bot:

[mediawiki/extensions/ORES@master] revertrisk: Rename model to use single lowercase word

https://gerrit.wikimedia.org/r/988048

Change 992423 had a related patch set uploaded (by Kosta Harlan; author: Kosta Harlan):

[operations/mediawiki-config@master] ORES: Enable renamed revertrisklanguageagnostic model

https://gerrit.wikimedia.org/r/992423

Change 992423 merged by jenkins-bot:

[operations/mediawiki-config@master] ORES: Enable renamed revertrisklanguageagnostic model

https://gerrit.wikimedia.org/r/992423

Mentioned in SAL (#wikimedia-operations) [2024-01-23T14:47:59Z] <logmsgbot> lucaswerkmeister-wmde@deploy2002 Started scap: Backport for [[gerrit:992423|ORES: Enable renamed revertrisklanguageagnostic model (T348298)]]

Mentioned in SAL (#wikimedia-operations) [2024-01-23T14:49:28Z] <logmsgbot> lucaswerkmeister-wmde@deploy2002 lucaswerkmeister-wmde and kharlan: Backport for [[gerrit:992423|ORES: Enable renamed revertrisklanguageagnostic model (T348298)]] synced to the testservers (https://wikitech.wikimedia.org/wiki/Mwdebug)

Mentioned in SAL (#wikimedia-operations) [2024-01-23T14:59:20Z] <logmsgbot> lucaswerkmeister-wmde@deploy2002 Finished scap: Backport for [[gerrit:992423|ORES: Enable renamed revertrisklanguageagnostic model (T348298)]] (duration: 11m 20s)

Change 992492 had a related patch set uploaded (by Kosta Harlan; author: Kosta Harlan):

[mediawiki/extensions/ORES@master] revertrisk: Fix i18n message

https://gerrit.wikimedia.org/r/992492

Change 992493 had a related patch set uploaded (by Kosta Harlan; author: Kosta Harlan):

[mediawiki/extensions/ORES@master] revertrisk: Fix i18n message reference

https://gerrit.wikimedia.org/r/992493

Change 983681 abandoned by Kosta Harlan:

[mediawiki/extensions/ORES@master] enable revertrisk

Reason:

See I633a7c37326202461d9de2ef20ef8a5f0940f3aa

https://gerrit.wikimedia.org/r/983681

Change 992493 merged by jenkins-bot:

[mediawiki/extensions/ORES@master] revertrisk: Fix i18n message reference

https://gerrit.wikimedia.org/r/992493

Change 992506 had a related patch set uploaded (by Kosta Harlan; author: Kosta Harlan):

[mediawiki/extensions/ORES@wmf/1.42.0-wmf.15] revertrisk: Fix i18n message reference

https://gerrit.wikimedia.org/r/992506

Change 992492 merged by jenkins-bot:

[mediawiki/extensions/ORES@master] revertrisk: Fix i18n messages

https://gerrit.wikimedia.org/r/992492

Change 992507 had a related patch set uploaded (by Kosta Harlan; author: Kosta Harlan):

[mediawiki/extensions/ORES@wmf/1.42.0-wmf.15] revertrisk: Fix i18n messages

https://gerrit.wikimedia.org/r/992507

Change 992506 merged by jenkins-bot:

[mediawiki/extensions/ORES@wmf/1.42.0-wmf.15] revertrisk: Fix i18n message reference

https://gerrit.wikimedia.org/r/992506

Change 992507 merged by jenkins-bot:

[mediawiki/extensions/ORES@wmf/1.42.0-wmf.15] revertrisk: Fix i18n messages

https://gerrit.wikimedia.org/r/992507

Mentioned in SAL (#wikimedia-operations) [2024-01-23T21:05:12Z] <kharlan@deploy2002> Started scap: Backport for [[gerrit:992506|revertrisk: Fix i18n message reference (T348298)]], [[gerrit:992507|revertrisk: Fix i18n messages (T348298)]]

Mentioned in SAL (#wikimedia-operations) [2024-01-23T21:25:59Z] <kharlan@deploy2002> kharlan: Backport for [[gerrit:992506|revertrisk: Fix i18n message reference (T348298)]], [[gerrit:992507|revertrisk: Fix i18n messages (T348298)]] synced to the testservers (https://wikitech.wikimedia.org/wiki/Mwdebug)

Mentioned in SAL (#wikimedia-operations) [2024-01-23T21:36:04Z] <kharlan@deploy2002> Finished scap: Backport for [[gerrit:992506|revertrisk: Fix i18n message reference (T348298)]], [[gerrit:992507|revertrisk: Fix i18n messages (T348298)]] (duration: 30m 51s)

My recommendation is that we start lightweight with one filter level at 0.95. This should give a decent balance between accuracy and coverage without bogging us down too much in the details.

@KStoller-WMF suggested that we start by releasing to wikis which don't already have ORES support, as there will be no issues with too many filters. How feasible is this?

In T348298#9559718, @Samwalton9-WMF wrote:

My recommendation is that we start lightweight with one filter level at 0.95. This should give a decent balance between accuracy and coverage without bogging us down too much in the details.

To clarify, would 0.5-0.95 be flagged as "May be would be reverted" and >0.95 would be "Very likely to be reverted"? Existing ORES filters use "May be", "Likely" and "Very likely"

@KStoller-WMF suggested that we start by releasing to wikis which don't already have ORES support, as there will be no issues with too many filters. How feasible is this?

Can we pick one or two wikis as a trial run, to see how that will work in practice?

Also noting that T356280: Use multilingual revert risk for anonymous edits and T356281: Exclude first revision on page from scoring are probably blockers to releasing.

In T348298#9576023, @kostajh wrote:

In T348298#9559718, @Samwalton9-WMF wrote:

My recommendation is that we start lightweight with one filter level at 0.95. This should give a decent balance between accuracy and coverage without bogging us down too much in the details.

To clarify, would 0.5-0.95 be flagged as "May be would be reverted" and >0.95 would be "Very likely to be reverted"? Existing ORES filters use "May be", "Likely" and "Very likely"

Could we just have one level at >0.95 called something like "Likely to be reverted"?

In T348298#9576023, @kostajh wrote:

@KStoller-WMF suggested that we start by releasing to wikis which don't already have ORES support, as there will be no issues with too many filters. How feasible is this?

Can we pick one or two wikis as a trial run, to see how that will work in practice?

It sounds like ideally we can pick two wikis that don't have ORES support for edit quality or article quality filters.
https://ores-support-checklist.toolforge.org/ It looks like I can use this list to determine which wikis have no coverage, "basic" coverage or "advanced" model coverage. My assumption is that wikis that aren't listed don't have any coverage, but someone should please correct me if that seems mistaken!

Let me chat with @Johan to see if we could connect with two Product Ambassadors at wikis that have no coverage, so we have a point of contact for feedback.

@kostajh , to the best of my knowledge @KStoller-WMF is leading this project. We had a meeting on January and I gave my input there. I think other teams that have done community testing process can talk more about this. Technically we could go for targeting certain precision, what would involve different thresholds per wiki. Using the Knowledge Observatory data it should be easy to compute these numbers, however, maintenance could be hard, so my understanding was the decision was to go for a single threshold for all wikis.

Change #1014519 had a related patch set uploaded (by Jsn.sherman; author: Jsn.sherman):

[mediawiki/extensions/ORES@master] update revertrisk-language-agnostic min & desc

https://gerrit.wikimedia.org/r/1014519

Change #1014519 merged by jenkins-bot:

[mediawiki/extensions/ORES@master] update revertrisk-language-agnostic min & desc

https://gerrit.wikimedia.org/r/1014519