Change 992423 had a related patch set uploaded (by Kosta Harlan; author: Kosta Harlan):

[operations/mediawiki-config@master] ORES: Enable renamed revertrisklanguageagnostic model

https://gerrit.wikimedia.org/r/992423

Change 992423 merged by jenkins-bot:

[operations/mediawiki-config@master] ORES: Enable renamed revertrisklanguageagnostic model

https://gerrit.wikimedia.org/r/992423

Mentioned in SAL (#wikimedia-operations) [2024-01-23T14:47:59Z] <logmsgbot> lucaswerkmeister-wmde@deploy2002 Started scap: Backport for [[gerrit:992423|ORES: Enable renamed revertrisklanguageagnostic model (T348298)]]

Mentioned in SAL (#wikimedia-operations) [2024-01-23T14:49:28Z] <logmsgbot> lucaswerkmeister-wmde@deploy2002 lucaswerkmeister-wmde and kharlan: Backport for [[gerrit:992423|ORES: Enable renamed revertrisklanguageagnostic model (T348298)]] synced to the testservers (https://wikitech.wikimedia.org/wiki/Mwdebug)

Mentioned in SAL (#wikimedia-operations) [2024-01-23T14:59:20Z] <logmsgbot> lucaswerkmeister-wmde@deploy2002 Finished scap: Backport for [[gerrit:992423|ORES: Enable renamed revertrisklanguageagnostic model (T348298)]] (duration: 11m 20s)

Change 992492 had a related patch set uploaded (by Kosta Harlan; author: Kosta Harlan):

[mediawiki/extensions/ORES@master] revertrisk: Fix i18n message

https://gerrit.wikimedia.org/r/992492

Change 992493 had a related patch set uploaded (by Kosta Harlan; author: Kosta Harlan):

[mediawiki/extensions/ORES@master] revertrisk: Fix i18n message reference

https://gerrit.wikimedia.org/r/992493

Change 983681 abandoned by Kosta Harlan:

[mediawiki/extensions/ORES@master] enable revertrisk

Reason:

See I633a7c37326202461d9de2ef20ef8a5f0940f3aa

https://gerrit.wikimedia.org/r/983681

Change 992493 merged by jenkins-bot:

[mediawiki/extensions/ORES@master] revertrisk: Fix i18n message reference

https://gerrit.wikimedia.org/r/992493

Change 992506 had a related patch set uploaded (by Kosta Harlan; author: Kosta Harlan):

[mediawiki/extensions/ORES@wmf/1.42.0-wmf.15] revertrisk: Fix i18n message reference

https://gerrit.wikimedia.org/r/992506

Change 992492 merged by jenkins-bot:

[mediawiki/extensions/ORES@master] revertrisk: Fix i18n messages

https://gerrit.wikimedia.org/r/992492

Change 992507 had a related patch set uploaded (by Kosta Harlan; author: Kosta Harlan):

[mediawiki/extensions/ORES@wmf/1.42.0-wmf.15] revertrisk: Fix i18n messages

https://gerrit.wikimedia.org/r/992507

Change 992506 merged by jenkins-bot:

[mediawiki/extensions/ORES@wmf/1.42.0-wmf.15] revertrisk: Fix i18n message reference

https://gerrit.wikimedia.org/r/992506

Change 992507 merged by jenkins-bot:

[mediawiki/extensions/ORES@wmf/1.42.0-wmf.15] revertrisk: Fix i18n messages

https://gerrit.wikimedia.org/r/992507

Mentioned in SAL (#wikimedia-operations) [2024-01-23T21:05:12Z] <kharlan@deploy2002> Started scap: Backport for [[gerrit:992506|revertrisk: Fix i18n message reference (T348298)]], [[gerrit:992507|revertrisk: Fix i18n messages (T348298)]]

Mentioned in SAL (#wikimedia-operations) [2024-01-23T21:25:59Z] <kharlan@deploy2002> kharlan: Backport for [[gerrit:992506|revertrisk: Fix i18n message reference (T348298)]], [[gerrit:992507|revertrisk: Fix i18n messages (T348298)]] synced to the testservers (https://wikitech.wikimedia.org/wiki/Mwdebug)

Mentioned in SAL (#wikimedia-operations) [2024-01-23T21:36:04Z] <kharlan@deploy2002> Finished scap: Backport for [[gerrit:992506|revertrisk: Fix i18n message reference (T348298)]], [[gerrit:992507|revertrisk: Fix i18n messages (T348298)]] (duration: 30m 51s)

My recommendation is that we start lightweight with one filter level at 0.95. This should give a decent balance between accuracy and coverage without bogging us down too much in the details.

@KStoller-WMF suggested that we start by releasing to wikis which don't already have ORES support, as there will be no issues with too many filters. How feasible is this?

In T348298#9559718, @Samwalton9-WMF wrote:

My recommendation is that we start lightweight with one filter level at 0.95. This should give a decent balance between accuracy and coverage without bogging us down too much in the details.

To clarify, would 0.5-0.95 be flagged as "May be would be reverted" and >0.95 would be "Very likely to be reverted"? Existing ORES filters use "May be", "Likely" and "Very likely"

@KStoller-WMF suggested that we start by releasing to wikis which don't already have ORES support, as there will be no issues with too many filters. How feasible is this?

Can we pick one or two wikis as a trial run, to see how that will work in practice?

Also noting that T356280: Use multilingual revert risk for anonymous edits and T356281: Exclude first revision on page from scoring are probably blockers to releasing.

In T348298#9576023, @kostajh wrote:

In T348298#9559718, @Samwalton9-WMF wrote:

My recommendation is that we start lightweight with one filter level at 0.95. This should give a decent balance between accuracy and coverage without bogging us down too much in the details.

To clarify, would 0.5-0.95 be flagged as "May be would be reverted" and >0.95 would be "Very likely to be reverted"? Existing ORES filters use "May be", "Likely" and "Very likely"

Could we just have one level at >0.95 called something like "Likely to be reverted"?

In T348298#9576023, @kostajh wrote:

@KStoller-WMF suggested that we start by releasing to wikis which don't already have ORES support, as there will be no issues with too many filters. How feasible is this?

Can we pick one or two wikis as a trial run, to see how that will work in practice?

It sounds like ideally we can pick two wikis that don't have ORES support for edit quality or article quality filters.
https://ores-support-checklist.toolforge.org/ It looks like I can use this list to determine which wikis have no coverage, "basic" coverage or "advanced" model coverage. My assumption is that wikis that aren't listed don't have any coverage, but someone should please correct me if that seems mistaken!

Let me chat with @Johan to see if we could connect with two Product Ambassadors at wikis that have no coverage, so we have a point of contact for feedback.

@kostajh , to the best of my knowledge @KStoller-WMF is leading this project. We had a meeting on January and I gave my input there. I think other teams that have done community testing process can talk more about this. Technically we could go for targeting certain precision, what would involve different thresholds per wiki. Using the Knowledge Observatory data it should be easy to compute these numbers, however, maintenance could be hard, so my understanding was the decision was to go for a single threshold for all wikis.

Change #1014519 had a related patch set uploaded (by Jsn.sherman; author: Jsn.sherman):

[mediawiki/extensions/ORES@master] update revertrisk-language-agnostic min & desc

https://gerrit.wikimedia.org/r/1014519

Change #1014519 merged by jenkins-bot:

[mediawiki/extensions/ORES@master] update revertrisk-language-agnostic min & desc

https://gerrit.wikimedia.org/r/1014519

Test wiki on Patch demo by ISarantopoulos-WMF using patch(es) linked to this task was deleted:

https://patchdemo-legacy.wmcloud.org/wikis/15efa9543f/w/

Test wiki on Patch demo by ISarantopoulos-WMF using patch(es) linked to this task was deleted:

https://patchdemo-legacy.wmcloud.org/wikis/e1febe2e5d/w/

Adding the thresholds we arrived at from the analysis that was completed for idwiki;

Language Agnostic:

Multilingual:

I'm thinking since we're not confident about the middle threshold we could remove it from the filters entirely and have two thresholds we're more confident about:

Likely to be reverted:

• This would map to the threshold 0.85 and above for language agnostic and 0.58 and above for multilingual.

Unlikely to be reverted:

• This would map to the threshold 0.26 and below for language agnostic and 0.44 and below for multilingual.

In T348298#10694988, @Kgraessle wrote:

Adding the thresholds we arrived at from the analysis that was completed for idwiki;

Language Agnostic:

Multilingual:

I'm thinking since we're not confident about the middle threshold we could remove it from the filters entirely and have two thresholds we're more confident about:

Likely to be reverted:

• This would map to the threshold 0.85 and above for language agnostic and 0.58 and above for multilingual.

Unlikely to be reverted:

• This would map to the threshold 0.26 and below for language agnostic and 0.44 and below for multilingual.

@Kgraessle are you planning to pick one or the other model as the backend? I think it might be confusing for end users to see both language agnostic and multilingual as choices.

In the tables for each model, it would be very useful to see how contributions from logged-out users are impacted.

I agree with not showing the middle threshold, and just having "likely to be reverted" and "unlikely to be reverted".

In T348298#10695756, @kostajh wrote:

In T348298#10694988, @Kgraessle wrote:

Adding the thresholds we arrived at from the analysis that was completed for idwiki;

Language Agnostic:

Multilingual:

I'm thinking since we're not confident about the middle threshold we could remove it from the filters entirely and have two thresholds we're more confident about:

Likely to be reverted:

• This would map to the threshold 0.85 and above for language agnostic and 0.58 and above for multilingual.

Unlikely to be reverted:

• This would map to the threshold 0.26 and below for language agnostic and 0.44 and below for multilingual.

@Kgraessle are you planning to pick one or the other model as the backend? I think it might be confusing for end users to see both language agnostic and multilingual as choices.

In the tables for each model, it would be very useful to see how contributions from logged-out users are impacted.

I agree with not showing the middle threshold, and just having "likely to be reverted" and "unlikely to be reverted".

Yep, we're planning on picking the model that works best for the wiki. (similar to what we plan on doing for AutoModerator)

I added the percentage of anon users to the tables:

Let me know what you think.

In my opinion, "unlikely to be reverted" is unlikely to be of any use for patrollers (it's harmless, though). In fact, patrollers would (besides the "very likely to be reverted") benefit most from a filter that excludes edits below a threshold instead (i.e., edits unlikely to be reverted). Such a filter would increase the efficiency of patrolling as it would prevent recent changes from presenting edits that patrollers (probably) don't have to deal with while keeping a reasonable recall (85-95%).

See also the diagram on Help:New filters for edit review/Quality and Intent Filters. What I suggest is having a filter(s) for the orange or yellow (or both) triangle.

Also, note that the report sheets are somewhat inconsistent. "Language Agnostic Model" reports recall per interval (hence, the sum of recalls is almost 1), whereas MRRM reports recall per threshold (hence, the lowest one yields almost 1).

In T348298#10746593, @matej_suchanek wrote:

In my opinion, "unlikely to be reverted" is unlikely to be of any use for patrollers (it's harmless, though). In fact, patrollers would (besides the "very likely to be reverted") benefit most from a filter that excludes edits below a threshold instead (i.e., edits unlikely to be reverted). Such a filter would increase the efficiency of patrolling as it would prevent recent changes from presenting edits that patrollers (probably) don't have to deal with while keeping a reasonable recall (85-95%).

See also the diagram on Help:New filters for edit review/Quality and Intent Filters. What I suggest is having a filter(s) for the orange or yellow (or both) triangle.

Also, note that the report sheets are somewhat inconsistent. "Language Agnostic Model" reports recall per interval (hence, the sum of recalls is almost 1), whereas MRRM reports recall per threshold (hence, the lowest one yields almost 1).

@matej_suchanek Great catch! I have updated the multilingual recall and precision scores to match recall/precision per interval as we've done with language agnostic.

@Ladsgroup @tstarling We'd like to move forward with this by early Q3. In the context of the WE1.4 RC performance work I wondered if this is blocked by RecentChanges performance, and if so, what might we be able to do to alleviate performance concerns?

From my side: as long as the other rc models are disabled and shut down (damaging and goodfaith), I think it'll be fine. Commons is somewhat unstable for other reasons so please avoid that for now (I hope that's fine?)

In T348298#11202425, @Ladsgroup wrote:

From my side: as long as the other rc models are disabled and shut down (damaging and goodfaith), I think it'll be fine. Commons is somewhat unstable for other reasons so please avoid that for now (I hope that's fine?)

This would only be in the context of Wikipedias, so that part is fine.

In this case, would we be OK to deploy Revert Risk to wikis that don't have damaging/goodfaith models?

That should be fine but keep me in the loop so I can monitor slow queries and might ask for rollback in some certain wikis if thing don't go as planned. Hopefully it shouldn't be an issue given the improvements done in this Q.