Decrease FlaggedRevs backlog by using ORES predictions models
Open, Needs TriagePublic

Description

Feedback from T163197: Flagged revisions discussion: Current state and future plans.

On wikis where ORES predictions and FlaggedRevs are both enabled, consider to have good faith edits to be automatically marked as patrolled. This would decrease FlaggedRevs backlog, very high on some wikis.

Restricted Application added a project: Scoring-platform-team. · View Herald TranscriptMay 20 2017, 9:18 AM
Restricted Application added subscribers: TerraCodes, Aklapper. · View Herald Transcript
Zache added a comment.May 20 2017, 9:52 AM

In wikis where the backlog is very long the backlog will need to be reseted somehow before it will work because one problem is that the there is old pending change and new edits will be pending too because that even if they were normally autoreviewed. Basic case is like this where there is trivial change blocks the automatic reviewing in newer edits.

Basic concept is solid though and it you could be using also the historical data too. Eg edit is automatically patrolled IF the edit is goodfaith AND if the last N edits from that user is good faith too.

If needed one safety check could be that if there is Abusefilter hits/added tags then do not automatically review it.

I tested the idea little bit. It would work with registered users pretty nicely, but least in fiwiki scores for ip-editors scores are generally lower than what they should be and it would need some help with that. In pl wiki if the scores are reliable it would work well.

Average value of scorer goodfaith (true) counted from last 5000 edits in selected wikipedias

  • all = all editors
  • users = all registered editors
  • autoreviewed users = edits which were automatically approved with FlaggedFevs
  • non_autoreviewed = edits from registered editors which were not approved automatically
  • ip = edits from ip users
lang	 all  users autoreviewed non_autoreviewed ip
ru	0.831	0.951	0.996	0.879	0.354
pt	0.751	0.904	0	0	0.495
pl	0.962	0.972	0.974	0.964	0.889
he	0.978	0.995	0	0	0.807
fi	0.818	0.943	0.979	0.818	0.395
fa	0.858	0.926	0	0	0.392
en	0.831	0.919	0	0	0.56
cs	0.862	0.935	0	0	0.341

@Trizek-WMF did you think that automatic review based on ORES would be realtime like current autopatrol / autoreview is OR something which is like cronjob which checks the review backlog periodically and reviews if there is something new which can be reviewed OR both?

@Trizek-WMF did you think that automatic review based on ORES would be realtime like current autopatrol / autoreview is OR something which is like cronjob which checks the review backlog periodically and reviews if there is something new which can be reviewed OR both?

I'm not familiar with those processes.
Like I understand them, the best would be to have something done immediately, to avoid backlog or usefulness log recording (flagged, and then unflagged).

jeblad added a subscriber: jeblad.EditedMay 22 2017, 7:19 AM

For some languages a vandalized revision should not be shown if it is a biography, that is the article is in a tagged category, which imply the ORES prediction must have an effect immediately. ORES should then hide the revision pending further review.

Elitre added a subscriber: Elitre.May 23 2017, 10:30 AM
Zache added a comment.Jun 10 2017, 9:01 AM

Here is a proof-of-concept style pywikibot script for reviewing the Flagged Revs backlog which also works in other Wikipedias apart from the Finnish Wikipedia.

Rules used for approvals are:

  • approve if the edit was made by a user who was autoreviewed/patrolled
  • approve if the edit was made by a bot
  • approve if the edit was a revert or reverted
  • approve if the edit was an interviwiki change
  • approve if the edit was well scored using ORES

Here is a proof-of-concept style pywikibot script for reviewing the Flagged Revs backlog which also works in other Wikipedias apart from the Finnish Wikipedia.

Thanks! Log: https://fi.wikipedia.org/w/index.php?title=Toiminnot:Loki&offset=&limit=500&type=review&user=Zache&page=&tagfilter=&subtype=

What would be the next wiki to try? One of these, which have FlaggedRevsOverride true? https://tools.wmflabs.org/fiwiki-tools/fr_stats_example.html?db=fiwiki_p,huwiki_p,dewiki_p,plwiki_p&frs_key=pendingLag-average
Among them, idwiki and arwiki seem to have basic ORES support.

Based on https://quarry.wmflabs.org/query/19421, I asked one id.wiki user.

Meno25 added a subscriber: Meno25.Jun 10 2017, 9:32 PM

Thanks,

My guess is that the best use cases would plwiki and ruwiki. Both have advanced ORES support enabled and strong technical community for running and modifying the bot if they want to do so. Next after that would be huwiki which is currently doing ORES advanced support labeling. Ukwiki would be also interesting choice if Bunyk who was at the hackathon is interested for participating.

ORES advanced support is important because we can't really pull out the reviews from thin air and goodfaith scorer seems to be a lot better predictor than the reverted scorer.

Also a project page at the meta would be useful if we want to get different wikiprojects to participate.

awight moved this task from Backlog to Monitor on the Scoring-platform-team board.Wed, Sep 6, 5:26 PM