Feedback from T163197: Flagged revisions discussion: Current state and future plans.
On wikis where ORES predictions and FlaggedRevs are both enabled, consider to have good faith edits to be automatically marked as patrolled. This would decrease FlaggedRevs backlog, very high on some wikis.
| Status | Subtype | Assigned | Task | ||
|---|---|---|---|---|---|
| Open | None | T165848 Decrease FlaggedRevs backlog by using ORES predictions models | |||
| Resolved | Halfak | T166235 Flagged revs approve model to fiwiki | |||
| Open | None | T194589 Use data from Flagged Revisions to train ORES models | |||
| Resolved | Cirdan | T194594 Create FlaggedRevs data set for dewiki | |||
| Open | None | T196241 Verify and document reliable method to extract manually approved and rejected revisions from flaggedrevs and revision tables | |||
| Open | None | T198586 Enable the use of last flagged revision as parent revision in revscoring API extractor |
In wikis where the backlog is very long the backlog will need to be reseted somehow before it will work because one problem is that the there is old pending change and new edits will be pending too because that even if they were normally autoreviewed. Basic case is like this where there is trivial change blocks the automatic reviewing in newer edits.
Basic concept is solid though and it you could be using also the historical data too. Eg edit is automatically patrolled IF the edit is goodfaith AND if the last N edits from that user is good faith too.
If needed one safety check could be that if there is Abusefilter hits/added tags then do not automatically review it.
I tested the idea little bit. It would work with registered users pretty nicely, but least in fiwiki scores for ip-editors scores are generally lower than what they should be and it would need some help with that. In pl wiki if the scores are reliable it would work well.
Average value of scorer goodfaith (true) counted from last 5000 edits in selected wikipedias
lang all users autoreviewed non_autoreviewed ip ru 0.831 0.951 0.996 0.879 0.354 pt 0.751 0.904 0 0 0.495 pl 0.962 0.972 0.974 0.964 0.889 he 0.978 0.995 0 0 0.807 fi 0.818 0.943 0.979 0.818 0.395 fa 0.858 0.926 0 0 0.392 en 0.831 0.919 0 0 0.56 cs 0.862 0.935 0 0 0.341
@Trizek-WMF did you think that automatic review based on ORES would be realtime like current autopatrol / autoreview is OR something which is like cronjob which checks the review backlog periodically and reviews if there is something new which can be reviewed OR both?
I'm not familiar with those processes.
Like I understand them, the best would be to have something done immediately, to avoid backlog or usefulness log recording (flagged, and then unflagged).
For some languages a vandalized revision should not be shown if it is a biography, that is the article is in a tagged category, which imply the ORES prediction must have an effect immediately. ORES should then hide the revision pending further review.
A somewhat similar problem to T72452: Mark edit as pending review for all users from the same IP-address/range
Items in the backlog could be revisited if the log is too long.
See T72457: Set a limit on pages with forced review when there are too few active reviewers
Here is a proof-of-concept style pywikibot script for reviewing the Flagged Revs backlog which also works in other Wikipedias apart from the Finnish Wikipedia.
Rules used for approvals are:
What would be the next wiki to try? One of these, which have FlaggedRevsOverride true? https://tools.wmflabs.org/fiwiki-tools/fr_stats_example.html?db=fiwiki_p,huwiki_p,dewiki_p,plwiki_p&frs_key=pendingLag-average
Among them, idwiki and arwiki seem to have basic ORES support.
Based on https://quarry.wmflabs.org/query/19421, I asked one id.wiki user.
My request was posted by the local users in their village pumps, thanks!
https://id.wikipedia.org/w/index.php?title=Wikipedia:Warung_Kopi_(Teknis)&diff=12987288&oldid=12977169
https://ar.wikipedia.org/w/index.php?diff=prev&oldid=23471574
Thanks,
My guess is that the best use cases would plwiki and ruwiki. Both have advanced ORES support enabled and strong technical community for running and modifying the bot if they want to do so. Next after that would be huwiki which is currently doing ORES advanced support labeling. Ukwiki would be also interesting choice if Bunyk who was at the hackathon is interested for participating.
ORES advanced support is important because we can't really pull out the reviews from thin air and goodfaith scorer seems to be a lot better predictor than the reverted scorer.
Also a project page at the meta would be useful if we want to get different wikiprojects to participate.