Page MenuHomePhabricator

Selective patrol: an AI-based system to prioritize patrolling of edits
Open, Needs TriagePublic

Description

Currently, wikis have the choice of implementing either new page patrol that allows patrolling new pages only, or full recent changes patrol. The problem of full RC patrol is that on big wikis, the absolutely massive number of edits to patrol makes the task almost impossible (the patrol of new articles is already heavily backlogged on enwiki). There is no way to find out (besides edit summaries and size changes) which edits need patrolling the most.

With ORES, we have an additional indication that an edit is in particular need of patrol, but on wikis not using RC patrol, we still can't patrol the edits.
Change tags allow to flag an edit in a particular way, and can help prioritize patrol, but we have the same problem as above.
With deferred changes, we have an effective way to patrol those edits, but for various reasons the backlog must be kept low. So it is good to prioritize patrol of edits identified as highly suspicious by AIs, but not for those edits identified as somewhat suspicious.

Therefore, we need a way to:

  • on wikis using full RC patrol UI, highlight some edits as being especially in need of patrol, end users would there have three filtering options: none, unpatrolled, needs patrol
  • on wikis not using full RC patrol UI, make it possible to patrol edits considered especially in need of patrol, end users would there have two filtering options: none, needs patrol.

It should be possible to mark an edit as being in need of patrol either via: extensions: ORES, as it does already but this could be integrated in this framework, AbuseFilter, and via the API for bots. In addition FlaggedRevs should mark the edit as needing patrol when it defers an edit.

So we would need three states of patrol: -1 for 'needs patrol', 0 for 'unpatrolled' and +1 for 'patrolled'. Unfortunately rc_patrolled is unsigned in the recentchange table, but if we can fix this, the implementation should be relatively straightforward.

Event Timeline

Cenarium created this task.Feb 9 2017, 6:49 PM
Restricted Application added a project: Scoring-platform-team. · View Herald TranscriptFeb 9 2017, 6:49 PM
Restricted Application added a subscriber: Aklapper. · View Herald Transcript
Cenarium updated the task description. (Show Details)Feb 17 2017, 4:54 PM

@Cenarium wrote that "we need a way to"

on wikis using full RC patrol UI, highlight some edits as being especially in need of patrol, end users would there have three filtering options: none, unpatrolled, needs patrol
on wikis not using full RC patrol UI, make it possible to patrol edits considered especially in need of patrol, end users would there have two filtering options: none, needs patrol.
It should be possible to mark an edit as being in need of patrol either via: extensions: ORES, as it does already but this could be integrated in this framework, AbuseFilter, and via the API for bots. In addition FlaggedRevs should mark the edit as needing patrol when it defers an edit.
So we would need three states of patrol: -1 for 'needs patrol', 0 for 'unpatrolled' and +1 for 'patrolled'. Unfortunately rc_patrolled is unsigned in the recentchange table, but if we can fix this, the implementation should be relatively straightforward.

You will be happy to learn that I think we have this covered in the Edit Review Improvements/RC Page Improvements project. All we have to do is turn on RC Patrol, and then we're already building everything else you need.

As part of our RC Page Improvements, we are implementing:

  • A filter for patrolled/unpatrolled
  • Various ORES filters that will enable users to find the edits that are most in need of review.
  • User-defined highlighting as an alternative to or in addition to filtering, so users can decide which edit properties they want to spotlight.

Using these tools, which will be out this spring, users can easily find edits that are likely damaging and unpatrolled. (See this video for a demonstration of the tools generally.)

Elsewhere you've noted:

The biggest obstacle to turning on RC patrol at enwiki is that the vast majority of edits would go unpatrolled due to the sheer number of edits and they would thus clutter recent changes and watchlists, while not being helpful to patrollers since those patrollers would have no idea which edits need patrolling the most.

I gather that this is partly a visual display issue (users in the past objected to the red ! that currently denotes an unpatrolled edit) and partly because seeing big backlogs makes people feel bad. Is that right?

If those are the issues, would it help if, at least on RC Page and related pages, we just turned on RC Patrol but didn't show an indication for patrol status? Patrol status would be just like most other edit properties; if you want to see it, you can use the highlighting or the filtering to find it. If not, ignore it. I'd really like to turn RC Patrol on. Do you think this approach has promise, because it would be easy for us to do I think?

If those are the issues, would it help if, at least on RC Page and related pages, we just turned on RC Patrol but didn't show an indication for patrol status? Patrol status would be just like most other edit properties; if you want to see it, you can use the highlighting or the filtering to find it. If not, ignore it. I'd really like to turn RC Patrol on. Do you think this approach has promise, because it would be easy for us to do I think?

If we were to do this mostly concerned about enwiki, we could just temporarily add a line to Common.css hiding it, and later remove once users who definitely don't want it can add to their personal CSS (or a gadget or whatever). This would be quick and easy and not add additional technical debt to the current patrol flag code.

As part of our RC Page Improvements, we are implementing:

  • A filter for patrolled/unpatrolled
  • Various ORES filters that will enable users to find the edits that are most in need of review.
  • User-defined highlighting as an alternative to or in addition to filtering, so users can decide which edit properties they want to spotlight.

Using these tools, which will be out this spring, users can easily find edits that are likely damaging and unpatrolled. (See this video for a demonstration of the tools generally.)

This sounds promising, though I'd still like a way to integrate other sources than ORES, such as AbuseFilter and bots.

This looks like it would be addressed to the "power" user too, we'd need to expose a default to the "average" user.

I gather that this is partly a visual display issue (users in the past objected to the red ! that currently denotes an unpatrolled edit) and partly because seeing big backlogs makes people feel bad. Is that right?

Red exclamation marks aren't a problem I think, as long as they don't appear on too many edits but only on the ones that need the most attention.

If those are the issues, would it help if, at least on RC Page and related pages, we just turned on RC Patrol but didn't show an indication for patrol status? Patrol status would be just like most other edit properties; if you want to see it, you can use the highlighting or the filtering to find it. If not, ignore it. I'd really like to turn RC Patrol on. Do you think this approach has promise, because it would be easy for us to do I think?

I think we need a middle ground for the default, otherwise people won't bother to delve into filtering options to show unpatrolled edits and we need to expose some edits as needing patrol otherwise they risk not being dealt with in time. Specifically for ORES, that would be edits that have not been patrolled and that ORES considers damaging (according to the user threshold if set, or the default). We could show them by default even if the user didn't enable the ORES beta.

If we were to do this mostly concerned about enwiki, we could just temporarily add a line to Common.css hiding it, and later remove once users who definitely don't want it can add to their personal CSS (or a gadget or whatever). This would be quick and easy and not add additional technical debt to the current patrol flag code.

This would also hide those on new pages and moves (T98617) . There should be a middle ground.

Also, since enwiki decided to restrict new page patrolling to a special usergroup, we would need a new userright that allows to patrol edits only (not new pages or moves). This could be granted to pending changes reviewers.

As part of our RC Page Improvements, we are implementing:

  • A filter for patrolled/unpatrolled
  • Various ORES filters that will enable users to find the edits that are most in need of review.
  • User-defined highlighting as an alternative to or in addition to filtering, so users can decide which edit properties they want to spotlight.

Using these tools, which will be out this spring, users can easily find edits that are likely damaging and unpatrolled. (See this video for a demonstration of the tools generally.)

This sounds promising, though I'd still like a way to integrate other sources than ORES, such as AbuseFilter and bots.

I also prefer to do this with dynamic filtering, rather than the -1, 0, 1 solution. As more of the clearly problematic (needs patrol) edits are patrolled (which should be possible with more patrollers), people can move on to the not-as-obvious ones. So it makes sense to either dynamically determine what "needs patrol" (this could still be a filter, but a dynamically calculated one), or just have people use their own judgment with ORES, etc. filters (possibly with this on by default, e.g. to hide the most obviously-good)

Krinkle removed a subscriber: Krinkle.Sep 6 2017, 8:29 PM
Cirdan added a subscriber: Cirdan.Jul 3 2018, 5:39 AM