Page MenuHomePhabricator

What should our criteria be for edits to display in the Recent Activity module?
Closed, ResolvedPublic

Description

We want to show editors recent edits to their project that may require reverting. The most obvious way to do this would be to use edit Revert Risk scores and utilise the same filter as is accessible on Recent Changes - edits with a score over 0.85. We also want to avoid showing edits which have already been reverted, so that users are only seeing edits which need some action taken.

We want to know how many edits we will have available in our pool of edits we can display to users, and see if we can make further limitations to improve the edits we're showing, without limiting our pool too much.

  • id.wiki
  • tr.wiki
  • bn.wiki
  • tl.wiki
  • is.wiki

There are four filters we're considering applying, in descending order of importance:

  1. Revert Risk score > 0.85
  2. Not reverted
  3. Not patrolled (RCPatrol)/reviewed (FlaggedRevs)
  4. The edit is the most recent revision on the page

We'd like to know the cumulative (in order of above) number of unique Recent Changes edits meeting these criteria on each of the listed wikis.

Event Timeline

@Samwalton9-WMF
I've completed an analysis of the number of unique Recent Changes edits that meet each of the identified criteria. The results are currently summarized in this table for review.

Overview of table(s):
"Cumulative impact on edit availability" sheet: The table shows the cumulative impact of adding each additional filter to the criteria (in the order outlined in the task description). For example, row 2 shows the number of available recent changes edits at idwiki that have a revert risk score > 0.85 and row 3 shows the number of available edits that have a revert risk score > 0.85 and are not reverted. The Criteria Applied = "All four criteria" shows the number of edits available if all filters are applied to recent changes edits.

"Edit availability by each criteria" sheet: This table shows the number of recent changes edits that meet each individual criteria.

Some initial insights:

  • The impacts of each filter vary based on Wikipedia.
  • Overall, the addition of "the edit is the most recent revision on the page" filter causes the most significant decrease in available edits.
  • The "Not reverted" filter causes a decrease in edits ranging from 11% at idwiki to 50% at tlwiki when added to the "Revert risk score > 0.85" filter.

Methodology:
Data reflects all recent changes edits completed on a main namespace at the identified wikis. See code repo for more details on queries and methodology.

Perfect, thank you! It looks like this module, as designed, just isn't going to be that useful on the smaller wikis. When I divide the figures by 30 (imagining that a user might visit this page once per day), is.wiki has an average of 2.4 edits per day meeting just the Revert Risk threshold criterion, and tl.wiki has 6.5. That doesn't feel like enough edits to me that we would have much of anything to show multiple users on a daily basis.

On id.wiki having all four criteria active seems like it would be fine - 42 edits per day on average, but for tr.wiki and bn.wiki the numbers are worse - in the 1-3 region. Patrollers on those wikis could burn through those edits pretty quickly and be left with nothing to review.

I'm surprised there's such a difference between id.wiki and tr.wiki, given that they're wikis of the same approximate size and editor count. bn.wiki is about half the size, so not as surprising. I sanity checked idwiki and trwiki with RecentChanges filters and it seems accurate, just noting my surprise!

Given that this makes id.wiki about the only viable option to apply all four criteria to I think we need to decide whether to:

  • Lose the last criterion (most recent edit to the page),
  • Adjust the Revert Risk filter threshold down (which precludes us from linking to a pre-filtered Special:RecentChanges, and makes the edits displayed much more likely to be good), or
  • Only launch our pilot on id.wiki.

I don't think launching on only one wiki is a good idea, and I'd like to be able to link to a pre-filtered RC. The edit not being the most recent one to a page isn't a terrible thing, it just increases the likelihood of a user running into a case where they can't cleanly undo the edit. Based on this I suggest we move forward with the first three criteria, and limit our pool of potential wikis to deploy to to ones around the same size or bigger than bn.wiki, which leaves us with id.wiki, tr.wiki, simple.wiki, bn.wiki, az.wiki. This last point is a useful input for T402802: Who should be invited to view the moderator homepage experiment?.