Page MenuHomePhabricator

Run an experiment to evaluate impact of patrolling notifications for new users
Open, Needs TriagePublic

Description

It is well known that users are dissuaded from contributing by notices that their contributions have been reverted or are otherwise in contravention of a Wikimedia project's policies. On some wikis, however, individual edits can be marked as 'patrolled', indicating that they have been reviewed by an experienced user and deemed to be acceptable. We would like to experiment with sending notifications to new users notifying them that their edit has been patrolled, to understand what effect this has on their motivation to contribute.

Hypothesis: If we notify new editors when their edits are marked as patrolled, they will be more likely to continue contributing to their project.

If this hypothesis is supported, it would be another argument in favour of T409165: [SPIKE] What would it take to enable Recent Changes patrolling on all Wikipedias?.

Details
We could run an A/B test on a Wiki which already uses the RCPatrol feature to enable patrolling of individual edits. Some new users would receive a notification when their edits are marked as patrolled by a patroller, and others would continue to not receive any such notification.

The wikis currently using RCPatrol with more than 100 monthly new active editors (as of my outdated Dec 2020 Wiki comparison sheet) are: frwiki, itwiki, ptwiki, fawiki, nlwiki, viwiki, cswiki, hewiki. (idwiki removed because it uses FlaggedRevs)

This notification should probably only be sent when the user's edit is actively patrolled, not if it is autopatrolled.

For the purposes of this experiment we should only consider the MediaWiki patrolling feature, not Flagged Revisions, which also enabled per-edit patrolling.

There is an existing notification for page reviews:

PreferenceNotificationEmail
image.png (254×1 px, 36 KB)
Screenshot 2025-12-04 at 14.22.03.png (386×1 px, 53 KB)
Screenshot 2025-12-04 at 14.22.16.png (744×2 px, 117 KB)

Open questions

  • What should the notification text say?
  • Should the notification be an Alert or a Notice?
    • Thanks are a Notice; Undo is an Alert
  • Do we need to set any kind of rate limit? A user might have a large number of edits patrolled in a short period of time, whereas reverts are usually less frequent.
    • We could just sent a notification the first time a user's edit is patrolled.
  • [Data request probably needed] How quickly are new editors' edits generally patrolled? How many users could we expect to receive these notifications and how many would they receive?

Event Timeline

@Samwalton9-WMF do you need help from our team to set up the test, to consult on the experiment design, or something else specific?

@Samwalton9-WMF do you need help from our team to set up the test, to consult on the experiment design, or something else specific?

I think if we get to this (probably led by Growth since the intended outcome is improved new editor retention, and they have recently worked on other notification experiments), support with experiment design would definitely be appreciated, and then we'd see how we were getting on from there.

My understanding is that this fits into Growth's long-term roadmap under (T411817) but likely not into Growth's next focus (T409236).
But if we can find a time to squeeze this experiment in earlier, I would love to see that happen! I'll discuss with the Contributor Group PM.