Page MenuHomePhabricator

Broadcast Reply Tool A/B test results and opt-out thinking
Closed, ResolvedPublic


This ticket involves the work with ensuring volunteers, at all projects, are aware of: 1) the results of the Reply Tool A/B test (T252057) and 2) how the Editing Team is approaching offering the tool as an opt-out preference at all Wikipedias and Wikimedia projects.


CommunicationLink to announcementPlanned publication dateTicketActual publication date
Phase 1: Wikipedias where Reply Tool is already avail. by defaultearly May, 2021T282298
Phase 2: Wikipedias participating in the Reply Tool A/B testSee additional links in T281533early May, 2021T281533
Phase 3: Small-to-medium Wikipedias, plus all single-language non-Wikipedias (e.g., the Wikiquotes and Wikivoyages)June 2021 newsletterT283866
Phase 4: : Multi-lingual projects: Wikidata, Commons, Meta-Wiki,, etc.June 2021 newsletter
Phase 5: Remaining large Wikipedias (English, German, Spanish, and Japanese; maybe Russian and Portuguese)June 2021 newsletter
Editing Team NewsletterJune 2021 newsletterJune 2021T280384June 2021
Tech/News (Reply Tool project page)Talk_pages_project/Replyinglate-April/early-May, 2021 T27035223 April 2021

Announcement components

Broadly speaking, people should finish reading each of the communications listed in the ===Announcements section above knowing the answers to the following questions:

  1. "What is the Reply Tool A/B test leading the Editing Team to conclude?"
    • Juinor Contributors are more likely to successfully publish comments using the Reply Tool than they are using page editing.
      • Evidence: Of all the Junior Contributors who attempted to publish a comment to a talk page during the test, 72.9% of the Junior Contributors who had access to the Reply Tool were able to publish at least 1 comment, compared to 27.6% of all Junior Contributors who did not have access to the Reply Tool. This represents a 164% observed increase or 2.6x increase in comment completion rate. See slide 5.
    • The comments Junior Contributors post with the Reply Tool are less likely to be reverted than the comments they post with page editing.
      • Evidence: 10.76% of the comments Junior Contributors posted with page editing were reverted within 48 hours of being made compared to 2.21% of the comments Junior Contributors posted with the Reply Tool. See slide 13.
  1. "What decision(s) is the Editing Team using these conclusions to make?"
    • To proceed with the plan to offer the Reply Tool as an opt-feature at all projects. Note: a part of this "plan" involves consulting with volunteers to learn whether they see issues they think need to be resolved before the tool is made more widely available at their project.
  2. "Where can I review the results for myself?"
  3. "What is the best way to share the questions I have about: A) the conclusions the Editing Team is drawing from these results and/or B) the decision(s) they are using these results to make?"
  4. "What is the best way to share the issues I think need to be resolved before the Reply Tool is offered as an opt-out preference at this project?"


  • The list of "Communications" channels listed in the ===Announcement section is finalized (read: we have defined all of the announcements we are going to make, where we are going to make them and when we are going to make them).
  • Links to said announcements are added to the table in the ===Announcements section above

Event Timeline


Planned publication date
late-April/early-May, 2020
late-April/early-May, 2020

You mean late April/early May, 2021, don’t you? You would be a bit late for 2020…

You mean late April/early May, 2021, don’t you? You would be a bit late for 2020…

Yikes. A typo or a moment where I was lost in space and time? We may never know. In either case, thank you for catching this, @Tacsipacsi :)

ppelberg added a subscriber: MNeisler.

Note: I've updated the task description in light of the findings @MNeisler shared with the team today. See: Reply Tool AB Test Results.

ppelberg updated the task description. (Show Details)

The Reply Tool A/B test results are now available on the Reply Tool project page:

Task description update

  • ADDED the five phases, and the projects within each of those phases, we defined in T280388 to the task description.