This task is about publishing the rationale for the A/B test on the Reply Tool project page on mw.org. [i]
Update components
- What questions we are trying to use this A/B test to answer
- What metrics we will use to compare the two A/B test's two test groups
- What decision(s) the results of this A/B test will help us make
Update language
In the broadest of terms, this A/B test is a "temperature check"...a way for us to check whether the Reply Tool causes people to be less successful publishing edits to talk pages.
Decision to be made
This A/B test is intended to help us decide whether the Reply Tool should be made available to all people, at all wikis, as an opt-out user preference. See T262331.
Decision inputs
To make this decision, we are interested in learning whether the Reply Tool causes people to be meaningfully less likely to publish comments they start than they are with full-page editing. We are also curious to know whether the Reply Tool causes a significant increase in the number of disruptive edits being made to talk pages.
The metrics we will use to evaluate evaluate the above can be found in T252057.
It is important to note that we do not consider notable positive changes in the rates at which people start and continue participating on talk pages as well as the rates at which people are participating on talk pages in disruptive ways as prerequisites for the Reply Tool being made available to all people, at all wikis, as an opt-out user preference. We've come to think this for the following reasons:
- Evidence to date suggests [ii] the Reply Tool is having the intended short-term impact: Junior and Senior Contributors have an easier time participating in existing conversations, "correctly."
- We assume notable changes in the rates at which people start and continue participating on talk pages will likely get expressed over a longer period of time (read: they will not be detectable during the course of this A/B test).
Done
- An update is published to the Reply Tool project page [i] that contains the ===Update components listed above
i. https://www.mediawiki.org/wiki/Talk_pages_project/replying
ii. https://www.mediawiki.org/wiki/Talk_pages_project/replying#Metrics