[SPIKE] How many people have turned off the New Discussion Tool since it's been available as a Beta Feature
Closed, ResolvedPublic
Actions

Description

Prior to starting the A/B test of the New Discussion Tool, we would like to ensure people are not experiencing any major issues with the tool.

To help us evaluate whether people are experiencing any major issues with the tool, we would like to know whether a significant number of people disabled the tool after using it to publish an edit.

Requirements

An analysis is done that enables us to see the percentage of contributors that explicitly disabled the New Discussion Tool after making at least one non-reverted edit with it, grouped by experience level and wiki.

Related Objects
Search...

View Standalone Graph

This task is connected to more than 200 other tasks. Only direct parents and subtasks are shown here. Use View Standalone Graph to show more of the graph.

Status	Assigned	Task
		· · ·
Resolved	DLynch	T291308 Make config change to start New Discussion Tool A/B Test
Resolved	• ppelberg	T294491 [SPIKE] How many people have turned off the New Discussion Tool since it's been available as a Beta Feature
		· · ·

Event Timeline

• ppelberg created this task.Oct 27 2021, 8:49 PM

• ppelberg moved this task from Backlog to Triaged on the DiscussionTools board.Nov 4 2021, 3:25 PM

• ppelberg updated the task description. (Show Details)Nov 4 2021, 11:09 PM

• ppelberg added a project: Goal.Nov 25 2021, 12:42 AM

• ppelberg added a parent task: T291308: Make config change to start New Discussion Tool A/B Test.Nov 25 2021, 12:53 AM

• ppelberg removed a parent task: T277825: Run A/B test to evaluate impact of New Topic Tool.

MNeisler claimed this task.Dec 2 2021, 4:43 PM

MNeisler triaged this task as Medium priority.

MNeisler added a project: Product-Analytics (Kanban).

MNeisler moved this task from Next 2 weeks to Doing on the Product-Analytics (Kanban) board.

@ppelberg

Below are the percentages of contributors that explicitly disabled the New Discussion Tool after making at least one non-reverted edit with it since the tool was deployed in Feb 2021. I've currently provided overall percentages across all wikis as well as breakdowns by experience level. Please review summary and open question below.

Note: There appear to be two possible ways to disable the new discussion tool: (1) By explicitly disabling the Enable quick topic adding option in your editing preferences OR (2) By explicitly disabling the Discussion Tools beta feature in your beta preferences. I provided the opt-out rate for both these methods.

Percentage of users that made at an edit with the new discussion tool and explicitly disabled the Enable quick topic adding option in their editing preferences

Overall: 0.6%
By Experience Level:
- under 100 edits: 0.4%
- 100-500: 0.2%
- over 500: 0.8%

Percentage of users that made at an edit with the new discussion tool and explicitly disabled the Discussion Tools beta option in their editing preferences

Overall: 7.5%
By Experience Level:
- under 100 edits: 5.3%
- 100-500: 4.7%
- over 500: 9.2%

Data Notes and Caveats

Reviewed Time Period: 18 Feb 2021 (date of initial deployment) through 31 November 2021.
Data Sources: I reviewed non-reverted new discussion tool edit events logged in mediawiki_history and non-default user preferences logged in mediawiki user_properties table to determine the current number of new discussion tool editors that have explicitly disabled the tool.
Data reflects the user's current state and does not provide detail on if the user changed their preference multiple times.
There are users that have made an edit with the New Discussion tool but don't have a new discussion tool beta preference set in the user properties table. Possible reasons for this include: (1) the user disabled the setting by selecting 'restore all default preferences' in their user preference or (2) the user enabled discussion tools in their global preferences but not in their local preferences.

OPEN QUESTION:
@ppelberg