Page MenuHomePhabricator

New Discussion Tool: calculate adoption metrics
Open, MediumPublic

Description

Prior to making the New Discussion Tool available to all volunteers at our partner wikis, we would like to know how the tool is being used/adopted as a Beta Feature.

Specifically, we are curious to know the answer to the following questions:

  1. Are people finding the tool behaves in the ways they expect?
  2. Are people finding the tool to be disruptive?

To help answer the questions above, we would like to understand the "Adoption metrics" listed below.

Note: this "adoption metrics analysis" varies from the adoption metrics analysis we ran for the Reply Tool (T249386). This analysis varies in so far as we will not be considering the "usefulness of the tool" [i][ii] as part of this initial opt-out deployment decision. Reason being: the New Discussion Tool is targeted to Junior Contributors [iii] who: A) are not likely to use Beta Features and accordingly B) not likely to be adequately represented in this data. For now

Adoption metrics

For each of our four partner wikis (see T265953) we would like to understand the metrics described below, ranked from highest priority ("1.") to lowest priority ("3.").

Scale
1. From TBD DATE onward, how widely is the New Discussion Tool being used?

  • How many new topics (read: edits) have people used the New Discussion Tool to start, grouped by wiki?
  • How many distinct pages have people used the New Discussion Tool to start topics on, grouped by wiki?
  • How many distinct people used the New Discussion Tool to start at least one topic (read: publish one edit), grouped by wiki and experience level?

Disruption
2. From DATE onward, how many people used, and then disabled, the New Discussion Tool?

  • How many distinct people turned off the New Discussion Tool after making at lease one edit with it, grouped along the following dimensions?
    • Wiki
    • People who explicitly turned the feature on [iv] and people who had the feature turned on for them. [v]
    • Experience level (read: cumulative edit count )

Usage
3. From DATE onward, how much have people used the New Discussion Tool?

  • How many distinct people used the New Discussion Tool to create the following number of topics?
    • 1 topic
    • 1-5 topics
    • 6-10 topics
    • 11-15 topics
    • 16-20 topics
    • 25+ topics

4. From DATE onward, how many people used the New Discussion Tool to add the following proportions of new topics ?

  • 1-10 percent
  • 11-20 percent
  • 21-30 percent
  • 31-40 percent
  • 41-50 percent
  • 51-60 percent
  • 61-70 percent
  • 71-80 perent
  • 90-100 percent

Note: only people who have published at least one edit with the New Discussion Tool should be counted and the "denominator" should be section=new edits made on desktop.

Open question

  • In what ways does this task relate to T268094? Are there opportunities to transfer the work done as part of this task to T268094?

Done

  • The "Adoption metrics" above have been calculated

i. https://www.mediawiki.org/wiki/Talk_pages_project/replying#Metrics
ii. As measured by things like, "the percentage of new topics people are using the New Discussion Tool to create."
iii. https://www.mediawiki.org/wiki/Talk_pages_project/Glossary
iv. "Explicitly": meaning they did not have the following preference checked: Automatically enable all new beta features
v. "Turned on for them": meaning they did have the following preference checked: Automatically enable all new beta features