I've updated the task description with suggested adoption metrics.

I tried to draft metrics I believed would be (1) useful in answering the research questions identified in the task description and (2) provides a lightweight high-level assessment of tool adoption we can use to determine further analysis/breakdowns might be needed.

@ppelberg - Assigning to you for review. Let me know if you have suggested additions or changes. Once definitions are finalized, I can then plan to work on calculating the defined metrics.

In T263053#7158010, @MNeisler wrote:

I've updated the task description with suggested adoption metrics.

Excellent.

I tried to draft metrics I believed would be (1) useful in answering the research questions identified in the task description and (2) provides a lightweight high-level assessment of tool adoption we can use to determine further analysis/breakdowns might be needed.

Noted.

@ppelberg - Assigning to you for review. Let me know if you have suggested additions or changes. Once definitions are finalized, I can then plan to work on calculating the defined metrics.

A couple of questions for you below, @MNeisler.

Questions

The percent of all edits made with the New Discussion Tool that are reverted within 48 hours of being published. (Purpose: Do people NOT using the New Discussion Tool find it disruptive?)

• Are we able to group these edits by experience level?
• Are we able to calculate the same metric – percent of all edits that are reverted within 8 hours of being published – for the existing &action=edit&section=new workflow? I ask this thinking it would be helpful to have a benchmark to compare the New Discussion Tool revert rate to.

The percent of distinct contributors by experience level who click an affordance to start at least 1 new discussion (Purpose: Who is using it?).

• Can you share the thinking that's leading you to suggest we measure/define usage as the people who initiate the workflow? I ask this thinking "usage" more cleanly maps to people who publish, or attempt, to publish a new topic with the New Discussion Tool.

For contributors that have posted 1 new topic with the New Discussion Tool, what percent of distinct contributors used the New Discussion Tool to create the following number of topics? (Purpose: How much are they using it?)

• What's leading you to suggest we calculate the absolute number of topics people have started vs. the percentage of topics people used the New Discussion Tool to start? I ask this wondering whether knowing how many times people "chose" to use the New Discussion Tool in relation to the number of opportunities they had to use it would help us to better understand how much people are using the tool and whether it is behaving in the ways they expect it to.

@ppelberg - See responses inline below and let me know if you have any follow-up questions or suggested changes.

Are we able to group these edits by experience level?

Yes, this data will come from mediawiki_history which includes an event_user_revision_count field we can use to group this data by the user's experience level.

Are we able to calculate the same metric – percent of all edits that are reverted within 48 hours of being published – for the existing &action=edit&section=new workflow? I ask this thinking it would be helpful to have a benchmark to compare the New Discussion Tool revert rate to.

Unfortunately, no. Revert data comes from mediawiki_history (not EditAttemptStep) and we would need a revision tag to specifically identify new topic edits created using the existing &action=edit&section=new workflow, which I don't believe exists.

Instead, we could compare to the revert rate of edits on talk pages made using page editing methods (VE or wikieditor). It wouldn't be specific to new topics but could help give us a rough baseline.

Can you share the thinking that's leading you to suggest we measure/define usage as the people who initiate the workflow? I ask this thinking "usage" more cleanly maps to people who publish, or attempt, to publish a new topic with the New Discussion Tool.

When originally drafting this metric, I was interpreting "usage" to mean "tried the tool" and concerned that looking at only published edits would exclude people that tried to use the tool and were not able to successfully complete an edit for some reason.

But I also see that looking just at init events may be noisy as it includes people just clicking on the affordance to test it or unintentional clicks. I'm comfortable with switching this to published edits or perhaps looking at people that start typing (event.action = 'firstChange') would make sense as an indicator of who is trying the tool?

What's leading you to suggest we calculate the absolute number of topics people have started vs. the percentage of topics people used the New Discussion Tool to start? I ask this wondering whether knowing how many times people "chose" to use the New Discussion Tool in relation to the number of opportunities they had to use it would help us to better understand how much people are using the tool and whether it is behaving in the ways they expect it to.

I thought through this metric a little more and agree that we should replace the absolute number of topics with the percentage of topics. See reasoning below:

I considered looking at the percentage of topics people used the New Discussion Tool to start for the same reasons you described but ended up initially removing it because of the following:

This metric is a little noisy and could be difficult to interpret as there could be cases where the following people end up looking the same in the data. Person A: added two new topics to talk pages in the reviewed timeframe, one of which was with the new discussion tool; Person B: made a total of 150 new topics to talk pages, 75 of which were with the New Discussion tool.

However, thinking it through some more, I think the absolute number of topics metric is more misleading for this particular feature since people will likely only use it if they have an opportunity/reason to use it. In this metric the following people end up looking the same in the data: Person A used the new discussion tool to post 1 topic and did not use it again as they did not need to start another new topic during the reviewed time period. Person B used the new discussion tool to post 1 topic, did not like it and ended up using standard workflows for the other new topics they started during the time period.

The "percentage of topics people used the New Discussion Tool to start" metric gives us context on the number of opportunities, which I think is more valuable in this case (despite the slight noise).

In T263053#7188960, @MNeisler wrote:

@ppelberg - See responses inline below and let me know if you have any follow-up questions or suggested changes.

Excellent. Responses below.

Are we able to group these edits by experience level?

Yes, this data will come from mediawiki_history which includes an event_user_revision_count field we can use to group this data by the user's experience level.

Great. I've updated the task description to reflect the fact that we'll group reverted edits by experience level.

Are we able to calculate the same metric – percent of all edits that are reverted within 48 hours of being published – for the existing &action=edit&section=new workflow? I ask this thinking it would be helpful to have a benchmark to compare the New Discussion Tool revert rate to.

Unfortunately, no. Revert data comes from mediawiki_history (not EditAttemptStep) and we would need a revision tag to specifically identify new topic edits created using the existing &action=edit&section=new workflow, which I don't believe exists.

Ah, I see.

Instead, we could compare to the revert rate of edits on talk pages made using page editing methods (VE or wikieditor). It wouldn't be specific to new topics but could help give us a rough baseline.

Let's do this. I've updated the task description to reflect the fact that we'd like to compare the revert rate of edits made with the New Discussion Tool to the revert rates of talk page edits made with full-page editing.

Can you share the thinking that's leading you to suggest we measure/define usage as the people who initiate the workflow? I ask this thinking "usage" more cleanly maps to people who publish, or attempt, to publish a new topic with the New Discussion Tool.

When originally drafting this metric, I was interpreting "usage" to mean "tried the tool" and concerned that looking at only published edits would exclude people that tried to use the tool and were not able to successfully complete an edit for some reason.

But I also see that looking just at init events may be noisy as it includes people just clicking on the affordance to test it or unintentional clicks. I'm comfortable with switching this to published edits or perhaps looking at people that start typing (event.action = 'firstChange') would make sense as an indicator of who is trying the tool?

At this stage, let's define people who have tried the tool as those who published at least one new topic with it. I've updated the task description to reflect this.

Thinking: this will help us get a sense of scale of how many people have engage with the workflow/tool in its entirety.

What's leading you to suggest we calculate the absolute number of topics people have started vs. the percentage of topics people used the New Discussion Tool to start? I ask this wondering whether knowing how many times people "chose" to use the New Discussion Tool in relation to the number of opportunities they had to use it would help us to better understand how much people are using the tool and whether it is behaving in the ways they expect it to.

I thought through this metric a little more and agree that we should replace the absolute number of topics with the percentage of topics. See reasoning below:

Sounds good to me. I've updated the task description to reflect this.

I considered looking at the percentage of topics people used the New Discussion Tool to start for the same reasons you described but ended up initially removing it because of the following:

This metric is a little noisy and could be difficult to interpret as there could be cases where the following people end up looking the same in the data. Person A: added two new topics to talk pages in the reviewed timeframe, one of which was with the new discussion tool; Person B: made a total of 150 new topics to talk pages, 75 of which were with the New Discussion tool.

However, thinking it through some more, I think the absolute number of topics metric is more misleading for this particular feature since people will likely only use it if they have an opportunity/reason to use it. In this metric the following people end up looking the same in the data: Person A used the new discussion tool to post 1 topic and did not use it again as they did not need to start another new topic during the reviewed time period. Person B used the new discussion tool to post 1 topic, did not like it and ended up using standard workflows for the other new topics they started during the time period.

The "percentage of topics people used the New Discussion Tool to start" metric gives us context on the number of opportunities, which I think is more valuable in this case (despite the slight noise).

What you described above makes sense to me; seeing the assumptions that informed a particular approach is helpful...I value you investing the time to share it here, our future selves will thank you!

Notes from the conversation @MNeisler and I had on 28 July
It is not practical to provide per wiki breakouts without a creating dashboard considering the New Discussion Tool is available as an opt-in beta feature at all Wikipedias.

As such, to start, we will look at the specific wikis where we are considering opt-out deployments. To start, these wikis will ar.wiki and hu.wiki per T280390.

@ppelberg Here is the first draft of the new discussion tool adoption metrics report for review.

Assigning this over to you for review. Please let me know if you have any questions or specific metrics you'd like to explore further.

Summary of key metric results below (please see report for details on how these numbers were calculated and additional breakdowns):
(Data based on events logged since the deployment of the beta feature on 18 February 2018 through 31 July 2021).

Disruption

• What percent of contributors explicitly disabled the New Discussion Tool after making at least one edit with it?
  • Overall across all Wikimedia projects, 9.1% of contributors that saved at least one new discussion tool edit explicitly opted out of the new discussion tool (and did not opt-in again during the reviewed time period).
  • Junior contributors (users with under 100 edits) had the highest opt-out rate (15.04%).
• What percent of all edits made with the New Discussion Tool are reverted within 48 hours of being published?
  • Overall, the revert rate for the new discussion tool is only slightly higher than the revert rate for full page editing on talk pages (2.75% for the new discussion tool compared to 2.21% for full-page editing).
  • However, by experience level, the revert rate for the new discussion tool is lower than full-page editing for Junior Contributors. For editors with under 100 cumulative edits, the revert rate was 6.22% for edits made with the new discussion tool compared to 7.9% for page editing (a -27% percent decrease).
  • The new discussion tool also had a lower revert rate on both Arabic and Czech Wikipedia compared to page editing on those Wikipedias.

Usage

• What percent of distinct contributors publish at least one new topic with the tool?
  • Since deployment as a beta feature on 18 February 2021, a total of 5,388 distinct users across all Wikimedia Projects have posted at least one new topic using the new discussion tool.
  • Overall, 2.5% of contributors that have edited a talk page and 22.4% of contributors that have posted a new topic during the reviewed timeframe, made at least 1 edit with the new discussion tool.
  • Senior contributors more commonly used the tool at least once to create a new topic compared to Junior Contributors. Almost half (46.5%) of contributors with over 100 edits that created a new topic on a talk page posted at least one of their new topics using the new discussion tool.
  • Similar to the proportion across all Wikis identified above, 19.4% of Arabic contributors and 20.3% of Czech contributors that posted a new topic used the new discussion tool at least once.

• For contributors that have posted more than one new topic with the New Discussion Tool, what percent of distinct contributors used the New Discussion Tool to create the following percentage of all new topics within the time period?[^4]
  • Most contributors (90.82%) that used the new discussion tool posted just one new topic with the tool during the reviewed timeframe. Of the contributors that posted more than one new topic on a talk page, 95.3% of these contributors posted between 75 to 100 percent of their new topics using the new discussion tool, indicating that these contributors typically chose to use the tool when presented with an opportunity to start a new topic.

@ppelberg - See responses in line below. Happy to discuss further in our meeting this afternnon.

• @MNeisler: Can you think of another tool's opt-out rate we can compare the New Discussion Tool's opt-out rate to?
  • I ask this thinking that 9.1% seems low (a good thing), tho it's hard to have confidence in that sense not knowing how it compares to the opt-out rate of a similar experience.
  • Note: in T249386#6309760 we learned, "...20.4% of all reply tool users explicitly turned off the feature after making at least 1 edit with the reply tool and did not turn it back on again."

I believe the reply tool opt-out rate would be the best reference point in this case. Off-hand, I can't think of any other data points that would be useful for comparison but can investigate further if needed (but agree that 9.1% seems fairly low)

• @MNeisler: Are you able to look at the opt-out rates of users with under 100 edits with finer granularity? E.g. breaking them out into groups of 0-10 edits, 10-20 edits, 30-40 edits, etc.
  • Thinking: it is counterintuitive, although not yet concerning, to me that Junior Contributors would opt-out of the New Discussion Tool at higher rates than Senior Contributors.

Yes, I was a little surprised by that too. It's still low but not expected. I'll break down the opt-out rates for Junior Contributors into smaller edit group counts as suggested to see if that helps provide some insight. I'd also suggest we look at the wikis with the highest opt-out rate among Junior Contributors in case there is a specific wiki or two that has a significantly higher opt-out rate compared to the other projects.

• Senior contributors more commonly used the tool at least once to create a new topic compared to Junior Contributors. Almost half (46.5%) of contributors with over 100 edits that created a new topic on a talk page posted at least one of their new topics using the new discussion tool.

• @MNeisler: would it be accurate for me to understand the corresponding metric for Junior Contributors as follows? Of all the Junior Contributors who published a new topic between 18 February 2021 and 31 July 2021, including those who did NOT have access to the New Discussion Tool, 12.34% used the the New Discussion Tool at least once to do so.

For the metric identified above and all usage metrics, I limited the reviewed time period to the dates when the tool was deployed as a beta feature to all Wiki projects (17 March 2021 through 31 July 2021) when reviewing overall and by experience level metrics [1]. This allows us to more equitably compare events across wikis as they had the same access during that time. During this time, all users on wikis had access to the tool as a beta feature; however, not all users may have enabled the tool. I took a shot at revising your statement above to clarify (changes marked in bold):

"Of all the Junior Contributors who published a new topic between 17 March 2021 and 31 July 2021, including those who may NOT have enabled the New Discussion Tool, 12.34% used the New Discussion Tool at least once to do so.**"

[1] For Arabic and Czech Wikipedia specific usage metrics, I reviewed the time period when the tool was deployed as a beta feature to these wikis (18 February 2021- 31 July 2021)

In T263053#7275696, @MNeisler wrote:

@ppelberg - See responses in line below. Happy to discuss further in our meeting this afternnon.

Excellent. Below are the outcomes of what we talked about during today's meeting.

• @MNeisler: Can you think of another tool's opt-out rate we can compare the New Discussion Tool's opt-out rate to?
  • I ask this thinking that 9.1% seems low (a good thing), tho it's hard to have confidence in that sense not knowing how it compares to the opt-out rate of a similar experience.
  • Note: in T249386#6309760 we learned, "...20.4% of all reply tool users explicitly turned off the feature after making at least 1 edit with the reply tool and did not turn it back on again."

I believe the reply tool opt-out rate would be the best reference point in this case. Off-hand, I can't think of any other data points that would be useful for comparison but can investigate further if needed (but agree that 9.1% seems fairly low)

DECIDED: no further analysis needed. Reason: we think a 9.1% opt-out rate is acceptable and expected.

• @MNeisler: Are you able to look at the opt-out rates of users with under 100 edits with finer granularity? E.g. breaking them out into groups of 0-10 edits, 10-20 edits, 30-40 edits, etc.
  • Thinking: it is counterintuitive, although not yet concerning, to me that Junior Contributors would opt-out of the New Discussion Tool at higher rates than Senior Contributors.

Yes, I was a little surprised by that too. It's still low but not expected. I'll break down the opt-out rates for Junior Contributors into smaller edit group counts as suggested to see if that helps provide some insight. I'd also suggest we look at the wikis with the highest opt-out rate among Junior Contributors in case there is a specific wiki or two that has a significantly higher opt-out rate compared to the other projects.

ACTION: we are going to iterate on the analysis and do what @MNeisler is describing above.

• Senior contributors more commonly used the tool at least once to create a new topic compared to Junior Contributors. Almost half (46.5%) of contributors with over 100 edits that created a new topic on a talk page posted at least one of their new topics using the new discussion tool.

• @MNeisler: would it be accurate for me to understand the corresponding metric for Junior Contributors as follows? Of all the Junior Contributors who published a new topic between 18 February 2021 and 31 July 2021, including those who did NOT have access to the New Discussion Tool, 12.34% used the the New Discussion Tool at least once to do so.

For the metric identified above and all usage metrics, I limited the reviewed time period to the dates when the tool was deployed as a beta feature to all Wiki projects (17 March 2021 through 31 July 2021) when reviewing overall and by experience level metrics [1]. This allows us to more equitably compare events across wikis as they had the same access during that time. During this time, all users on wikis had access to the tool as a beta feature; however, not all users may have enabled the tool. I took a shot at revising your statement above to clarify (changes marked in bold):

"Of all the Junior Contributors who published a new topic between 17 March 2021 and 31 July 2021, including those who may NOT have enabled the New Discussion Tool, 12.34% used the New Discussion Tool at least once to do so.**"

DECIDED: no further action needed. Reason: it is expected to us that Junior Contributors would use the New Discussion Tool as infrequently as they have at this point, considering the feature is only available to people explicitly opted-in to using it (this is NOT something we expect Junior Contributors to do).

In T263053#7289161, @MNeisler wrote:

Note: There are contributors that have used the new discussion tool but don't have a discussion tool preference set in the User Properties Table.

Couldn’t it be (at least partly) people like me who turned on DiscussionTools in their global preferences? I used DT on quite a number of wikis, yet I have turned it on locally on very few if any wikis (maybe on huwiki, before I found the global preferences). If it’s true, these people have not only not opted out of DT, but are probably the most passionate about it (since they decided to have it on all wikis instead of only one or a few), so the statistics are even better.

Couldn’t it be (at least partly) people like me who turned on DiscussionTools in their global preferences? I

Good point @Tacsipacsi. I've updated the analysis (both the report and summary in T263053#7289161) to clarify the currently identified reasons for why a new discussion tool contributor might not have the discussion tool local preference recorded in the User Properties table.

If needed a query could be run against the centralauth db to determine how many of the new discussion tool contributors have DiscussionTool enabled in global preferences but not locally; as you pointed out, this will likely result in a slight decrease in the overall opt-out rate percentages presented above.

I'll discuss next steps with @ppelberg