This task is about analyzing how people are engaging with the new Replying tool to:
**Help answer these questions:**
1. Are people are having success using the feature?
2. Are people using the tool in ways that negatively impact the experience of others?
**So we can decide:**
What – if any – changes should be made to the workflow before deploying the feature to more people and with it, beginning an A/B to determine its impact?
=== Workflow engagement metrics
//This section is still being drafted. Once complete, it will continue the information needed make the two decisions listed above.//
- For each metric, we'd like to see a breakdown by wiki where the Reply Tool is currently deployed. //You can see where and how the Reply Tool is deployed here: https://www.mediawiki.org/wiki/Talk_pages_project#Active_initiatives .//
- "Experience level": we'd like to see experience level expressed using the following buckets:
-- 0 cumulative edits
-- 1-4 cumulative edits
-- 5-99 cumulative edits
-- 100-999 cumulative edits
-- 1000+ cumulative edits
**1. Are people publishing the replies they start?**
- Reply Tool funnel (grouped by experience level and the input mode – `source` / `visual` – they are shown by default):
-- Of the people who click a `[ reply ]` link (`action = 'init'`), what % of people see the interface ready (`action = 'ready'`)?
-- Of the people who see the interface is ready `action = 'ready`, what % of people start typing a comment (`action = 'firstChange'`)?
-- Of the people who start typing a comment (`action = 'firstChange'`), what % of people click the Reply button (`action = 'saveIntent'`)?
-- Of the people who click the `Reply` button (`action = 'saveIntent'`), what % of people successfully publish the comment they were drafting (`action = 'saveSuccess'`)?
- Reply Tool comment completion (grouped by experience level and the input mode – `source` / `visual` – they are shown by default):
-- Of the people who click the Reply link (`action = 'init'`), what % of people successfully publish the comment they were drafting (`action = 'saveSuccess'`)?
- Full-page wikitext talk page edit completion rate (grouped by experience level):
-- Of the people who click an `Edit` link on a talk page (`action = 'init'`), what % of people successfully save an edit (`action = 'saveSuccess'`)?
**2. Are people satisfied with the feature?**
- % of people who publish at least one comment and turn off the Reply Tool preference (name: `Enable quick replying`) in [Special:Preferences](https://cs.wikipedia.org/wiki/Speci%C3%A1ln%C3%AD:Nastaven%C3%AD#mw-prefsection-editing).
- Distribution of people who posted a comment with the tool on only **one day**?
- Distribution of people who posted a comment with the tool on ___ (see grouping below) distinct days:
-- 1-5 days
-- 6-10 days
-- 11-15 days
-- 16-20 days
-- 21-25 days
-- 26-30 days
- Distribution of people who have posted the following number of comments with the Reply Tool:
-- 1-5 comments
-- 6-10 comments
-- 11-15 comments
-- 16-20 comments
-- 21-25 comments
-- 26-30 comments
-- 31-35 comments
-- 50+ comments
**3. Are people posting disruptive comments?**
- % of comments posted using the Reply Tool that are reverted within 48 hours
- # of people blocked after posting a reply using DiscussionTools
=== Metric refinements
//The below are ordered by priority. Where "1." is the highest priority and "3." is the lowest priority.//
- [ ] **1.**Where applicable, can you please add charts that aggregate Junior Contributors and Senior Contributors into two separate "bins"? //Where "Junior Contributors" are people who have made <100 cumulative edits edits across namespaces ? Where "Senior Contributors" are people who have made >500 cumulative edits across namespaces?//
- [ ] **2.** Can we see companion charts to the already-existing ["Reply Tool Wikitext vs Full Talk Page Wikitext Completion Rates"](https://nbviewer.jupyter.org/github/wikimedia-research/Discussion-tools-analysis-2020/blob/master/Engagement-Metrics/Reply-Tool-Workflow-Engagement-Metrics.ipynb#Reply-Tool-Wikitext-vs-Full-Talk-Page-Wikitext-Completion-Rates) chart that shows "Reply Tool visual vs Full Talk Page Wikitext Completion Rates" and "Reply Tool (visual //and// source) vs. Full Talk page Wikitext"?
- [ ] **3.** Filter out `section=new` edits from the "Reply Tool Wikitext/Visual/Both vs Full Talk Page Wikitext Completion Rates" charts.
=== Open questions
- [x] How long after v2.0 is released should this analysis happen?
-- This analysis is scheduled to begin the week of 2-November per T263050#6565226.
- [x] Is new information required to know what text input mode (`source` or `visual`) a user is when they take a certain action (e.g. switching text input modes, abandoning/publishing a comment)?
-- No. For any event in EditAttemptStep we will be able to know what editing input mode (source or visual) the user was in when they took the action.
-- Caveat: the switching event (between source and visual) has not yet been implemented. //See: T247139#6114499//.
- [ ] Analysis is completed that contains the `===Workflow engagement metrics` and `===Metric refinements` listed above
i. The pinging feature became available on 17-June-2020: https://www.mediawiki.org/wiki/Talk_pages_project/replying#17_June_2020
//This task/analysis, had previously been scoped as a follow up to a then-planned analysis of engagement with Version 1.0: T247138.//