This task is about analyzing how people are engaging with the new Replying tool to:
Help answer these questions:
- Are people are having success using the feature?
- Are people using the tool in ways that negatively impact the experience of others?
So we can decide:
What – if any – changes should be made to the workflow before deploying the feature to more people and with it, beginning an A/B to determine its impact?
Workflow engagement metrics
This section is still being drafted. Once complete, it will continue the information needed make the two decisions listed above.
- For each metric, we'd like to see a breakdown by wiki where the Reply Tool is currently deployed. You can see where and how the Reply Tool is deployed here: https://www.mediawiki.org/wiki/Talk_pages_project#Active_initiatives .
- "Experience level": we'd like to see experience level expressed using the following buckets:
- 0 cumulative edits
- 1-4 cumulative edits
- 5-99 cumulative edits
- 100-999 cumulative edits
- 1000+ cumulative edits
1. Are people publishing the replies they start?
- Reply Tool funnel (grouped by experience level and the input mode – source / visual – they are shown by default):
- Of the people who click a [ reply ] link (action = 'init'), what % of people see the interface ready (action = 'ready')?
- Of the people who see the interface is ready action = 'ready, what % of people start typing a comment (action = 'firstChange')?
- Of the people who start typing a comment (action = 'firstChange'), what % of people click the Reply button (action = 'saveIntent')?
- Of the people who click the Reply button (action = 'saveIntent'), what % of people successfully publish the comment they were drafting (action = 'saveSuccess')?
- Reply Tool comment completion (grouped by experience level and the input mode – source / visual – they are shown by default):
- Of the people who click the Reply link (action = 'init'), what % of people successfully publish the comment they were drafting (action = 'saveSuccess')?
- Full-page wikitext talk page edit completion rate (grouped by experience level):
- Of the people who click an Edit link on a talk page (action = 'init'), what % of people successfully save an edit (action = 'saveSuccess')?
2. Are people satisfied with the feature?
- % of people who publish at least one comment and turn off the Reply Tool preference (name: Enable quick replying) in Special:Preferences.
- Distribution of people who posted a comment with the tool on only one day?
- Distribution of people who posted a comment with the tool on ___ (see grouping below) distinct days:
- 1-5 days
- 6-10 days
- 11-15 days
- 16-20 days
- 21-25 days
- 26-30 days
- Distribution of people who have posted the following number of comments with the Reply Tool:
- 1-5 comments
- 6-10 comments
- 11-15 comments
- 16-20 comments
- 21-25 comments
- 26-30 comments
- 31-35 comments
- 50+ comments
3. Are people posting disruptive comments?
- % of comments posted using the Reply Tool that are reverted within 48 hours
- # of people blocked after posting a reply using DiscussionTools
The below are ordered by priority. Where "1." is the highest priority and "3." is the lowest priority.
- 1.Where applicable, can you please add charts that aggregate Junior Contributors and Senior Contributors into two separate "bins"? Where "Junior Contributors" are people who have made <100 cumulative edits edits across namespaces ? Where "Senior Contributors" are people who have made >500 cumulative edits across namespaces?
- 2. Can we see companion charts to the already-existing "Reply Tool Wikitext vs Full Talk Page Wikitext Completion Rates" chart that shows "Reply Tool visual vs Full Talk Page Wikitext Completion Rates" and "Reply Tool (visual and source) vs. Full Talk page Wikitext"?
- 3. Filter out section=new edits from the "Reply Tool Wikitext/Visual/Both vs Full Talk Page Wikitext Completion Rates" charts.
- How long after v2.0 is released should this analysis happen?
- This analysis is scheduled to begin the week of 2-November per T263050#6565226.
- Is new information required to know what text input mode (source or visual) a user is when they take a certain action (e.g. switching text input modes, abandoning/publishing a comment)?
- No. For any event in EditAttemptStep we will be able to know what editing input mode (source or visual) the user was in when they took the action.
- Caveat: the switching event (between source and visual) has not yet been implemented. See: T247139#6114499.
- Analysis is completed that contains the ===Workflow engagement metrics and ===Metric refinements listed above
i. The pinging feature became available on 17-June-2020: https://www.mediawiki.org/wiki/Talk_pages_project/replying#17_June_2020
This task/analysis, had previously been scoped as a follow up to a then-planned analysis of engagement with Version 1.0: T247138.