Change Details

=== Purpose This task is about analyzing how people are engaging with the new Replying tool to: **Help answer these questions:** 1. Are people are having success using the feature? 2. Are people using the tool in ways that negatively impact the experience of others? **So we can decide:** What – if any – changes should be made to the workflow before deploying the feature to more people and with it, beginning an A/B to determine its impact? === Workflow engagement metrics //This section is still being drafted. Once complete, it will continue the information needed make the two decisions listed above.// **1. Are people publishing the replies they start?** - % of people who click reply AND who start typing a comment, grouped by which input mode – `source` / `visual` – they are shown by default - % of people who start typing a comment AND who publish their comment to page, grouped by which input mode – `source` / `visual` – they are shown by default **2. Are people abandoning the replies they start?** - % of people who start typing a comment AND who actively abandon their comment **3. Are people satisfied with the feature?** - % of people who publish at least one comment and turn off DiscussionTools in Beta Features - Of the people who have made at least 1 edit with the Replying workflow, the % of people who posted 1, 2-5, 5-10 and 10+ comments **4. Are people posting disruptive comments?** - % of comments posted using Replying workflow that are reverted - # of people blocked after posting a reply using DiscussionTools - % of comments posted using Replying workflow that are made in good faith **5. What text input mode (`source` or `visual`) are people using to publish comments?** - % of published comments drafted using the `Visual` mode by experience level - % of published comments drafted using the `Source` mode by experience level - % of published comments drafted that involved a contributor switching from the `Source` to the `Visual` and vice versa **6. Of the people who have made at least 1 edit with the Reply Tool //after// 17-June-2020 [i], after how many edits do people use the tool's Pinging Feature for the first time?* ** //This question is relevant to us deciding whether the existence of the tool's pinging feature ought to be communicated more explicitly via something like an onboarding experience per T253434#6334269.// - % of people who tried the tool's pinging feature in their 1st edit with the Reply Tool - % of people who tried the tool's pinging feature in their 2nd edit with the Reply Tool - % of people who tried the tool's pinging feature in their 3rd edit with the Reply Tool - % of people who tried the tool's pinging feature in their 4th edit with the Reply Tool - % of people who tried the tool's pinging feature in their 5th edit with the Reply Tool - % of people who tried the tool's pinging feature in their >5th edit with the Reply Tool **7. Are Junior Contributors becoming confused/distracted by the `Advanced` affordance?* ** - % of people engage with the `Advanced` affordance, by experience level - Compare the comment completion rate between people who engage with the `Advanced` affordance, and those who do not //*⚠️ Note: @MNeisler and I need to discuss this before we can consider it within the scope of this task.// === Open questions - [ ] How long after v2.0 is released should this analysis happen? - [x] Is new information required to know what text input mode (`source` or `visual`) a user is when they take a certain action (e.g. switching text input modes, abandoning/publishing a comment)? -- No. For any event in EditAttemptStep we will be able to know what editing input mode (source or visual) the user was in when they took the action. -- Caveat: the switching event (between source and visual) has not yet been implemented. //See: T247139#6114499//. === Done - [ ] Analysis is completed that contains the "Workflow engagement metrics" listed above --- i. The pinging feature became available on 17-June-2020: https://www.mediawiki.org/wiki/Talk_pages_project/replying#17_June_2020 --- //This task/analysis, had previously been scoped as a follow up to a then-planned analysis of engagement with Version 1.0: T247138.//

=== Purpose This task is about analyzing how people are engaging with the new Replying tool to: **Help answer these questions:** 1. Are people are having success using the feature? 2. Are people using the tool in ways that negatively impact the experience of others? **So we can decide:** What – if any – changes should be made to the workflow before deploying the feature to more people and with it, beginning an A/B to determine its impact? === Workflow engagement metrics //This section is still being drafted. Once complete, it will continue the information needed make the two decisions listed above.// **META** - For each metric, we'd like to see a breakdown by wiki where the Reply Tool is currently deployed. //You can see where and how the Reply Tool is deployed here: https://www.mediawiki.org/wiki/Talk_pages_project#Active_initiatives .// **1. Are people publishing the replies they start?** - % of people who click reply AND who start typing a comment, grouped by which input mode – `source` / `visual` – they are shown by default - % of people who start typing a comment AND who publish their comment to page, grouped by which input mode – `source` / `visual` – they are shown by default **2. Are people abandoning the replies they start?** - % of people who start typing a comment AND who actively abandon their comment **3. Are people satisfied with the feature?** - % of people who publish at least one comment and turn off DiscussionTools in Beta Features - Of the people who have made at least 1 edit with the Replying workflow, the % of people who posted 1, 2-5, 5-10 and 10+ comments **4. Are people posting disruptive comments?** - % of comments posted using Replying workflow that are reverted - # of people blocked after posting a reply using DiscussionTools - % of comments posted using Replying workflow that are made in good faith **5. What text input mode (`source` or `visual`) are people using to publish comments?** - % of published comments drafted using the `Visual` mode by experience level - % of published comments drafted using the `Source` mode by experience level - % of published comments drafted that involved a contributor switching from the `Source` to the `Visual` and vice versa **6. Of the people who have made at least 1 edit with the Reply Tool //after// 17-June-2020 [i], after how many edits do people use the tool's Pinging Feature for the first time?* ** //This question is relevant to us deciding whether the existence of the tool's pinging feature ought to be communicated more explicitly via something like an onboarding experience per T253434#6334269.// - % of people who tried the tool's pinging feature in their 1st edit with the Reply Tool - % of people who tried the tool's pinging feature in their 2nd edit with the Reply Tool - % of people who tried the tool's pinging feature in their 3rd edit with the Reply Tool - % of people who tried the tool's pinging feature in their 4th edit with the Reply Tool - % of people who tried the tool's pinging feature in their 5th edit with the Reply Tool - % of people who tried the tool's pinging feature in their >5th edit with the Reply Tool **7. Are Junior Contributors becoming confused/distracted by the `Advanced` affordance?* ** - % of people engage with the `Advanced` affordance, by experience level - Compare the comment completion rate between people who engage with the `Advanced` affordance, and those who do not //*⚠️ Note: @MNeisler and I need to discuss this before we can consider it within the scope of this task.// === Open questions - [ ] How long after v2.0 is released should this analysis happen? - [x] Is new information required to know what text input mode (`source` or `visual`) a user is when they take a certain action (e.g. switching text input modes, abandoning/publishing a comment)? -- No. For any event in EditAttemptStep we will be able to know what editing input mode (source or visual) the user was in when they took the action. -- Caveat: the switching event (between source and visual) has not yet been implemented. //See: T247139#6114499//. === Done - [ ] Analysis is completed that contains the "Workflow engagement metrics" listed above --- i. The pinging feature became available on 17-June-2020: https://www.mediawiki.org/wiki/Talk_pages_project/replying#17_June_2020 --- //This task/analysis, had previously been scoped as a follow up to a then-planned analysis of engagement with Version 1.0: T247138.//

=== Purpose This task is about analyzing how people are engaging with the new Replying tool to: **Help answer these questions:** 1. Are people are having success using the feature? 2. Are people using the tool in ways that negatively impact the experience of others? **So we can decide:** What – if any – changes should be made to the workflow before deploying the feature to more people and with it, beginning an A/B to determine its impact? === Workflow engagement metrics //This section is still being drafted. Once complete, it will continue the information needed make the two decisions listed above.// **META** - For each metric, we'd like to see a breakdown by wiki where the Reply Tool is currently deployed. //You can see where and how the Reply Tool is deployed here: https://www.mediawiki.org/wiki/Talk_pages_project#Active_initiatives .// **1. Are people publishing the replies they start?** - % of people who click reply AND who start typing a comment, grouped by which input mode – `source` / `visual` – they are shown by default - % of people who start typing a comment AND who publish their comment to page, grouped by which input mode – `source` / `visual` – they are shown by default **2. Are people abandoning the replies they start?** - % of people who start typing a comment AND who actively abandon their comment **3. Are people satisfied with the feature?** - % of people who publish at least one comment and turn off DiscussionTools in Beta Features - Of the people who have made at least 1 edit with the Replying workflow, the % of people who posted 1, 2-5, 5-10 and 10+ comments **4. Are people posting disruptive comments?** - % of comments posted using Replying workflow that are reverted - # of people blocked after posting a reply using DiscussionTools - % of comments posted using Replying workflow that are made in good faith **5. What text input mode (`source` or `visual`) are people using to publish comments?** - % of published comments drafted using the `Visual` mode by experience level - % of published comments drafted using the `Source` mode by experience level - % of published comments drafted that involved a contributor switching from the `Source` to the `Visual` and vice versa **6. Of the people who have made at least 1 edit with the Reply Tool //after// 17-June-2020 [i], after how many edits do people use the tool's Pinging Feature for the first time?* ** //This question is relevant to us deciding whether the existence of the tool's pinging feature ought to be communicated more explicitly via something like an onboarding experience per T253434#6334269.// - % of people who tried the tool's pinging feature in their 1st edit with the Reply Tool - % of people who tried the tool's pinging feature in their 2nd edit with the Reply Tool - % of people who tried the tool's pinging feature in their 3rd edit with the Reply Tool - % of people who tried the tool's pinging feature in their 4th edit with the Reply Tool - % of people who tried the tool's pinging feature in their 5th edit with the Reply Tool - % of people who tried the tool's pinging feature in their >5th edit with the Reply Tool **7. Are Junior Contributors becoming confused/distracted by the `Advanced` affordance?* ** - % of people engage with the `Advanced` affordance, by experience level - Compare the comment completion rate between people who engage with the `Advanced` affordance, and those who do not //*⚠️ Note: @MNeisler and I need to discuss this before we can consider it within the scope of this task.// === Open questions - [ ] How long after v2.0 is released should this analysis happen? - [x] Is new information required to know what text input mode (`source` or `visual`) a user is when they take a certain action (e.g. switching text input modes, abandoning/publishing a comment)? -- No. For any event in EditAttemptStep we will be able to know what editing input mode (source or visual) the user was in when they took the action. -- Caveat: the switching event (between source and visual) has not yet been implemented. //See: T247139#6114499//. === Done - [ ] Analysis is completed that contains the "Workflow engagement metrics" listed above --- i. The pinging feature became available on 17-June-2020: https://www.mediawiki.org/wiki/Talk_pages_project/replying#17_June_2020 --- //This task/analysis, had previously been scoped as a follow up to a then-planned analysis of engagement with Version 1.0: T247138.//