Change Details

This task is about determining what – if any – new instrumentation is needed for us to measure the impact the notification interventions being implemented (T273920) as part of the [Talk pages project](https://www.mediawiki.org/wiki/Talk_pages_project/Notifications). === Background Work on this task will begin once we know: 1. The experiments we will run to evaluate the impact of the notification interventions //and// 2. The metrics we will use in these "experiments" "1." and "2." will be finalized once the [notifications measurement plan](https://docs.google.com/document/d/15eD960rWRzxqYNQpjgsqd8nGwW9cRrpzOfYVd7nkX-0/edit#) (T274215) is drafted. === Instrumentation changes //This section will contain the instrumentation changes needed to evaluate the impact of the notification interventions we have planned (T273920).// |Event to be tracked| Currently being tracked? Yes/No?|If "no" ticket where tracking will be implemented| |---|---|---| |//TBD//|//TBD//|//TBD// === Open questions - [x] 1. How are we – collectively – defining "topics," "comments," and "replies"? -- See T277349#7087010 for written definitions and/or visit a talk page in debug mode (e.g. https://en.wikipedia.beta.wmflabs.org/wiki/Talk:Cats?dtdebug=1) to see these definitions in practice. -- //Note: "topics" in the context of the topic subscription feature are limited to `==H2==` headings.// - [x] 2. Which of the below do we care to know/measure? -- A. ✅//When/if someone posts a comment in response to a new topic 'you've' started?// --- We will //potentially// use this information to evaluate the impact of this feature. ---- Emphasis placed on "potentially" above to signal the possibility that T263821 could be de-scoped. -- B. ✅ //When/if someone comments in a section after you have already commented in it?// --- We will not use this information to evaluate the impact of this feature; however, we will use it as a "Curiosity" to understand how topic subscriptions might be impacting the cadence of conversations. This has been added as "Curiosity #3" in T280895's task description. -- ✅ C. //When/if someone replies directly to a comment you've written?// --- We will use this data as an input to our KPI metric (T280895) in the scenario that T263821 is de-scoped. - [ ] ~~3. How will we relate comments to sections?~~ -- //Note: this assumes our interest is limited to comments and topic NOT replies.// -- This question will be answered in T280100. - [ ] ~~4. How will we relate replies to comments and comments to sections?~~ -- //Note: this assumes we are interested in topics, comments and replies posted in direct response to comments.// -- This question will be answered in T284200 - [x] 5. Is it a priority to be able to track the full "lifecycle" of the notification flow? -- //Where "Notification flow" = notifications is sent (start) → response is posted (end).// -- No and the rationale can be found in T277349#7087010. Instead, we would like to be able to track/define the notification flow as follows: --- 1) Notification sent --- 2) Notification delivered --- 3) Notification opened/read --- 4) Notification interacted with === Done - [ ] Answers to all `===Open questions` are answered and documented in the ticket - [ ] The table in the `Instrumentation changes` section above is populated with the new instrumentation and/or changes to existing instrumentation that is needed to evaluate the impact of the notification interventions we have planned.

This task is about determining what – if any – new instrumentation is needed for us to measure the impact the notification interventions being implemented (T273920) as part of the [Talk pages project](https://www.mediawiki.org/wiki/Talk_pages_project/Notifications). === Background Work on this task will begin once we know: 1. The experiments we will run to evaluate the impact of the notification interventions //and// 2. The metrics we will use in these "experiments" "1." and "2." will be finalized once the [notifications measurement plan](https://docs.google.com/document/d/15eD960rWRzxqYNQpjgsqd8nGwW9cRrpzOfYVd7nkX-0/edit#) (T274215) is drafted. === Instrumentation changes //This section will contain the instrumentation changes needed to evaluate the impact of the notification interventions we have planned (T273920).// |Event to be tracked| Currently being tracked? Yes/No?|If "no" ticket where tracking will be implemented| |---|---|---| |//TBD//|//TBD//|//TBD// === Open questions - [x] 1. How are we – collectively – defining "topics," "comments," and "replies"? -- See T277349#7087010 for written definitions and/or visit a talk page in debug mode (e.g. https://en.wikipedia.beta.wmflabs.org/wiki/Talk:Cats?dtdebug=1) to see these definitions in practice. -- //Note: "topics" in the context of the topic subscription feature are limited to `==H2==` headings.// - [x] 2. Which of the below do we care to know/measure? -- A. ✅//When/if someone posts a comment in response to a new topic 'you've' started?// --- We will //potentially// use this information to evaluate the impact of this feature. ---- Emphasis placed on "potentially" above to signal the possibility that T263821 could be de-scoped. -- B. ✅ //When/if someone comments in a section after you have already commented in it?// --- We will not use this information to evaluate the impact of this feature; however, we will use it as a "Curiosity" to understand how topic subscriptions might be impacting the cadence of conversations. This has been added as "Curiosity #3" in T280895's task description. -- ✅ C. //When/if someone replies directly to a comment you've written?// --- We will use this data as an input to our KPI metric (T280895) in the scenario that T263821 is de-scoped. - [ ] ~~3. How will we relate comments to sections?~~ -- //Note: this assumes our interest is limited to comments and topic NOT replies.// -- This question will be answered in T280100. - [ ] ~~4. How will we relate replies to comments and comments to sections?~~ -- //Note: this assumes we are interested in topics, comments and replies posted in direct response to comments.// -- This question will be answered in T284200 - [x] 5. Is it a priority to be able to track the full "lifecycle" of the notification flow? -- //Where "Notification flow" = notifications is sent (start) → response is posted (end).// -- No and the rationale can be found in T277349#7087010. Instead, we would like to be able to track/define the notification flow as follows: --- 1) Notification sent --- 2) Notification delivered --- 3) Notification opened/read --- 4) Notification interacted with - [ ] 6. For new notification instrumentation (read: Echo), to what schema will these new events be added? An existing schema? A new schema? === Done - [ ] Answers to all `===Open questions` are answered and documented in the ticket - [ ] The table in the `Instrumentation changes` section above is populated with the new instrumentation and/or changes to existing instrumentation that is needed to evaluate the impact of the notification interventions we have planned.

This task is about determining what – if any – new instrumentation is needed for us to measure the impact the notification interventions being implemented (T273920) as part of the [Talk pages project](https://www.mediawiki.org/wiki/Talk_pages_project/Notifications). === Background Work on this task will begin once we know: 1. The experiments we will run to evaluate the impact of the notification interventions //and// 2. The metrics we will use in these "experiments" "1." and "2." will be finalized once the [notifications measurement plan](https://docs.google.com/document/d/15eD960rWRzxqYNQpjgsqd8nGwW9cRrpzOfYVd7nkX-0/edit#) (T274215) is drafted. === Instrumentation changes //This section will contain the instrumentation changes needed to evaluate the impact of the notification interventions we have planned (T273920).// |Event to be tracked| Currently being tracked? Yes/No?|If "no" ticket where tracking will be implemented| |---|---|---| |//TBD//|//TBD//|//TBD// === Open questions - [x] 1. How are we – collectively – defining "topics," "comments," and "replies"? -- See T277349#7087010 for written definitions and/or visit a talk page in debug mode (e.g. https://en.wikipedia.beta.wmflabs.org/wiki/Talk:Cats?dtdebug=1) to see these definitions in practice. -- //Note: "topics" in the context of the topic subscription feature are limited to `==H2==` headings.// - [x] 2. Which of the below do we care to know/measure? -- A. ✅//When/if someone posts a comment in response to a new topic 'you've' started?// --- We will //potentially// use this information to evaluate the impact of this feature. ---- Emphasis placed on "potentially" above to signal the possibility that T263821 could be de-scoped. -- B. ✅ //When/if someone comments in a section after you have already commented in it?// --- We will not use this information to evaluate the impact of this feature; however, we will use it as a "Curiosity" to understand how topic subscriptions might be impacting the cadence of conversations. This has been added as "Curiosity #3" in T280895's task description. -- ✅ C. //When/if someone replies directly to a comment you've written?// --- We will use this data as an input to our KPI metric (T280895) in the scenario that T263821 is de-scoped. - [ ] ~~3. How will we relate comments to sections?~~ -- //Note: this assumes our interest is limited to comments and topic NOT replies.// -- This question will be answered in T280100. - [ ] ~~4. How will we relate replies to comments and comments to sections?~~ -- //Note: this assumes we are interested in topics, comments and replies posted in direct response to comments.// -- This question will be answered in T284200 - [x] 5. Is it a priority to be able to track the full "lifecycle" of the notification flow? -- //Where "Notification flow" = notifications is sent (start) → response is posted (end).// -- No and the rationale can be found in T277349#7087010. Instead, we would like to be able to track/define the notification flow as follows: --- 1) Notification sent --- 2) Notification delivered --- 3) Notification opened/read --- 4) Notification interacted with - [ ] 6. For new notification instrumentation (read: Echo), to what schema will these new events be added? An existing schema? A new schema? === Done - [ ] Answers to all `===Open questions` are answered and documented in the ticket - [ ] The table in the `Instrumentation changes` section above is populated with the new instrumentation and/or changes to existing instrumentation that is needed to evaluate the impact of the notification interventions we have planned.