This task is about determining what – if any – new instrumentation is needed for us to measure the impact the notification interventions being implemented (T273920) as part of the [Talk pages project](https://www.mediawiki.org/wiki/Talk_pages_project/Notifications).
=== Background
Work on this task will begin once we know:
1. The experiments we will run to evaluate the impact of the notification interventions //and//
2. The metrics we will use in these "experiments"
"1." and "2." will be finalized once the [notifications measurement plan](https://docs.google.com/document/d/15eD960rWRzxqYNQpjgsqd8nGwW9cRrpzOfYVd7nkX-0/edit#) (T274215) is drafted.
=== Instrumentation changes
//This section will contain the instrumentation changes needed to evaluate the impact of the notification interventions we have planned (T273920).//
|Event to be tracked| Currently being tracked? Yes/No?|If "no" ticket where tracking will be implemented|
|---|---|---|
|//TBD//|//TBD//|//TBD//
[Instrumentation Spec](https://docs.google.com/spreadsheets/d/1wlGzy6oZ8jbY5-EpydN7SqZnG1ikHTk05ZsZx1euWME/edit#gid=0)
=== Open questions
- [x] 1. How are we – collectively – defining "topics," "comments," and "replies"?
-- See T277349#7087010 for written definitions and/or visit a talk page in debug mode (e.g. https://en.wikipedia.beta.wmflabs.org/wiki/Talk:Cats?dtdebug=1) to see these definitions in practice.
-- //Note: "topics" in the context of the topic subscription feature are limited to `==H2==` headings.//
- [x] 2. Which of the below do we care to know/measure?
-- A. ✅//When/if someone posts a comment in response to a new topic 'you've' started?//
--- We will //potentially// use this information to evaluate the impact of this feature.
---- Emphasis placed on "potentially" above to signal the possibility that T263821 could be de-scoped.
-- B. ✅ //When/if someone comments in a section after you have already commented in it?//
--- We will not use this information to evaluate the impact of this feature; however, we will use it as a "Curiosity" to understand how topic subscriptions might be impacting the cadence of conversations. This has been added as "Curiosity #3" in T280895's task description.
-- ✅ C. //When/if someone replies directly to a comment you've written?//
--- We will use this data as an input to our KPI metric (T280895) in the scenario that T263821 is de-scoped.
- [ ] ~~3. How will we relate comments to sections?~~
-- //Note: this assumes our interest is limited to comments and topic NOT replies.//
-- This question will be answered in T280100.
- [ ] ~~4. How will we relate replies to comments and comments to sections?~~
-- //Note: this assumes we are interested in topics, comments and replies posted in direct response to comments.//
-- This question will be answered in T284200
- [x] 5. Is it a priority to be able to track the full "lifecycle" of the notification flow?
-- //Where "Notification flow" = notifications is sent (start) → response is posted (end).//
-- No and the rationale can be found in T277349#7087010. Instead, we would like to be able to track/define the notification flow as follows:
--- 1) Notification sent
--- 2) Notification delivered
--- 3) Notification opened/read
--- 4) Notification interacted with
- [ ] 6. For new notification instrumentation (read: Echo), to what schema will these new events be added? An existing schema? A new schema?
=== Done
- [ ] Answers to all `===Open questions` are answered and documented in the ticket
- [ ] The table in the `Instrumentation changes` section above is populated with the new instrumentation and/or changes to existing instrumentation that is needed to evaluate the impact of the notification interventions we have planned.