User story & summary:
As the Growth team, I want to measure the impact of this new feature, so that I can determine if our experiment is successful.
As the Growth team PM, I want to measure the impact of this new feature, so that I can determine if we should make further improvements to this feature or rollback and end this experiment.
The plan is due by the end of Q1 FY 24-25 (Sept 30th) so that we can begin the experiment in Q2.
Background & research:
https://www.mediawiki.org/wiki/Growth/Community_Updates
The Growth team will create a new module on the Newcomer homepage that can be used by communities to highlight important community events and initiatives. This Community Updates module will be Community Configurable, so communities can decide to highlight specific events, projects, campaigns, and initiatives, that are set by admins using Community Configuration.
Initial product-level measurement notes:
Do users interact with this new module?
Metric: The click through rate of homepage visitors in the experiment group (that receives this new module).
Did this new module lead to increased project/event/campaign interest?
Metric: When A/B tested, do more editors that receive this module on their Homepage sign up than editors who were part of the control group?
Are there any drawbacks, negative patterns, or anti-patterns that arise from using this module?
Metric: Review productivity and retention for experiment and control group.
Does this module help fill knowledge gaps?
TBD: This will be measured at the Key Result level, and I'm not sure if we can measure this for this experiment.
Design:
Acceptance Criteria
- Initial draft of Experimentation Plan created.
- Data Analyst + Dennis (Growth EM) + Kirsten(Growth PM) brainstorm and expand questions and initial ideas.
- Data Analyst writes up a rough draft.
- Dennis + Kirsten review draft.
- Draft is shared with the Growth team.
- Data Analyst revises the specification based on comments and iterates with Dennis (Growth EM) and/or Kirsten(Growth PM).
- document initial measurement plan on wiki: https://www.mediawiki.org/wiki/Growth/Community_Updates#Measurement_and_Results