We will need to instrument the newcomer homepage to facilitate a controlled experiment and to answer our high-level research questions about the feature:
- Does having the homepage treatment increase activation?
- Does having the homepage treatment increase retention?
- Do newcomers go to and return to the homepage as a central location to get oriented?
- Which modules engage newcomers and to what extent are they engaged?
- Are we able to effectively personalize the homepage?
- Do visits to the homepage lead to edits?
Note: "having the homepage treatment" refers to having access to the homepage as well as receiving whatever other features we implement to drive users to their homepage, such as links or banners elsewhere in the wiki.
An important element is to assemble our written measurement plan. Learning from previous retros, we will go through these steps in this order:
- @nettrom_WMF starts a draft in a Google Doc.
- @MMiller_WMF collaborate to refine the draft.
- @JTannerWMF circulates this draft to various approvers to begin to inform them and receive initial feedback.
- At the same time, the team starts to comment on the draft and @nettrom_WMF and @MMiller_WMF will incorporate changes. We will set a cutoff date when all changes should be complete.
- Implementation begins. Engineers may begin implementation before this step if there is sufficient confidence about parts of the plan.
- @JTannerWMF completes the approval process.
List of additional ideas that may be incorporated in our instrumentation:
- We should instrument all the personal tools links in the top navigation, not just the link we're adding by building the homepage.
- We may want to instrument "hovers" over the modules, since not all the modules have a call to action, and hovering may indicate engagement.