Page MenuHomePhabricator

[EPIC] Impact Module: Investigate activation findings from Experiment analysis
Closed, ResolvedPublic

Description

User story:

As a Product Manager, I want to explore the difference in activation rates between the new and old Impact module's empty state on mobile, because I want to be sure all Growth features improve activation and retention.

Background & research:

Impact module leading indicators seem healthy and we've received positive feedback about the new design from newcomers and experienced editors. However when we look at Activation data we see that is appears there is a slightly lower activation rate for the new impact module on mobile:

Screen Shot 2023-02-26 at 6.38.11 PM.png (806×1 px, 390 KB)

Before a newcomer has activated (edited for a first time) they will only see the impact module's empty state. So this change in activation only relates to the empty state, not the actual new impact module. The old empty state design is very similar to the new empty state design, so we would expect to see no change in the Activation rate.

Old Impact moduleNew Impact Module
Screen Shot 2023-02-26 at 6.51.37 PM.png (374×1 px, 201 KB)
Screen Shot 2023-02-26 at 6.52.11 PM.png (472×1 px, 196 KB)
image.png (1×720 px, 104 KB)
Screen Shot 2023-02-26 at 6.57.56 PM.png (1×1 px, 426 KB)
Acceptance Criteria:
  • Resolve all subtasks and ensure the new impact module is ready to scale to more wikis

Related Objects

Event Timeline

From what I can tell we've deployed a patch to have the new impact module only switch out of deactivated state through article edits (Main namespace) per T338640, but that there was also a bug that affected it that got fixed in T341994. I'm going to restart the experiment clock in the notebook that aggregates statistics so that we can monitor whether there's a change after this patch. The aggregated leading indicators from the Levelling Up experiment will continue to exist in our database for comparison.

From what I can tell we've deployed a patch to have the new impact module only switch out of deactivated state through article edits (Main namespace) per T338640, but that there was also a bug that affected it that got fixed in T341994. I'm going to restart the experiment clock in the notebook that aggregates statistics so that we can monitor whether there's a change after this patch. The aggregated leading indicators from the Levelling Up experiment will continue to exist in our database for comparison.

Thank you for reading my mind! :)

It appears that T338640 or one of our other impact module improvements have resolved the activation issue.
There is no longer a significant difference in activation between the experiment and control group:

Screenshot 2023-08-04 at 1.10.45 PM.png (740×1 px, 64 KB)

This is what we would expect as the empty state (before activation) is nearly identical between the two groups.

I consider this epic complete.