New Impact Module: experiment analysis
Open, MediumPublic
Actions

Assigned To

Authored By

	KStoller-WMF
	Jan 21 2023, 4:22 PM

Description

User story:

As the Growth team, I want to understand how the new Impact Module has changed newcomer behavior, because I want to see if this work was effective and I want to know if any improvements should be made.
As a Wikimedian, I want to make sure new Growth features are working well for newcomers, because then I'll be more accepting of the change when they are rolled out on my wiki.

Acceptance Criteria:

Status	Subtype	Assigned	Task
Open		KStoller-WMF	T248330 [EPIC] Growth: Positive reinforcement
Open		None	T222310 [EPIC] Positive reinforcement: New Impact Module
Open		nettrom_WMF	T327581 New Impact Module: experiment analysis
Resolved		nettrom_WMF	T330614 New Impact module's empty state on mobile: research spike
Resolved	BUG REPORT	kostajh	T330692 Impact Module: don't show "Updated design" info drawer to new users
Declined		None	T330695 New Impact module's empty state on mobile: Design iteration

KStoller-WMF triaged this task as Medium priority.Jan 21 2023, 4:22 PM

KStoller-WMF created this task.

Just noting that this initial experiment analysis is leading us to look into activation and the current Impact Module empty state further: T330614: New Impact module's empty state on mobile: research spike

The team noted a few questions that would help us decide on next steps:

Comparing the funnel of users who tap on empty impact module, then press "See suggested edits". (If there is a marked decrease in users who click on "See suggested edits" in the new impact module, that could help explain the lower activation rate.)
Can we correlate the activation data with the device information (ex. Mobile screen sizes)?
We are using ~6 weeks of data, and there have been several changes during that time. If we compare the first 3 weeks to the second 3 weeks, does the difference in activation rate between the experiment groups look about the same?