Background
We want to measure the impact of adding Image Recommendations feature to the iOS app at 15 days after release, and 30 days after release.
- Release date: TBD - end of April, early May
- 15 Days- X
- 30 Days- X
The Task
- Compare results to baseline data that was collected, and results from Android Image Recommendations 30-day analysis T349357
- Visualize and present the data in a way that is easily understandable to the team
Requirements
- The data should be based on the metrics in the Epic T355270
At 15 days:
Check metric-specific leading indicators:
- Image rejection rate does not exceed 29%
- Edit over-acceptance rate (never skip or reject recommended images) does not exceed 35%
- Task completion rate is not below 30%
- Revert rate does not exceed 18%
At 30 days:
Validate:
- KR 1.1: 600 articles have images in a 30 day period
- KR 1.2: Average at least 6 edits per day per unique user with 50+ edits
- KR 1.3: 10% of eligible Suggested Editors try image recommendations task
- KR 1.4: 30% of those users complete the task again on a separate day in a 15 day period
- KR 1.5: Accept rate does not deviate from Mobile Web or Android by more than 10 percentage points
- KR 1.6: 10% increase in unreverted edits from iOS in the main namespace
Check guardrails:
- KR 1.1: Less than 5% of users report NSFW or offensive content
- KR 1.3: All users spend at least 10s evaluating a task before publishing it
- KR 1.4: Bounce rate does not exceed 30%
- Bounce rate defined as users that click Yes then abandon the flow before publishing
- KR 1.5: At least a 80% task completion rate
- Defined as users that click on Add an image as a task, and actually clicks Yes, No or Not sure (interact with the feature)
- KR 1.6: Revert rate does not exceed 5%
Check curiosities (nice to have):
- KR 1.1: Do these numbers differ by user tenure?
- KR 1.2: Are we seeing differences in these metrics when focusing on our target audiences as compared to general population?
- KR 1.3 At what point in workflow are most frequent dropoff events?
- KR 1.4 How often are users adding captions and alt-text (distinguish between the two)?
- KR 1.5 How often are reverted edits captioned?