Background
Conduct an analysis at 15 days after release into product and 30 days after release.

15 Days- October 25, 2023
30 Days- November 9, 2023

The Task

• Compare results to baseline data that was collected
• Visualize and present the data in a way that is easily understandable to the team

Requirements

• The data should be based on the metrics in the epic

At 15 days we want to measure:

Metric specific Leading Indicators: Indicators to be captured after 15 days
Image rejection rate does not exceed 29%
Edit over-acceptance rate (never skip or reject recommended images) does not exceed 35%
Task completion rate is not below 30%
Revert rate does not exceed 18%

At 30 days we want to validate

• KR 1.1: 2000 articles have images in a 30 day period
• KR 1.2: Average at least 8 edits per day per unique user
• KR 1.3: 15% of eligible Suggested Editors try image recommendations task
  • by app_install_id, just by username in mediawiki_history)
• KR 1.4: 70% of those users complete the task again on a separate day in a 15 day period
• KR 1.5: Reject and Accept rate does not deviate from Mobile Web or MVP by more than 10% (redundant metrics from
• KR 1.6: DAU of Suggested Edits increase overall

At 30 days check guardrails

• KR 1.1: Feature does not worsen gender or geographic bias*
• KR 1.2: Less than 5% of users report NSFW or offensive content
• KR 1.3: Users spend at least 10s evaluating a task before publishing it
• KR 1.4: Bounce rate does not exceed 50%
  • Bounce rate defined as users that click Yes then abandon the flow before publishing
• KR 1.5: At least a 35% task completion rate
  • Defined as users that click on Add an image as a task, and actually clicks Yes, No or Not sure (interact with the feature)
• KR 1.6: Revert rate does not exceed 18%

At 30 days check curiosities (nice to have)
KR 1.1: Do these numbers differ by language or user tenure?
KR 1.2: If this is a user’s first suggested edit, do they go on to try others?
KR 1.4 At what point in workflow are most frequent dropoff events?
KR 1.3: Feature perception by geographically underrepresented groups on large language wikis

Target Quant Regions and Languages to check geo bias

• Spanish Wikipedia
• Portuguese Wikipedia
• Persian Wikipedia
• Hindi Wikipedia