Page MenuHomePhabricator

Add an image: leading indicators
Closed, ResolvedPublic


Corresponding task from "add a link": T277355: Add a link: leading indicators

After "add an image" is deployed for two weeks in our pilot wikis, we will calculate the following leading indicators. We will follow the recommended courses of action if the indicators are problematic, and we will use these indicators to decide if and when we should deploy the feature to the other Growth wikis.

IndicatorPlan of Action
Revert rateThis suggests that the community finds the "add an image" edits to be unconstructive. If the revert rate for "add an image" is substantially higher than that of unstructured tasks, we will analyze the reverts in order to understand what causes this increase, then adjust the task in order to reduce the likelihood of edits being reverted.
User rejection rateThis can indicate that we are suggesting a lot of images that are not good matches. If the rejection rate is above 40%, we will QA the image suggestion algorithm and adjust thresholds or make changes to improve the quality of the recommendations.
Over-acceptance rateThis might indicate that some users aren't actually applying judgment to their tasks, meaning we might want to implement different quality gates. (What percentage of users who have a complete session have never rejected or skipped an image? What percentage of users who have five or more complete sessions have never rejected or skipped an image? How many sessions across all users contained only acceptances?)
Task completion rateThis might indicate that there’s an issue with the editing workflow. If the proportion of users who start the "add an image" task and complete it is lower than 55% (completion rate for "add a link"), we investigate where in the workflow users leave and deploy design changes to enable them to continue.


Due Date
Dec 14 2021, 4:00 PM

Event Timeline

@nettrom_WMF -- I created this leading indicators task by modeling it off of "add a link", since the same basic metrics apply well to both structured tasks. We should aim to run these indicators on Dec 14, which gives us two weeks after our Nov 29 deployment.

kostajh triaged this task as Medium priority.Nov 25 2021, 4:23 PM

We have an internal slide deck with the analysis results, and they were posted on-wiki before the holiday break:

What remains is to share these findings with the team, which will happen shortly. Closing this task as resolved.