Page MenuHomePhabricator

Update guardrail for time spent evaluating after Warning implementation
Closed, ResolvedPublic

Description

Background

For the Add an image feature on iOS, we had a guardrail KR 1.3: All users spend at least 10s evaluating a task before publishing it. In our initial analysis, we did not meet this, 15.3% of edits were made in under 10s (3.02% of unique editors made edits in under 10 seconds). The time was measured from the point a user accepts the image suggestion, to the point of publishing.

Since that point, we have added a warning screen that warns users who have not spent at least 10 second evaluating the image. (T364049: [S] Add review article/image warning screen for Image Recs), deployed on July 8.

Task

  • Measure Guardrail 1.3 again to see if fewer editors made edits in under 10s during the time period of ~July 8 - August 8.

Event Timeline

HNordeenWMF moved this task from Needs Triage to Up next on the Wikipedia-iOS-App-Backlog board.

We are still seeing 22% of edits made by users using app VERSION 7.5.3.3679 and later that show time_spent as under 10 sec:
Date Range: 2024-07-08 - 2024-08-08

Total EditsUnder 10 sec Edits% Under 10 sec Edits
69515522.3%

For this dataset I pulled only events from users using app VERSION 7.5.3.3679 and later which have new Warning function, counts only reflect those users. Data

Note: I also omitted events from these counts where revision_id = 0 of which there were 283 in this date range - these may not be actual edit events, and 75.3% of these had under 10 sec time_spent values. I pulled data for these events to start to investigate but we should discuss these edits in next iOS Data meeting.