Page MenuHomePhabricator

Document Instrumentation and Schema Needs for Suggested Edits on iOS Proof of Concept
Closed, ResolvedPublic

Description

Background
This fiscal year, the iOS team is seeking to increase unreverted mobile contributions. One of our hypotheses to achieve this growth is by introducing Suggested Edits to the iOS app.

We would like to explore a Suggested Edits task that encourages users to add helpful and accurate alt-text to the 95% of images across Wikipedias that are missing them. Before investing too much into this idea, we will start with a proof of concept and get feedback at the GLAM conference in Uruguay and with LATAM and Caribbean users. If the task proves successful, we will build out the full feature and scale it, if it is not successful we will pivot to Image Recommendations or Article Descriptions.

Decisions Matrix

  • If less than 45% of edits are scored as a 3 or higher then we will pivot to a different suggested edit. If 46%-70% of edits are scored a 3 or higher we will improve guidance or use AI to better assist users. If 71% or higher of edits are scored a 3 or higher we will scale feature.
  • If average edit per unique user is under 3 we will pivot to a different suggested edit. If it is 3-6, we will consider interventions to reduce friction. If it is 7 or higher we can scale
  • If user edit through feature a second day we should proceed with improvements and scaling
  • If less than 55% or less of users are satisfied with feature we will not scale without making changes
  • If more there are more than 30% of users find the task too difficult we will create an intervention to reduce difficulty before scaling. If 80% or more users find the task too difficult we will consider abandoning depending on supplementary responses
  • If the skip rate is 20% higher than image captions on Android we will consider pivoting to a different suggested edit unless evidence points to an intervention that could reduce this rate
  • If we do not have at least 50 people try the feature we will do direct outreach to gain more edits

Curiosity

  • How does the predicted revert rate compare to Android's Image Captions Suggested Edit?
  • Is there a difference user reported feedback based on their familiarity with editing and writing alt-text
  • Is there a difference in metrics by language?

Must haves for proof of concept

  • Entry point in Settings
  • Prominent guidance for writing good alt-text
  • Users ability to get context about the image from the article
  • Users ability to access relevant metadata (can take user to Web)
  • Detection of which images do not have alt-text
  • Ability to store the responses we get to evaluate if they are good alt-text

User stories
Primarily

  • As a participant at the GLAM Conference in Uruguay, I want to test out an alt-text Suggested Edits Task, to get a concrete idea of the concept and provide meaningful feedback
  • As the Director of Product, I want concrete proof an alt-text Suggested Edits would increase edits on iOS, without creating a burden for patrollers

Secondarily

  • As a user with limited data, I want to read alt-text, so even if images are not loaded, I am aware of what is in the image
  • As a user navigating articles with a screen reader, I want alt-text to be available , so that I have the same additional context about an article as users that are not using screen readers

Primary target audiences

  • Round 1: GLAM Conference Attendees
  • Round 2: Spanish and Portuguese speakers in the Americas and Caribbean
  • Round 3: French and English Speakers in the Americas and Caribbean

For the sake of this task, lets create metrics for Round 1 target audiences at a minimum, but there is a possibility of expanding to Round 2.

Task

  • Set and review key indicators to establish success metrics in partnership with Product
  • Submit Measurement Plan for Legal approval
  • Create schema and instrumentation documentation for engineers
  • Create data relates tasks

Reference
For Designs check T347121

Event Timeline

Would be interesting to see avg. task completion rate from start to finish compared to Android's Image Captions Suggested Edits, the Android task is more complex with more steps so it's not apples to apples but still think this would be a useful metric. @JTannerWMF are you ok with adding this to Curiosities?

@SNowick_WMF I discussed with @Sharvaniharan and the thinking here is for us to use a feature-specific stream name, that will use the App Interaction schema. Do you have a particular stream name you would like to use for this prototype? Since this is an iOS-only prototype I was thinking it should be something like ios.suggested_edits_alt_text_prototype.

@Tsevener sounds good, let's use that name, I'll add it to the docs, working on mapping data events now.

Change 968259 had a related patch set uploaded (by Tsevener; author: Tsevener):

[operations/mediawiki-config@master] Add stream config for iOS schema

https://gerrit.wikimedia.org/r/968259

Change 968259 merged by jenkins-bot:

[operations/mediawiki-config@master] Add stream config for iOS schema

https://gerrit.wikimedia.org/r/968259

Mentioned in SAL (#wikimedia-operations) [2023-10-24T13:36:12Z] <samtar@deploy2002> Started scap: Backport for [[gerrit:968259|Add stream config for iOS schema (T347122)]]

Mentioned in SAL (#wikimedia-operations) [2023-10-24T13:37:06Z] <samtar@deploy2002> samtar and tsev: Backport for [[gerrit:968259|Add stream config for iOS schema (T347122)]] synced to the testservers (https://wikitech.wikimedia.org/wiki/Mwdebug)

Mentioned in SAL (#wikimedia-operations) [2023-10-24T13:43:35Z] <samtar@deploy2002> Finished scap: Backport for [[gerrit:968259|Add stream config for iOS schema (T347122)]] (duration: 07m 52s)