Background

In fiscal year FY2324, the iOS team worked to increase unreverted mobile contributions T344258. One of our hypotheses to achieve this growth is by introducing Suggested Edits to the iOS app. We first explored a suggested Edits task that encouraged users to add helpful and accurate alt-text to the 95% of images across Wikipedias that are missing them. After getting feedback at the GLAM conference through an experimental build T347121, we feel confident with scaling our experiment in the production version of app. Part of this experiment builds on top of our recent addition to the iOS app of the "Add an image" suggested edit.

This work is part of the FY2425 annual plan to WE1 for Contributor Experience

Hypothesis WE 1.2.5: If we conduct an A/B/C test with the alt-text suggested edits prototype in the production version of the iOS app we can learn if adding alt-text to images is a task newcomers are successful with and ultimately, decide if it's impactful enough to implement as a suggested edit on the Web and/or in the Apps.

Experiment Design

Research questions

• Does reminding users to add alt text prove an effective way to increase the number of edits that include alt text?
• Should adding Alt text for images in articles be a stand-alone suggested edit? (And if so, would it be an appropriate task for newcomers?)
• Would users appreciate prompts for other improvements they can make to the article they are editing?
• Does providing alt text guidance during the flow create higher quality alt-text?

Experiment B - Image Recs editing flow A/B:

• Hypothesis: if we prompt users once to add missing alt text to their image after they have published an image using the Image Recommendations suggested edit, we'll see 60% of editors choose to additionally publish alt text, and 10% will add alt-text for subsequent image recommendation edits made in the next 15 days.
• Audience: logged-in editors have completed an edit in the "Add an image" feature where alt text = null, and who have not already been sorted into experiment group C, excluding iPad devices
• Group A = 50% of "Add an image" editors are sorted into Control group when they forget alt text. They are not prompted to add missing alt text.
• Group B = 50% of "Add an image" editors are sorted into Experiment group when they forget alt text. They are prompted once once to add missing alt text
• Sampling into group A/B happens when a user completes an "Add an image" suggested edit where alt text = null.

Experiment C - Article editing flow D/C:

• Hypothesis: if we prompt users to add missing alt text after they have published an edit on any article containing an image in need of alt text, 4% of editors will go on to add alt text to the image.
• Audience: logged-in editors who edited an article with an image lacking alt text, and who have not already been sorted into experiment group B, excluding iPad devices
• Group D = 50% of editors are sorted into Control group, they do not see any prompt after completing an eligible edit
• Group C = 50% of editors are sorted into Experiment group, they see 1 alt-text prompt after completing an eligible edit.
• Sampling into group D/C should happen immediately after a user publishes an edit that’s eligible for the prompt.
• Eligible edits for follow-up prompt: any edit made by a logged-in user on an article in the main namespace

How will we measure success

Leading indicators to be measured at 15 days

1. 100 edits with alt text values, from at least 25 unique editors. At least 25 edits are from newer editors
2. More than 15 unique editors have been assigned to each experiment group
3. 70% task acceptance rate for group B, at least 10% acceptance rate for group C (# of people who enter the flow / impressions of prompt)
4. Revert rate for newer editors edits in any single group does not exceed 18%

Key Indicators

1. 60% of group B editors publish an additional edit with alt text for the image they were prompted on
2. 4% of group C users add alt text when prompted after editing an article
3. Of group B editors who saw treatment and then make a subsequent image recommendations edit in the next 15 days, 25% add alt-text as a part of that edit
4. 200 images are improved with Alt text, by at least 50 unique editors

Decision Matrix for next steps:

1. If 71% of edits are scored a 3 or higher we will scale the feature. If less than 70% of edits are scored a 3 or higher we will improve guidance or use AI to better assist users.
2. If quality scores for newer editors are more than 50% worse than quality scores for experienced editors, we will not recommend this task be available to newer editors.
3. If we see at least 60% say they would use feature that provided a feed of images in need of alt text, then we will have the confidence to pursue a feed of alt-text suggested edits
4. If 60% or more of respondents say they would be interested in similar edit notifications for articles they are working on, and 60% of respondents are satisfied with the feature (Group C survey responses), we share this information and consider future edit prompts.

Guardrails

1. Edit return rate of editors in group B or C who have received an Alt text prompt does not differ from controls by more than 10%
2. Revert rate for edits from experiment groups does not exceed controls by more than 5 percentage points
3. Human-graded* or actual revert rate for newer editors in experiment groups does not exceed 18%
4. Alt text task completion rate for newer editors is above 25% (Completion rate = number of alt text edits published / those who said “yes” to the prompt and started the flow)

Curiosities

1. Did the overall constructive activation rate in the iOS app increase when we made Image recommendations available to brand new editors?
2. How does the task completion rate, return rate, and revert rate for newer editors’ alt text edits compare with that of experienced editors? With that of comparable rates from Growth suggested edits?
3. How does the human-graded revert rate compare to Android's Image Captions Suggested Edit?
4. Is there a difference in Number of alt text edits & Unique editors by language and geography? (For example, breaking down edits from Latin America vs Europe for Spanish)
5. What is the most common reason that users decide not to act on the prompt?

*Note: for the quality scores and human-graded revert scores, we will partner with an accessibility organization who will be reviewing and grading alt text.

{1} Definition of newer editors: Editors who had fewer than 10 edits on that wiki they are currently editing at the point they entered the experiment

Requirements

Must haves

• Entry point will vary based on experiment group (see below)
• If a user answers "no" to "Add alt text" prompt, gain reason for why
• Prominent guidance for writing good alt-text (onboarding)
• Users ability to get context about the image from the article
• Users ability to access relevant metadata (can take user to Web)
• Detection of which images do not have alt-text
• Ability to publish alt-text
• Alt-text edit should count as a separate edit ONLY when submitted separately
• Alt text is published with automatic edit summary
• Users should be prompted provide feedback about the feature
• Instrumentation that allows us to evaluate the alt-text submitted
• Input field for alt-text in context of image, preview of alt-text in context of our existing editor
• Warning when alt-text exceeds 125 characters
• Entry field should not allow users to add line breaks

Nice to have

• Tooltip tutorial guidance for alt text interface
• Prefilled email option from overflow menu "Problem with Feature"
• Do not allow users to copy and paste in image caption
• Allow users to pinch and zoom on the image thumbnail (in half sheet)

Experiment B Must Haves

Experiment Group B (50% of Image Recs Suggested Edits user): Dedicated flow for Captions and Alt Text AFTER Image Recs

• Updates to Image recs
  • 50+ edit requirement is removed for target languages only, for duration of experiment
  • Additional instrumentation is added to collect alt text & context
• Entry point:
  • After image recommendations edit is submitted; if alt-text = null:
  • Educate user on importance of alt-text and ask if they’d be willing to add alt-text to image
  • If user selects yes; launch alt-text adding flow for most recently added image
  • If users selects No, gain reason
• After task completion:
  • Show a survey when to ask users if it should be a separate task
    • Would you use a feature that provided a feed of images in need of alt text? (Y/N, if no allow for text reason)
  • Return users to next image recommendations suggestion

Experiment C Must Haves

Experiment Group C: 50% of all editors receive encouragement to add alt-text to an image when editing an article with an image

• Entry point:
  • After edit where:
    • An edit was made to any part of the article, and there is an image in the article that is lacking alt text
  • Show prompt that an image in the article is in need of alt-text and explain importance of adding alt-text and ask users if they’re willing to add alt-text
  • If users selects Yes, launch alt-text adding flow for one image in article in need of alt text
  • If users selects No, gain reason

• After task completion
  • Ask satisfaction
  • Ask if they are interested in similar in-time edit suggestions: Would you like to be notified about suggested edits on articles after you edit them? (Y/N, if no allow for text reason)
  • Bring user back to article they were reading

User stories

Primarily

• As a Spanish Wikipedia editor in LATAM using the Add an Image feature for the first time on the Wikipedia iOS app, I would like to build confidence to add alt-text while on the bus, so that I can add alt-text to future image related tasks without concern of making the state of alt-text worse on Wikipedia for low vision users.
• As a Portuguese Wikipedia editor in Brazil, I would like to become aware of opportunities to add alt-text to images in articles I care about, so that I can ensure all editors are able to gain the full context of those articles

Secondarily

• As a user with limited data, I want to read alt-text, so even if images are not loaded, I am aware of what is in the image
• As a user navigating articles with a screen reader, I want quality alt-text to be available , so that I have the same additional context about an article as users that are not using screen readers

Primary target audiences

• Spanish, French and Portuguese speakers in the Americas and Caribbean
• Chinese speakers in North America

Qualitative Testing
Guardrail Groups

• Low Bandwidth Readers
• Blind and Low Vision Users / Accessibility Experts
• Moderators

Key Stakeholders

• Research Team
• GLAM team