This task involves the work of conducting usability tests with newcomers to learn how they experience the Suggestion Mode (MVP) and the initial set of suggestions that will be available within it.

Decision(s) to be made

This test is an effort to make the following decision(s)...

• What (if any) adjustments will we make to the Suggestion Mode UX (broadly defined) before evaluating the impact of the experience through a controlled experiment (T404600)?

Research questions

I. Discoverability

• Do newcomers realize there is a new feature where they can find suggestions?

II. Usefulness & clarity

• Which (if any) suggestions do newcomers consider compelling, clear, and ultimately, worth acting on?
• Which (if any) suggestions do newcomers consider confusing, opaque, and ultimately, unmotivated to pursue/act on?

III. Helpfulness

• How do newcomers feel after completing various suggestions?
• Which (if any) suggestions do newcomers find meaningful and inspire them to want to complete more of?
• Which (if any) suggestions do newcomers find boring, not impactful, and ultimately, unlikely to complete again?

IV. Overall usability

• Do newcomers notice edit suggestions?
• What do newcomers understand suggestions to mean? Asked another way: what do newcomers intuitively understand the suggestion "cards" to mean? Do they recognize them as optional actions they can consider taking to improve the content of the article they're editing?
• What – if any – facets of the overarching Suggestion Mode experience do newcomers find confusing? E.g. see open questions "1." and 2." below.

V. Miscellaneous

• What kinds of suggestions would newcomers be excited to receive?

Open questions

This section contains the questions/uncertainties that surface through designing the MVP. Prior to beginning the test, we'll decide which of these questions we'll prioritize using the usability test to help us answer.

• 1. To what extent – if any – do people become confused by the similarities between the blue text highlight which T404604#11233838 specified for suggestions and the blue text highlight which is used for existing text selections and hove effects?@DLynch raised this point in T404604#11233971. - We were not able to test this since we conducted this test using a Figma prototype. We will test this and other interactions in T412389: Suggestion Mode: Conduct usability testing using a coded prototype
• 2. What do people intuitively understand the icon proposed in T404604 to mean?
  • Yes, users naturally noticed the lightbulb icons across the page when entering edit mode, on either desktop and mobile. Then, after completing the onboarding, they clearly understood what suggestions are.
• 3. Do people understand the difference between checks and suggestions? Do they understand why they use different colors and icons?
  • Yes, most users clearly understood that the Check is a warning message that appears because they added something wrong while editing. Users clearly perceive Checks (yellow, warning) as more urgent or corrective than Suggestions.
  • Just one user didn’t understand the difference between blue and yellow message, so we will need to clearly explain in the onboarding that suggestions came from previous user’s edits so they can understand the Checks appear on their own edits instead.

Usability testing findings

Key findings

• Discoverability:
  • Users naturally noticed the Suggestions across the page when entering edit mode, on either desktop and mobile.
  • Overall positive first impression of suggestions.
  • Users understood that suggestions are optional and they can hide or skip them.
  • While reading the onboarding, one participant initially assumed Suggestions were AI-generated.
• Progressive expansion of suggestions (desktop)
  • Users understood the collapsed→expanded pattern once they discovered how to open a card. However, several tried to click the icon itself (not the whole card), causing confusion.
  • Auto-expanding the first Suggestion after completing the onboarding could improve the learning curve and reduce friction.
• Interaction flow & usability
  • Users were able to open, read, and complete Suggestions.
  • They appreciated having the option to turn Suggestions off when distracted.
  • We will need to present a success message/animation when completing a suggestion to give users clear feedback, reinforce progress, and encourage them to complete additional suggestions.
  • We will need to make the ToggleButton prominent on mobile so users don't accidentally turn off suggestions
• Understanding of Suggestions vs. Checks
  • Most users clearly understood that the Check is a warning message that appears because they added something wrong while editing. Users clearly perceive Checks (yellow, warning) as more urgent or corrective than Suggestions.
  • Just one user didn’t understand the difference between blue and yellow message, so we will need to clearly explain in the onboarding that suggestions came from previous user’s edits so they can understand the Checks appear on their own edits instead.
• Expectations and motivations
  • Participants were curious and motivated to open the Suggestions cards to see what they offer.
  • After completing one suggestion, most participants felt motivated to continue and would at least open the remaining cards to see what they offer.
  • Most engaging suggestions are those suggestions that felt more impactful to improve the article. They valued Suggestions that improve legitimacy (citations, verification) and find “writing-style” Suggestions less meaningful. We will address this in T412367 and T412373.
  • Users tend to use Suggestion Mode as a “one-off helper,” not a persistent mode. They see Suggestions as occasional guidance to complete tasks, not as a layer they want to keep visible throughout the entire edit session. So we will need to find other entry points in T412381 to show suggestions in different steps of the editing process.

Acceptance criteria (or done)

• Create prototype (Figma prototype)
• Conduct usability test with newcomers and analyze results
• Decide on what (if any) adjustments will we make to the Suggestion Mode UX

Future tasks

• T412367: Suggestions: Review copies to highlight their impact
• T412373: Suggestion Mode: Prioritize content-impacting suggestions the first times using the Suggestion Mode
• T412381: Suggestion Mode: Find other possible entry points (read mode, after publish, etc.)
• T412389: Suggestion Mode: Conduct usability testing using a coded prototype