Page MenuHomePhabricator

Conduct usability testing of initial edit check user experience (mobile)
Closed, ResolvedPublic

Description

This task involves the work of conducting an initial round of usability tests of the first edit check we are developing: a check that prompts people to explicitly decide whether the new content they are attempting to add warrants a reference or not (design work is happening in T325711).

Decisions to be made

  • What – if any – aspects of the proposed mobile reference check user experience (T325711) will we revise prior to investing in implementing these designs in code and subsequently, inviting volunteers to experiment with using in a production or production-like environment?

Learning Objectives/Research Questions

  1. What ambiguities/points of confusion/holes/etc. do people encounter when attempting to navigate the reference check experience.
    • Where "attempting to navigate" in this context refers to doing things like:
      • Knowing what they are being asked to decide
      • Knowing what content they are being asked make a decision about
        • Choosing not to add a citation to the content they are being asked about
        • Choosing to add a citation to the content they are being asked about
          • Generating a citation
          • Inserting said citation
      • Proceeding to publish the edit they will have just "finished" making
      • Moving "forward" and "backward" within the edit flow
        • E.g. "Say you notice a typo while deciding whether the content you're adding warrants a reference or not. How would you intuitively think to go back to edit this text?"

Test protocol

Edit Check – Unmoderated Research Protocol, Android Web

Recruitment filters

Test #1

  • People who:
    • Read Wikipedia with some regularity (e.g. monthly/weekly)
    • Speak English as a second language
    • Living in/from a non-Western context (e.g. outside of Norther America, Canada, UK, Australia)
    • Have some experienced editing Wikipedia

Test #2

  • People who:
    • Read Wikipedia with some regularity (e.g. monthly/weekly)
    • Speak English as a second language
    • Living in/from a non-Western context (e.g. outside of Norther America, Canada, UK, Australia)
    • Do NOT have experience editing Wikipedia

Findings

This section will eventually contain a synthesis of what we learned through conducting these usability tests.

FindingDescriptionResulting action (if any)

Done

Event Timeline

ppelberg added a subscriber: matmarex.

Update: I added a question about general flow navigation to the task description based on what @matmarex and @nayoub talked about during today's team discussion.

Updated the task description's Recruitment filters based on the conversation @nayoub and I had offline today.

@nayoub: can you please update the ticket with these test findings? IIRC, you had put together some slides. Tho, a a quick search didn't return the results I expected...

Usability test summary

Findings

Positive

  • All participants were able to complete & navigate all the steps
  • All participants found the Edit Check flow natural in their editing experience
  • Users appreciated the highlighting of their edit as area of focus
  • Users have an understanding that content on Wikipedia needs to verifiable and reliable
  • Aside from a few users mentioning citations as their next course of action, blue links were the most common.

Concerns

  • Due to the limitations of the study & prototype, some users chose NOT to add a citation
  • When navigating back to add another sentence, some users ended up publishing their edit and returning to make a new edit
  • One user was confused by the “Generate” button in Citoid
  • “Thank you” success banner can be distracting on the UI if it appears for too long
  • One user mentioned that the preview UI should be clearly labeled as such to avoid confusion

Surprising

  • All participants added information to the Edit Summary
  • A few participants found the “pencil” icon effective to express the ability to edit the article (inferring that it was the objective of the test we were running)
  • Some users did NOT understand that they needed to tap on the pencil icon in the section
  • Some users were confused by the pencil icon in the editor toolbar
Test design

Research question
What ambiguities do people encounter when attempting to navigate the Edit Check experience on mobile?

Participants

  • 5 junior editors
  • 5 newcomers
  • 50% Female + 50% Male
  • Outside of our most represented demographics

Test protocol

  1. Make an edit in the Biography section of artist Aida Muleneh
  2. Arrive to the Edit Check UI
  3. Decide whether to add or not add a citation
  4. Follow the steps based on step 3.
  5. Return to step 1. to add a new sentence to the article
  6. Continue all the way to publishing

NOTE: this testing was completed in March 2023.