Page MenuHomePhabricator

[Junior and Senior Contributors] Validate usability & usefulness of all Usability Improvements
Closed, ResolvedPublic

Description

This task involves the work with evaluating the extent to which the combination of the Topic Container, Clear Affordances/Visual Cues, and Page Frame interventions impacts Junior Contributors' ability to intuitively recognize and use talk pages to communicate with other people and Senior Contributors' ability to scan the page for the information they need to evaluate the state of a conversation.

Planning Document

Testing plan and protocol

Decisions to be made

What – if any – adjustments to the Topic Container, Clear Affordances/Visual Cues, and Page Frame designs will we prioritize making prior to making the feature available to partner wikis as an opt-in beta feature.

Test requirements

  • Platform: Desktop + Mobile
  • Languages: English, French, Japanese and Arabic
  • Target populations: junior and senior contributors who are not "early adopters" to talk page improvements

Test goals

  • Validate that the improvements help to make the page more recognizable as an interactive discussion space to junior contributors and a more scannable/useful experience for senior contributors
  • Identify any pain points or bugs

Research questions

Primary Questions

  • Do the changes increase the speed with which junior contributors can accurately describe what talk pages can be used for? (Do they recognize these as interactive spaces? Do they view topics as discussions? Do they see that “talk” pages are distinct from “article” pages?
  • Do the changes increase the confidence with which junior contributors can accurately describe what talk pages can be used for?
  • Do the changes increase the speed and confidence with which senior contributors can identify the state of a conversation?
  • Are junior and senior contributors able to successfully reply to and post new topics on discussion pages?

Supplemental Questions

  • Do Junior Contributors understand and find value in the two {Reply} button treatments: one version of the button that appears beneath comments posted without any indentation; a second version of the button that appears beneath comments posted with at least one level of indentation?

To do before study can begin

  • Have a stable beta build of the complete Talk Page experience on desktop
  • Have a stable beta build of the complete Talk Page experience on mobile
  • Write scripts for all of the tests
  • Create recruitment screeners for junior and senior contributors
  • Co-ordinate non-english tests with Research Team (this will need to include translation services)
  • Create all test pages on the beta build
  • Create all user accounts for testing purposes
  • Create document for data logging and synthesis

To do after the study has begun

  • Watch and log the tests
  • Synthesize and summarize the test findings
  • File new tickets for issues that surfaced during the tests (if applicable)

Findings

This section will be populated with a summary of the test findings. This section should also contain links to tickets filed to address usability issues that surfaced during the tests.

Related Objects

StatusSubtypeAssignedTask
OpenNone
OpenNone
OpenNone
OpenNone
Resolvedppelberg
OpenNone
OpenNone
OpenNone
OpenNone
OpenNone
OpenNone
Resolved iamjessklein
Resolvedppelberg
ResolvedEsanders
Resolved iamjessklein
Resolvedppelberg
Resolved iamjessklein
Resolvedmatmarex
Resolved iamjessklein
Resolved iamjessklein
Resolved iamjessklein

Event Timeline

ppelberg renamed this task from Topic Containers: Validate usability to [Junior Contributors] Validate usability & usefulness of topic containers.Jan 22 2022, 2:24 AM
ppelberg updated the task description. (Show Details)

@ppelberg between now and Fall, how many different features are we planning on testing? I am asking because I am wondering if it makes sense to test them at once rather than independently. The testing options would look like:

minimum of 12 tests:

  • Topic Containers: Test for junior contributors and senior contributors (via partner wikis) desktop
  • Topic Containers: Test for junior contributors and senior contributors (via partner wikis) mobile
  • Page Frame: Test for junior contributors and senior contributors (via partner wikis) desktop
  • Page Frame: Test for junior contributors and senior contributors (via partner wikis) mobile
  • Clearer Visual Cues: Test for junior contributors and senior contributors (via partner wikis) desktop
  • Clearer Visual Cues: Test for junior contributors and senior contributors (via partner wikis) mobile

or

minimum of 4 tests:

  • One design: Test for junior contributors and senior contributors (via partner wikis) desktop
  • One design: Test for junior contributors and senior contributors (via partner wikis) mobile

@ppelberg between now and Fall, how many different features are we planning on testing? I am asking because I am wondering if it makes sense to test them at once rather than independently.

Good spot, @iamjessklein. I agree with you in thinking that it will be more efficient and effective to use one design to evaluate the three Usability Improvements (Topic Containers, Clear Affordances/Visual Cues, and Page Frame) with Junior Contributors.

As for whether we take the same approach (one design that contains all three Usability Improvements) to evaluate the interventions with Senior Contributors, I'd like to revisit that question until after we resolve T302120.

I've adjusted our plans by doing the following:

  • Retitling this task to match its expanded scope
  • Marking this task (T293520) as a sub-task of T293523 and T293524 to communicate that usability testing with Junior Contributors of the Clear Affordances/Visual Cues interventions will happen in this task
  • Added the open question about whether we will take the same approach (one design that contains all three Usability Improvements) to evaluate the interventions with Senior Contributors to T302121. See: T302121#7802197.
ppelberg renamed this task from [Junior Contributors] Validate usability & usefulness of topic containers to [Junior Contributors] Validate usability & usefulness of all Usability Improvements .Mar 24 2022, 12:11 AM
ppelberg updated the task description. (Show Details)
iamjessklein renamed this task from [Junior Contributors] Validate usability & usefulness of all Usability Improvements to [Junior and Senior Contributors] Validate usability & usefulness of all Usability Improvements .Apr 5 2022, 7:36 PM
iamjessklein updated the task description. (Show Details)

@ppelberg - I've updated the description with what needs to happen before this test can run. Please review and add in any items that I may have left out.

I've drafted all of the scripts.
@ppelberg can you do a quick scan to validate all of the "goals" listed align with your expectations and if something is missing please add it as a comment.

Next step will be to make the test pages once the prototype is created.

@ppelberg - I've updated script D to include the following:

  • a task about using the sticky header to create a new topic
  • finding the posted new topic on the page after posting

I've updated the status to "done" on the following:

  • Have a stable beta build of the complete Talk Page experience on desktop
  • Have a stable beta build of the complete Talk Page experience on mobile
  • Write scripts for all of the tests
  • Create recruitment screeners for junior and senior contributors
  • Create all test pages on the beta build
  • Create all user accounts for testing purposes

With these items marked off we can go ahead and run the tests.
@BKots-WMF is going to test each test before we run them on usertesting to check for errors.

I'm going to add child tickets for each Test that we are running because it will get confusing.