Page MenuHomePhabricator

User Testing Plan for Isolated Section Editing Feature
Closed, ResolvedPublic

Description

Testing script

Google Doc for Testing script


Testing goals

Section Editing is a new feature on the mobile web Wikipedia experience. We will be utilizing an unmoderated, task-based user test to test the discoverability and usability of the feature as well as to receive qualitative feedback on the feature.

Discovery
  • Can users successfully find the section edit buttons?
  • Are users easily able to find the section that they intended to edit when in edit mode?
Editing
  • Can users easily edit content they intended to edit?
  • Can users easily publish content they intended to edit?
Excessive scrolling
  • Are users able to navigate through content with ease? (More ease than in current implementation)
Exiting
  • Can users cancel out of their edit and return to reading?

Testing Information

Test format

This test will utilize UserTesting.com to recruit users, record sessions and ‘facilitate’ the test.
The test is an unmoderated, task based test. Users will be given a set of tasks to complete in a 15 minute time frame, followed by a set of 4 written questions. Users will perform the test remotely.

Testers

5 users will be recruited using UserTesting.com.

Event Timeline

JTannerWMF triaged this task as Normal priority.Nov 20 2018, 5:27 PM
JTannerWMF created this task.
JTannerWMF updated the task description. (Show Details)
JTannerWMF moved this task from To Triage to Q4 on the VisualEditor board.Nov 20 2018, 5:31 PM
JTannerWMF updated the task description. (Show Details)

Yes, I recommend user testing against against the current design. This feature should be A/B testable.

The scenario for the test is:

When I find part of an article that I want to edit and click the section editing button ...
I want to click the section editing button and see that editable content secluded from the rest of the article...
so that I can focus my attention on the content I want to edit.

We might want to test these sorts of things:

  • Has this iteration improved the experience?
  • Is it easier to find what you intended to edit this way?
  • Are you more likely to complete your edit with this fix?

I'd love review from @Ryasmeen @dchen @Whatamidoing-WMF @DannyH

iamjessklein moved this task from Q4 to Current work on the VisualEditor board.Nov 30 2018, 12:32 AM
iamjessklein edited projects, added VisualEditor (Current work); removed VisualEditor.
iamjessklein moved this task from Code review to QA on the VisualEditor (Current work) board.
iamjessklein added a subscriber: DannyH.
iamjessklein renamed this task from User Testing of Feature to User Testing of Isolated Section Editing .Nov 30 2018, 12:37 AM

We can user usertesting.com and myrecruit to test this.
We want to know if this is expected behavior when a user selects the section editing button.

Is there somewhere that we will have access to a prototype to use for this?
cc @dchan

Ryasmeen moved this task from QA to Stalled/Waiting on the VisualEditor (Current work) board.

Yes, I recommend user testing against against the current design. This feature should be A/B testable.
The scenario for the test is:
When I find part of an article that I want to edit and click the section editing button ...
I want to click the section editing button and see that editable content secluded from the rest of the article...
so that I can focus my attention on the content I want to edit.
We might want to test these sorts of things:

  • Has this iteration improved the experience?
  • Is it easier to find what you intended to edit this way?
  • Are you more likely to complete your edit with this fix?

I'd love review from @Ryasmeen @dchen @Whatamidoing-WMF @DannyH

@iamjessklein: These are all good points to measure user satisfaction. Since, I am more inclined to do negative testing, I would like your opinion on the following two points to determine user dissatisfaction:

  • Has this caused some kind of dependency failure in terms of user experience? Such as re-using/re-adding references should work properly when in secluded section editing mode.
  • Is it causing addition of redundant information, since the entire article is not visible at the time of editing?

@Ryasmeen I think those questions perfectly compliment the ones I wrote. So what are the next steps? I think the first is finding out where the prototype will be and then we can work together on cobbling together a test script. Thoughts?

Ryasmeen added a comment.EditedDec 12 2018, 9:02 PM

@iamjessklein: Yup! I think @dchan already suggested to have the prototype on our demo page: https://doc.wikimedia.org/oojs-ui/master/

Once we have that ready, I will able to see more scenarios/cases and help with designing the user testing along with you and @DannyH.

iamjessklein renamed this task from User Testing of Isolated Section Editing to User Testing of Isolated Section Editing prototype.Jan 10 2019, 1:58 PM
  • Moderated Testing - I have the opportunity to do some moderated/guerrilla testing with mobile editors over the next few weeks. I would love to run the protocol that we identified above and have them test it out and provide feedback in person and if possible, on the project page. To do this, I will draft some copy with @DannyH and work with him and figure out with @JTannerWMF the best location on Wiki for this.
  • Usertesting.com - Considering that this prototype will effect mobile and desktop users, I would recommend us testing both - on usertesting.com.

cc/ @dchan @Ryasmeen

Over the weekend, I did a talk and performed some guerrilla testing at the NYC Wikipedia Day. In terms of the testing, I did a bunch in person and then also directed testers to a wiki page on the prototype with a call to action for testing it out and providing feedback on the talk page.

Here are some observations about the prototype (after testing about two dozen editors - almost all new to mobile web editing, but almost all experts at using Wikipedia on desktop web:

  • Most users expected this functionality
  • As we had slow, conference wifi, the lack of having some sort solution T210631 for engaging the user and informing them what's happening during these moments really showed.
  • Switching into visual mode is unexpected and disruptive for most users and represents a complete break in workflow.

Although I think that this ticket still stands, I'd recommend combining this section editing testing with the overlay testing T211597 when we are ready to test that.

iamjessklein renamed this task from User Testing of Isolated Section Editing prototype to User Testing Plan for Isolated Section Editing Feature.Feb 11 2019, 4:31 PM
iamjessklein updated the task description. (Show Details)
iamjessklein added a subscriber: ppelberg.

I updated the description of the ticket to include some framing around how we will do recruiting and user testing. @ppelberg - take a look and add any additional questions that you think we should be considering through this test.

iamjessklein updated the task description. (Show Details)EditedFeb 11 2019, 4:38 PM

Another thing that we should incorporate in some form is the asynchronous testing that we have up on the wiki already

iamjessklein updated the task description. (Show Details)Feb 12 2019, 2:48 PM
iamjessklein added a comment.EditedFeb 20 2019, 2:36 PM

Over the next two weeks, we will be conducting internal testing "dogfooding."
The internal cohort of testers will include the Editing team and the original cohort of testers included in the expert review that we did during the research phases of this work T201551

I wrote a draft script in the testing doc under the section "Instructions for Internal Testing." Please review and edit as necessary. Let me know here if you give this a 👍or 👎to send out to our internal cohort of testers. When I get a thumbs up, I will update the wiki page with similar instructions.

cc/ @ppelberg @Ryasmeen @Esanders

ppelberg added a subscriber: WhatamIdoing.EditedFeb 27 2019, 9:02 PM

Testing Plan

We will solicit, process and prioritize qualitative feedback and bugs through the following channels:

AudienceFeedback channelChannel ownerTranslatorFeedback outputPrioritization
Editing TeamGoogle Chat@iamjesskleinProgram + Eng. ManagerPhabricator@iamjessklein @marcella @ppelberg
WMF DesignSlack@iamjesskleinProgram + Eng. ManagerPhabricator@iamjessklein @marcella @ppelberg
All wikisMediaWiki project page@WhatamIdoingProgram + Eng. ManagerPhabricator@iamjessklein @marcella @ppelberg
Hebrew wikiHebrew Wikipedia: City Square@WhatamIdoingProgram + Eng. ManagerPhabricator@iamjessklein @marcella @ppelberg

Definitions

  • Audience : who we are seeking feedback from
  • Feedback : bugs, qualitative feedback
  • Feedback channel: the communication channel through which we will be seeking feedback
  • Channel owner: the person on the Editing Team who will facilitate the given "Feedback channel" and relate the feedback raised to Program Manager and Engineering
  • Translator: the person on the Editing Team who will translate the accumulated feedback (i.e. bugs, qualitative feedback) into Phabricator tickets
  • Feedback output: where and how the feedback we receive will be incorporated into the Editing Team's workflow
ppelberg added a comment.EditedFeb 27 2019, 9:40 PM

usertesting.com Testing

Absent from the "Audience" column" in the "Testing Plan" above is "usertesting.com" – a feedback channel that had previously been included in our testing plans.

Decision

We have decided not to pursue qualitative feedback through usertesting.com for this feature release.

Rationale

We have decided on the above for the following reasons:

  1. We think we will be able to reach the contributors for whom Section Editing has been intended [1] through the All Wikis and Hebrew wiki tests we are running.
  2. We think the questions our usertesting.com testing was originally intended to answer [2] will be satisfied through the test we are running with All Wikis and Hebrew wiki, described in Testing Plan table above

References

  1. https://www.mediawiki.org/wiki/VisualEditor_on_mobile_report#Personas
  2. The description of T209986 represents the intention of our usertesting.com testing: "...to test the discoverability and usability of the feature as well as to receive qualitative feedback on the feature."
ppelberg closed this task as Resolved.Feb 27 2019, 9:43 PM
Restricted Application added a project: User-Ryasmeen. · View Herald TranscriptFeb 27 2019, 9:43 PM