Yes, I recommend user testing against against the current design. This feature should be A/B testable.

The scenario for the test is:

When I find part of an article that I want to edit and click the section editing button ...
I want to click the section editing button and see that editable content secluded from the rest of the article...
so that I can focus my attention on the content I want to edit.

We might want to test these sorts of things:

• Has this iteration improved the experience?
• Is it easier to find what you intended to edit this way?
• Are you more likely to complete your edit with this fix?

I'd love review from @Ryasmeen @dchen @Whatamidoing-WMF @DannyH

We can user usertesting.com and myrecruit to test this.
We want to know if this is expected behavior when a user selects the section editing button.

Is there somewhere that we will have access to a prototype to use for this?
cc @dchan

In T209986#4787872, @iamjessklein wrote:

Yes, I recommend user testing against against the current design. This feature should be A/B testable.

The scenario for the test is:

When I find part of an article that I want to edit and click the section editing button ...
I want to click the section editing button and see that editable content secluded from the rest of the article...
so that I can focus my attention on the content I want to edit.

We might want to test these sorts of things:

• Has this iteration improved the experience?
• Is it easier to find what you intended to edit this way?
• Are you more likely to complete your edit with this fix?

I'd love review from @Ryasmeen @dchen @Whatamidoing-WMF @DannyH

@iamjessklein: These are all good points to measure user satisfaction. Since, I am more inclined to do negative testing, I would like your opinion on the following two points to determine user dissatisfaction:

• Has this caused some kind of dependency failure in terms of user experience? Such as re-using/re-adding references should work properly when in secluded section editing mode.

• Is it causing addition of redundant information, since the entire article is not visible at the time of editing?

@Ryasmeen I think those questions perfectly compliment the ones I wrote. So what are the next steps? I think the first is finding out where the prototype will be and then we can work together on cobbling together a test script. Thoughts?

@iamjessklein: Yup! I think @dchan already suggested to have the prototype on our demo page: https://doc.wikimedia.org/oojs-ui/master/

Once we have that ready, I will able to see more scenarios/cases and help with designing the user testing along with you and @DannyH.

• Moderated Testing - I have the opportunity to do some moderated/guerrilla testing with mobile editors over the next few weeks. I would love to run the protocol that we identified above and have them test it out and provide feedback in person and if possible, on the project page. To do this, I will draft some copy with @DannyH and work with him and figure out with @JTannerWMF the best location on Wiki for this.

• Usertesting.com - Considering that this prototype will effect mobile and desktop users, I would recommend us testing both - on usertesting.com.

cc/ @dchan @Ryasmeen

Over the weekend, I did a talk and performed some guerrilla testing at the NYC Wikipedia Day. In terms of the testing, I did a bunch in person and then also directed testers to a wiki page on the prototype with a call to action for testing it out and providing feedback on the talk page.

Here are some observations about the prototype (after testing about two dozen editors - almost all new to mobile web editing, but almost all experts at using Wikipedia on desktop web:

• Most users expected this functionality
• As we had slow, conference wifi, the lack of having some sort solution T210631 for engaging the user and informing them what's happening during these moments really showed.
• Switching into visual mode is unexpected and disruptive for most users and represents a complete break in workflow.

Although I think that this ticket still stands, I'd recommend combining this section editing testing with the overlay testing T211597 when we are ready to test that.

I updated the description of the ticket to include some framing around how we will do recruiting and user testing. @ppelberg - take a look and add any additional questions that you think we should be considering through this test.

Another thing that we should incorporate in some form is the asynchronous testing that we have up on the wiki already

Over the next two weeks, we will be conducting internal testing "dogfooding."
The internal cohort of testers will include the Editing team and the original cohort of testers included in the expert review that we did during the research phases of this work T201551

I wrote a draft script in the testing doc under the section "Instructions for Internal Testing." Please review and edit as necessary. Let me know here if you give this a 👍or 👎to send out to our internal cohort of testers. When I get a thumbs up, I will update the wiki page with similar instructions.

cc/ @ppelberg @Ryasmeen @Esanders

Testing Plan

We will solicit, process and prioritize qualitative feedback and bugs through the following channels:

Definitions

• Audience : who we are seeking feedback from
• Feedback : bugs, qualitative feedback
• Feedback channel: the communication channel through which we will be seeking feedback
• Channel owner: the person on the Editing Team who will facilitate the given "Feedback channel" and relate the feedback raised to Program Manager and Engineering
• Translator: the person on the Editing Team who will translate the accumulated feedback (i.e. bugs, qualitative feedback) into Phabricator tickets
• Feedback output: where and how the feedback we receive will be incorporated into the Editing Team's workflow

usertesting.com Testing

Absent from the "Audience" column" in the "Testing Plan" above is "usertesting.com" – a feedback channel that had previously been included in our testing plans.

Decision

We have decided not to pursue qualitative feedback through usertesting.com for this feature release.

Rationale

We have decided on the above for the following reasons:

1. We think we will be able to reach the contributors for whom Section Editing has been intended [1] through the All Wikis and Hebrew wiki tests we are running.
2. We think the questions our usertesting.com testing was originally intended to answer [2] will be satisfied through the test we are running with All Wikis and Hebrew wiki, described in Testing Plan table above

References

1. https://www.mediawiki.org/wiki/VisualEditor_on_mobile_report#Personas
2. The description of T209986 represents the intention of our usertesting.com testing: "...to test the discoverability and usability of the feature as well as to receive qualitative feedback on the feature."