Might it be useful to test on a Project-namespace discussion page as well, such as w:en:Wikipedia:Help desk?

In T243251#5822457, @Jc86035 wrote:

Might it be useful to test on a Project-namespace discussion page as well, such as w:en:Wikipedia:Help desk?

This is a good idea, @Jc86035 – thank you for raising this. We will consider including Project-namespace discussion pages in this test.

Thanks for the input @Jc86035
@ppelberg and @Whatamidoing-WMF Please provide your feedback on the draft protocol.

In T243251#5891000, @iamjessklein wrote:

@ppelberg and @Whatamidoing-WMF Please provide your feedback on the draft protocol.

You got it. Comments are in the doc.

Thanks @ppelberg - I've updated the testing protocol so that we will be testing this on usertesting.com mobile and desktop web.
I have the test written and ready to go on usertesting.com and have one question. I'm noticing on the prototype server that the infobox templates aren't visible. Is this something that we can fix before putting the test up? The reason that I am asking for this is so that we can recreate the actual experience as much as possible. cc/ @Esanders @matmarex

English Wikipedia:

Prototype Server:

Update: I ran the usability tests without the templates at the top of the page. I did five tests on mobile and five tests on desktop (both web). The next step here is reviewing and synthesizing the tests.

Findings:

We ran a test on usertesting.com on February 21, 2019 - The test recruited 10 random, technically - advanced web users. Participants were directed to an article page set up on the prototype server . The detailed findings can be found on limited access test log.

• 10 tests were conducted
• Participation: Desktop: 1 male ; 4 female// Mobile : 2 male/ 3 female
• 5 participants were desktop web users
• 5 participants were mobile web users
• All participants had familiarity with Wikipedia and some participants had previously edited an article.

Identifying where to add a new discussion
✅4/5 mobile web participants successfully identified the button to add a new discussion
✅5/5 desktop web participants successfully identified the tab to add a new discussion

Adding a new discussion
✅2/5 mobile web participants successfully added a new discussion to the Talk page
🚫1/5 mobile web participants edited the talk page but did not add a new discussion
🚫1/5 mobile web participants unsuccessfully completed task
🚫1/5 mobile web participants gave up on task
✅3/5 desktop web participants successfully added a new discussion to the Talk page
🚫1/5 desktop web participants unsuccessfully completed task
✏️ None of those who posted were able to sign their comments correctly.
✏️ The participants who unsuccessfully completed the task edited the page and added a comment but not a new discussion
✏️ Many participants were uncertain that their actions would lead them to a successful edit.
“yes it was very easy to use and straightforward. i found what i needed immediate and was able to add a discussion”- CT-D5
“i really don't know if i'm doing the right thing" - CT-M5
“It was very confusing how to edit this page. I’m not sure if i was taken to the wrong page or if i needed to know something extra to successfully edit this page." - CT-M1

Locating discussion in the Talk page
✅1/5 mobile web participants successfully located their discussion in the Talk page
🚫3/5 mobile web participants unsuccessfully completed task
🚫1/5 mobile web participants gave up prior to completing task
✅4/5 desktop web participants successfully located their discussion in the Talk page
🚫1/5 desktop web participants unsuccessfully completed task
✏️ For many participants, hunting and not finding their post signaled to them that they didn’t actually complete the task as they had intended.
"i don't see where I published the topic or if I am doing it wrong."" CT-M3
“I like that as soon as I published it went straight to the section and was easy to see.”- CT-D3
"Absolutely, no trouble whatsoever. All the tools you need to add, edit information are right there for you to see. No searching or looking all over the page to find it." CT-D2

Miscellaneous Notes
✏️Many participants did not realize that they were viewing a Talk page and provided comments as if they were looking at an Article page
✏️Several participants were put off by having to provide comments in Wikitext. One participant said that they expected to see more of a “forum.”
✏️This test validated some general ux changes that are already being integrated into features. This includes: automating summary, automating signatures, adding more visual editing.
✏️Several participants got lost in the course of their test and had to find the Talk page again through using the search bar.
“it would be nice if when you click "show preview" it looks exactly what it would look like on the page.” CT-D3
"I would say the only thing I was confused about what the summary box when you click "Edit Source" as I did not seem clear what was the purpose of it, as writing in the text box seemed to introduce the text anyways".
"i know that i can trust this tool." CT-M2
"i find it a little difficult to navigate. i wasn't sure if i needed to know how to code .... obviously i don't because I messed up." CT-M1

Recommendations:

1. To Test:

• Test on different namespaces.
• Prepare a test with “top” templates - I suspect it was easy for desktop web participants to identify the navigation and buttons with such ease because this was removed from the page (and therefore this isn’t a typical user scenario)
• Test with logged in participants.
• Test separately for Advanced Mobile Contribution interfaces

2. To Workflow:

• Improve discoverability of the “Add a new discussion” button. Although desktop web users were able to successfully identify the button, many did not truly understand what they were doing until it was published.
• Improve workflow for writing an initial comment on a thread. Many participants only left a topic and were not directed how to write the initial comment.
• Improve discoverability after posting new discussion.
• Differentiate between the design of the article and talk pages. This was pointed out in the Talk page consultation already, but definitely validated through testing.
• Consider having the visual editing reply optimized in the UI hierarchy so that junior contributors aren’t turned off from posting due to unfamiliar characters.

Great. @Esanders and I just made new test page - that has working templates!!! I'll update the test progress on this ticket.

In T243251#5963985, @iamjessklein wrote:

Great. @Esanders and I just made new test page - that has working templates!!! I'll update the test progress on this ticket.

Oh, this looks great. Awesome, Ed and Jess.