I'm running the test on usertesting.com. Here are the changes I've made so far:

1. url will contain the templates on the top of the page (so we can test discoverability)
2. removed the word "topic" from tasks directing users to find the button to add a new discussion (so we aren't leading the user)

I think that we should still:

• add a task asking the user to add their signature and format the new discussion
• add a task asking the user to cancel out their post'
• add a written question asking users to explain how they anticipate finding out when someone comments on/in the discussion they started
• copy this whole test and use it on the USER talk page

@ppelberg does that sound like the right set of next steps to you?

In T247485#5973676, @iamjessklein wrote:

@ppelberg does that sound like the right set of next steps to you?

@iamjessklein, +1 to everything you shared in T247485#5973676 with the addition of the following question:

• What do you think we stand to learn in asking test participants to, "...add their signature..."?
  • I ask the above considering none of the previous test participants attempted to sign their comments or remarked about their usernames not appearing next to their comment. This leads me to wonder whether this is sufficient evidence for us to assume: people do not expect to have to do anything manual to ensure their comment is properly attributed.

Update: 17-March
Documenting notes from the meeting @iamjessklein and I just had...

In this next iteration of the user test, we will make the following changes. Some of the changes listed below come from T247485#5973676 others came about during our conversation just now.

Note: we decided not to ask people to sign the discussions they started. We decided this because we think previous test findings make it sufficiently clear people do not expect to have to do anything manually for the content they post to be attributed to them.

Article talk page test

• For the article talk page test, ADD the templates on the top of the page (so we can test discoverability)
• REMOVE the word "topic" from tasks directing users to find the button to add a new discussion (so we aren't leading the user)
• ADD a task asking the user to format (e.g. bold, italicize) the content of the new discussions "body" text field
• ADD a task asking the user to cancel out their post
• ADD a task asking users to make sure they are notified when someone comments on/in the discussion they started
• CHANGE the article talk page test framing to something like: Imagine you are wanting to ask another editor how to add a reference to the article you are editing.

User talk page test

• REMOVE the word "topic" from tasks directing users to find the button to add a new discussion (so we aren't leading the user)
• ADD a task asking the user to format (e.g. bold, italicize) the content of the new discussions "body" text field
• ADD a task asking the user to cancel out their post
• ADD a task asking users to make sure they are notified when someone comments on/in the discussion they started
• COPY this whole test and use it on the USER talk page
• CHANGE the user talk page test framing to something like: Imagine you are wanting to thank an editor for helping you figure out how to add a reference to an article you were editing.
• ADD a task asking users to @-mention a specific person

Update April 9 - I ran another round of tests with the fixes above for the Article talk page test. While the test ran, I noticed that the templates we copied included a shortcut link to making a new conversation in them (which is found on many but not a majority of Talk pages). So I am doing one more round of testing on the article page and still need to do the user page. I will update this ticket with the results.

I still need to update the ticket with findings and I need to run a test on a signed in user talk page (not article page).

I've finished three rounds of testing on the article talk page for the new discussion flow.

In order to run a test on the signed in user talk page, I believe that I need another account to be made so that I can make up a user talk page with one and then use the other to have the usertester write on the initial talk page with. @Esanders can you help me to do this?

Findings from usability test on article talk page:

We ran a test on usertesting.com on April 16, 2020 - The test recruited 5 random, technically - advanced web users. Participants were directed to an article page set up on the prototype server and instructed to sign in. The detailed findings can be found on the limited access test log.

• 5 tests were conducted
• Participation: 2 male ; 3 female
• All participants had familiarity with Wikipedia and some participants had previously edited an article.

Identifying where to add a new discussion
✅ 5/5 desktop web participants successfully identified where to add a new discussion
✏️Every single test participant struggled to find the new discussion button.

Adding a new discussion
✅ 5/5 desktop web participants successfully added a new discussion

Locating discussion in the Talk page
✅ 4/5 desktop web participants successfully located the discussion

Insightful quotes:
"It was not intuitive - I thought I needed to go to the bottom of the page." CT3-D5
“It was challenging to understand that add topic meant starting a discussion.. these are not the same words and it could be mistaken as a fully new topic for example, a subsection of cats that would create an entirely new page, not just a discussion within the current page." CT3-D3
“"everything was painful, I got lost a couple of times, the layout is very difficult to digest with lots of text and too much colours that don´t mean much unless you are familiar with the site" CT3-D2
"One needs to adopt a coding mentality to understand the page. The problem is that we are living is a very visual society." CT3-D2
"I was not easily able to find it, although it was right in front of me." CT3-D1

Recommendations:
The test confirmed that the new discussion buttons are not discoverable in the current normal Talk page set up (yellow notification templates, table of contents, discussion being the page content). Although all of the testers were able to complete all of the tasks, they did it with a high amount of difficultly primarily due to the lack of discoverability of the "New Topic" tab. As a result, I recommend that we:

1. Improve discoverability
2. Use common taxonomies whenever possible. Testers were thrown off by the term “New Topic” and “Edit Source” “Edit Summary”
3. Improve discoverability after posting a new discussion.
4. Improve workflow for writing an initial comment on a thread. Many participants only left a topic and were not directed how to write the initial comment.
5. Differentiate between the design of the article and talk pages. This was pointed out in the Talk page consultation already, but definitely validated through testing.
6. Optimize the visual editing reply in the UI hierarchy so that junior contributors aren’t turned off from posting due to unfamiliar Wikitext characters.

Next steps: 21-April
Notes from today's conversation with @iamjessklein...

• 1. @iamjessklein: publish/run last control test of the user talk page experience. See test and test page
• 2. @iamjessklein: publish control test findings to this ticket
• 3. @ppelberg: publish usability test findings to mw.org project page

We ran a test on usertesting.com on April 24, 2020 - The test recruited 5 random, technically - advanced web users. Participants were directed to an article page set up on the prototype server and instructed to sign in. The detailed findings can be found on the limited access test log.

• 5 tests were conducted
• Participation: 5 male
• All participants had familiarity with Wikipedia and some participants had previously edited an article.

Identifying where to add a new discussion
✅ 3/5 desktop web participants successfully identified where to add a new discussion

Adding a new discussion
✅2/5 desktop web participants successfully added a new discussion
1/5 didn’t add a subject so their discussion was posted under another discussion topic.

Locating discussion in the Talk page
✅ 3/5 desktop web participants successfully located the discussion
2/5 Was able to locate their post, although it was not in a new discussion section.

Insightful quotes:

Look and feel:

“I think that the "user and discussion" pages were written in a larger font because it was hard to tell. what pages i was on.” CT4-D1
“im beginning to see this page is just a transcript for previous things. Maybe Alice is helping people so she is putting it on there for people to see.”- CT4-D3
on seeing summary message: "it's asking if ive changed something but i didn't change anything so I'm typing 'nothing changed'" - CT4 -D2
“It's weird because it's the exact same format as any article on Wikipedia.” - CT4-D4
“It seems odd that it looks likes there would be a message but there is nothing that's indicating that. it looks very official, not like a discussion forum." // - CT4-D5

Usability
"omg this is it! i don't know how i did it, i just did it." - CT4-D2
"It's really bugging me that it asks me to publish the question. I understand that I'm sending a direct message to the person, but It seems to me that I'm automatically editing the page. I feel like I'm doing something wrong." - CT4-D2
"if I known my subject line would be displayed so openly here, i would have written something better. I thought this was a personal message." CT4-D1

Recommendations:

The test confirmed the findings of the previous tests. The new topic button is not discoverable, the language used to describe buttons is confusing, many testers expected to see the new message post to the top of the thread (not the bottom), the page does not look like a discussion forum. The only additional bit of insight that this test provided was that testers were surprised to see their message posted publicly.

"if I known my subject line would be displayed so openly here, i would have written something better. I thought this was a personal message." CT4-D1

Several testers expected their messages were going to be sent directly to "Alice" . This was further evidenced in the observation by one tester who prior to completing the task thought that the page was a transcript posted by the talk page user, "Alice".

“im beginning to see this page is just a transcript for previous things. Maybe Alice is helping people so she is putting it on there for people to see.”- CT4-D3

I recommend that we look into possible ways to more effectively communicate to the user that their posts will be public through the UI.

One additional thing that felt off was what happens when a user doesn't add a title to their discussion but posts it. If a user doesn't provide a subject on their new topic post, the post goes under the previous topic. I find that to be unexpected and something that we should not reproduce when we do our iteration.

Here are the findings about expectations around notifications: All of the testers expected to receive an email and that their test account inbox would also get a notification. Here are some good quotes about that:

CT5 -D2:

I would hope that somehow my email account is linked to this so then I can get an email saying something like “Alice has responded to your question” and thenI can go and click on that link from my email that will open the Wikipedia link and then I can reply to Alice there.

CT5-D3:

I assume it will have something to do with me hitting “watch this page” when I was posting and editing this question. I didn’t uncheck the box that says watch this page so I assume I would be logging back in and watching what happened on this page and getting a notification in my feed, you know, my profile to see when someone responded.

CT5-D4:

A notification of some type would be good because you don’t want to check the same page over and over and over and you never know if it’s ever going to actually update or if they are actually going to respond. It’s kind of a waste of time if you have to go through and check it over and over again and they haven’t responded.

CT5-D5:

I would expect to see it as a notification sent to my Test1 inbox” (points to the person icon in top navigation

In T247485#6089355, @iamjessklein wrote:

Here are the findings about expectations around notifications: All of the testers expected to receive an email and that their test account inbox would also get a notification. Here are some good quotes about that:

This is helpful for us to know – thank you for re-running this, @iamjessklein.

Besides CT5-D3, did any other tester say whether they thought they would need to take an explicit action themselves (e.g. checking "Watch this page") to be notified when someone responded to them?

CT5-D3:

I assume it will have something to do with me hitting “watch this page” when I was posting and editing this question. I didn’t uncheck the box that says watch this page so I assume I would be logging back in and watching what happened on this page and getting a notification in my feed, you know, my profile to see when someone responded.

No and just a bit anecdotally, I was surprised to learn that anyone understood what Watchlist meant.

In T247485#6090682, @iamjessklein wrote:

No and just a bit anecdotally, I was surprised to learn that anyone understood what Watchlist meant.

Got it.