TASK DESCRIPTION UPDATE

• ADDED a reminder for us to use this usability test as an opportunity to learn how – if at all – topic subscription affordances not appearing on peoples' own user talk pages affects their understanding of the feature and their own user talk page.

I am going to start drafting the usability test for junior contributors.
For reference, I am going to make sure that it incorporates the thinking on:

T274215 Create response rate measurement plan
T275232 Learn what Senior Contributors think about topic subscription initial prototype

I've updated the description of this ticket.

I've started to write the test protocol and script.
The protocol is written on a high level for junior, senior and internal testers.
The only script that is written at this point is the JUNIOR test, as I made some assumptions there that I want to gut check on before moving forward with the other scripts.

Please review the protocol with the following questions in mind:

@Whatamidoing-WMF

• How will we tests this with junior non-english editors?
• How should we adapt this script for Senior Contributors? (This will happen in https://phabricator.wikimedia.org/T275232)

@Esanders @matmarex

• Does the way that I set up the test make sense in terms of what needs to be automated? (ultimately via https://phabricator.wikimedia.org/T280199)
• Is what I've written technically feasible? If not, please propose tweaks.

(after I get this feedback I will update any requirements in T279149 should there be changes).

@DLynch @MNeisler

• Does this test written in such a way that we are getting the kind of (qualitative feedback) that we are not getting in T274215?

@ppelberg

• All of the questions for the other people marked on this ticket ^^
• How should we adapt the script for internal testers? I believe this will be VERY lightweight but I'd like to be intentional about it.

@Ryasmeen

• How will you conduct QA on this via when it's ready to ship? (This is more of an FYI but I am experimenting by pinging you here to give you an early heads up)

In T273912#7009947, @iamjessklein wrote:

@Esanders @matmarex

• Does the way that I set up the test make sense in terms of what needs to be automated? (ultimately via T279149)
• Is what I've written technically feasible? If not, please propose tweaks.

(after I get this feedback I will update any requirements in T279149 should there be changes).

I assume you mean task T280199?

It makes sense to me and looks feasible, I asked for clarification in the doc on one thing though.

Yes thanks, I'll update that card @matmarex

In T273912#7009947, @iamjessklein wrote:

Please review the protocol with the following questions in mind:

@iamjessklein, before reviewing the script, a couple of questions for you about the task description's ===Goals of this usability test section. These questions are below.

Goals of this usability test

1. Gauge the discoverability of notifications

• Are you referring to the Subscribe/Unsubscribe affordance? The "badge" that appears next to the notices icon when someone has an unread new comment notification ? Both? Something else entirely? Fwiw: I think we should be evaluating both aspects.

2. Identify improvements to the usability of topic subscriptions
   • Do testers understand the copy within the toasts and notifications?
   • Are testers able to successfully follow the steps to post a question?

• What does "post a question" mean in this context? To post a comment to the talk page using the Reply Tool? If so, can you share a bit more about you think is importanat for us to learn about this particular workflow in the context of topic subscriptions?

• Is there anything testers like to be informed about but are missing?

Beyond the questions I posed above, where – if at all – do you think these questions fit within this usability test?

• "Do people intuitively understand how to elect to receive notifications about new comments posted in conversations they are interested in?"
• "Do people intuituively understand what the notifications they receive via eacho mean?"
• "Do people know how to locate the comment they are being notified about on the talk page where it appears?"
• "Do people know what to do to strop receiving notifications about a particular conversation?"
• "And more broadly, how do they expect to be alerted when someone postes a new comment in a conversation they've subscribed to?"

@ppelberg I believe that you are looking at an older version of the protocol, so I will answer based on the current iteration:

1. Gauge the discoverability of notifications - this refers to the unread comment

2. Identify improvements to the usability of topic subscriptions: - this refers to the "badge" (in your words)

Is it obvious to testers how to subscribe and unsubscribe and do the actions match their expectations?

This is incorporated

"Do people intuitively understand how to elect to receive notifications about new comments posted in conversations they are interested in?"

This is incorporated

"Do people intuitively understand what the notifications they receive via eacho mean?"

This is incorporated

"Do people know how to locate the comment they are being notified about on the talk page where it appears?">

This is incorporated
This is part of the usability

"Do people know what to do to stop receiving notifications about a particular conversation?"

This is incorporated
I actually had changed a step in tasks to make this the last task.

"And more broadly, how do they expect to be alerted when someone posts a new comment in a conversation they've subscribed to?"

This is incorporated

@iamjessklein: the answers you shared in T273912#7015076 are helpful.

I've reviewed the Usability test of Notifications doc and added the comments and questions the current draft brought to mind in-line.

Does this test written in such a way that we are getting the kind of (qualitative feedback) that we are not getting in T274215?

@iamjessklein

I took a quick scan of the protocol. It looks good to me and I think it covers a lot of the feedback that we can't get quantitatively. I just had one comment on the doc re getting feedback on the notification email itself, which I added to the doc.

Also, thanks for sharing this with me! It was really useful to see the qualitative information we're seeking to obtain and how that might help fill in some of the gaps or add context to the quantitive metrics.

Thanks @ppelberg @MNeisler :

I've answered all the questions in the protocol that you left and so next the next step would be to review them again so that we can finish up with writing the script. I think only the last one (about writing more steps... and unsubscribing) is blocking at this point.

I will set up a separate meeting with @Whatamidoing-WMF and @ppelberg about testing in different languages.

In T273912#7030133, @iamjessklein wrote:

I've answered all the questions in the protocol that you left and so next the next step would be to review them again so that we can finish up with writing the script.

Great, @iamjessklein. I've gone in and responded to the most recent round of comments you've left. See: doc.

I will set up a separate meeting with @Whatamidoing-WMF and @ppelberg about testing in different languages.

@iamjessklein: I've been operating with the assumption that we are planning to test with non-English speaking Junior Contributors through usertesting.com. Is this what you are thinking as well?

Context: I ask the clarifying question above considering we've already started seeking feedback from non-English speaking Senior Contributors, with the help of @Dyolf77_WMF, here.

I've updated the usability test per our recent conversations @ppelberg
I would like to close this ticket as protocol is now done and start working on T281438

I've been operating with the assumption that we are planning to test with non-English speaking Junior Contributors through usertesting.com. Is this what you are thinking as well?

yes