Page MenuHomePhabricator

Topic Subscriptions: Run manual topic subscription usability test with Junior Contributors
Closed, ResolvedPublic

Description

Objective

Run the test as described in the protocol in T273912 so that we can:

  1. Gauge the discoverability of notifications
  2. Identify improvements to the usability of topic subscriptions:
  3. Gauge user reactions to the notifications and expectations of how they will receive notifications

User stories

As a junior contributor, I want to subscribe to topics on wikipedia, so that I can receive notifications when people respond to the conversations that I am involved in.

As a junior contributor, I want to follow a call to action from a notification, so that I can easily engage in on-wiki conversations.

As a junior contributor, I want to unsubscribe to topics on wikipedia, so that I can stop receiving unwanted notifications.

Findings

Test Log
Presentation Deck

Discoverability:

✅ 5 out of 5 successfully identified notifications in the toolbar
"I knew because it was highlighted in yellow so I clicked it and it went to the page." [UT-A]
✅ 5 out of 5 successfully located the comment they are being notified about on the talk page where it appears
"It was quite easy to locate" [UT-B]

Usability:

✅ 5 out of 5 successfully subscribed to notifications
✅ 4 out of 5 successfully unsubscribed to notification via Notification Dropdown
✅ 5 out of 5 successfully unsubscribed to notification via inline button
"It's quite intuitive" [UT- D]
"Overall this was pretty smooth." [UT-F]

Miscellaneous Observations:

✏️ 1 tester was completely unaware that they unsubscribed by double clicking the inline button
✏️ 1 tester attempted to unsubscribe via Preferences -> Notifications
✏️ 1 tester who successfully unsubscribed from the dropdown, followed the notification link to preferences and got completely confused
✏️ No testers commented on the lack of subscribe buttons on the USER TALK page.
✏️ Several testers struggled with having two separate entry points for Notifications on the toolbar
✏️ One tester was testing in a low bandwidth environment and was unsure if the test was loading or if it crashed.
✏️ One tester was testing in a low bandwidth environment and was unsure if the test was loading or if it crashed.
✏️ Several testers were confused by the copy on the Reply button
✏️ Although testers completed the unsubscribe from notification dropdown task, I am still unsure of the discoverability of this button without prompting

💡 Tester Idea: Styling threaded comments: “ it's a little interesting how there's indenting to show the threads. Maybe if there was a line or something it might break up the threads better." [UT-F]
💡 Tester Idea: Subscriptions Portal: "You know what I'd suggest: If there could be a place where I essentially know where my subscriptions are what I'm subscribed to and I can easily subscribe and unsubscribe from there." [UT-A]

Recommendations:

Key: P0=Must fix ------ P3=Minor improvement, NA = change that is tangential to the release of the main feature

[PO] In future tests consider testing for true discoverability of unsubscribe feature
[NA] Create one entry point for notifications T142981
[NA] Consider creating a Subscriptions management feature T273342
[NA] Consider styling threaded comments to improve legibility T282269
[NA] Consider styling subscribe/unsubscribe as buttons from OOUI T282261

Requirements

  • testing environment created on wiki via patch demo
  • usertesting.com protocol is set up for 5 tests
  • 5 tests are run in English
  • Findings are documented on the ticket (including notable quotes)
  • If applicable, tickets are filed for issues that surface

Related Objects

Event Timeline

I am adding the pages and messages that are required for the test into the patch - T280199
After this is done, I will test the test again so that we can close out that ticket and just focus on getting everything up and running in this ticket.

I successfully ran one test of the test on usertesting.com. The reason this isn't the actual test is that the subscribe/unsubscribe link placement on the prototype needs to be updated to the latest design.

I am logging the test results here and will put a high level synthesis when all of the tests are run on this ticket.

In a planning meeting today, the team decided to go ahead with testing the prototype AS IS because we will be going with a similar placement, but a more minimal style for the first iteration.

I've updated the ticket description with the findings. I am going to attempt to recreate the bug witnessed in testing to see if it is in fact a bug, and if it is, I will file a ticket. I need to do that and then summarize the recommendations.

🐞 Bug: One tester unsubscribed first from in context subscribe button, then went back to his notification dropdown and unsubscribed again, which appears to have made him "Re-subscribe" [UT-E]

I've tried a few times now and can't re-create the bug so I've removed it from the findings.

Recommendations:
Key: P0=Must fix ------ P3=Minor improvement, NA = change that is tangential to the release of the main feature

[PO] In future tests consider testing for true discoverability of unsubscribe feature
[NA] Create one entry point for notifications T142981
[NA] Consider creating a Subscriptions management feature
[NA] Consider styling threaded comments to improve legibility
[NA] Consider styling subscribe/unsubscribe as buttons from OOUI

@ppelberg can you tell me if there are tickets for:

[NA] Consider creating a Subscriptions management feature
[NA] Consider styling threaded comments to improve legibility
[NA] Consider styling subscribe/unsubscribe as buttons from OOUI

If not, I'll file some

@ppelberg can you tell me if there are tickets for:

[NA] Consider creating a Subscriptions management feature

Yep, that ticket is T273342

[NA] Consider styling threaded comments to improve legibility

Nope; can you please file a ticket for this and once done, add it as a sub-task of T249579?

[NA] Consider styling subscribe/unsubscribe as buttons from OOUI

Can you please comment the issue/opportunity this suggestion is a response to on T282261?

Note: T282261 is the ticket where we'll design the "topic container" experience and by extension, any revisions to the Subscribe / Unsubscribe affordance.

I've updated the task description with the associated tasks (none of which are release blocking).

Before closing out this ticket, I wanted to gut check regarding:

[PO] In future tests consider testing for true discoverability of unsubscribe feature

I am wondering 2 things:

  1. Should we re-run the test?
  2. If so, should we shorten the protocol and add in the latest button design?

My instinct is telling me that this is not crucial to check now because a lot of the discoverability issues will be addressed in T249579 but it would be good to confirm that with @ppelberg

I've updated the task description with the associated tasks (none of which are release blocking).

Before closing out this ticket, I wanted to gut check regarding:

[PO] In future tests consider testing for true discoverability of unsubscribe feature

I am wondering 2 things:

  1. Should we re-run the test?
  2. If so, should we shorten the protocol and add in the latest button design?

@iamjessklein: can you describe in a bit more detail what "testing for true discoverability of unsubscribe feature" means in this context?

Asked another way: how would testing for "true discoverability" differ from what we tested for in the tests we've run to-date?

@ppelberg in the protocol for this test we asked testers to find the unsubscribe feature once they we had already instructed them to go into the Notice dropdown. Because we gave them these instructions they were in essence just hunting for a button within that container, rather than discovering it on their own. To test for this we would probably need to tweak the protocol to be more generic.

@ppelberg in the protocol for this test we asked testers to find the unsubscribe feature once they we had already instructed them to go into the Notice dropdown. Because we gave them these instructions they were in essence just hunting for a button within that container, rather than discovering it on their own. To test for this we would probably need to tweak the protocol to be more generic.

Per the conversation Jess and I had yesterday, we will not be re-running the usability test to evaluate whether Junior Contributors are able to figure out how to elect to stop receiving notifications about new comments from a particular conversation from within Echo.

Reasons:

  1. We are assuming Junior Contributors will intuitively think/know how to take the meta-action of electing to stop receiving notifications about new comments from a particular conversation from within Echo considering this is a design pattern that is used across many of the service we expect Junior Contributors to have experience with (Facebook, iOS).
  2. Even if we discovered that Junior Contributors did NOT intuitively think/know how to take a meta-action on an individual notification from within Echo, we would not have the capacity to prioritize resolving this would-be issue as part of this phase of the Talk Pages Project.

@iamjessklein before I close this, can you please file and link to the ticket for the issue you noticed during usability testing about people accessing the prototype on a low-bandwidth connection being uncertain what the Reply Tool's current loading experience is "saying"?

Thanks for the ping on that @ppelberg - I created T282764

Excellent. Thank you, Jess. Now resolving this ticket.