Testing script

https://docs.google.com/document/d/1E_hz3MqU4MUW8w1wlHY0vhnX6e0YiswvgKpM6F7lwtI/edit?usp=sharing

Testing goals

We will now be able to support all notifications and alerts in the iOS app. This update will make it easier for editors who use iOS devices to be notified about new messages and alerts that they have, without having to voluntarily open the app or website to check. As we prepare to release this feature, we want to ensure that the usability of notifications is clear and doesn’t disrupt editors' existing flows. We will also need to ensure that new editors and other logged in users who may not be used to the notifications system from web will be able to successfully use and take action on their notifications.

Notifications can be seen in two different areas:

1. when push notifications are turned on they will appear in the system notification center and lock screen
2. tapping on a push notification or navigating to the in app notification center will show notification messages in the app itself.

For this usability test participants will test the notification center with usertesting.com, where it is only possible to test notifications by using the in app notification center. Hence the push notifications are not tested in this round of usability testing.

Through testing, we will need to ensure that most elements of the feature are understandable and actionable. We will test with new users who may have never seen a notification from Wikipedia before.

Our aim in testing is to perform an unmoderated usability test of just the in-app features with new users.

The primary goals of this test is to gain insights on the following:

First time user experience

• Is it easy and intuitive for participants to opt into or out of push notifications?
• Are the user education and alerts clear and actionable?

In-app notification center

• Do individual notification cells provide enough information to be useful/actionable?
• Is anything missing in individual notification cells?
• Is anything superfluous presented in individual notification cells?
• Are participants able to easily distinguish between ‘read’ and ‘unread’ messages?
• Are participants able to easily identify the filters and project inboxes?
• Are participants able to easily customize their notification center using filters and project inboxes?
• Are empty states clear, helpful, and actionable?

Taking action on notifications

• Is it easy and intuitive for participants to mark their notifications as ‘read’ or ‘unread’?
• Are swipe actions discoverable and actionable?
• Does tapping on notifications navigate to expected pages?
• Does the transition between the notification center and the notification link destination work as expected?
• Are participants able to easily navigate back to their notification center after taking action on a notification?

Findings

The testers will be recruited from UserTesting.com and will not be required to have any former editing history or knowledge (the information was taken from this google sheet doc).

✅ 100% of testers quickly found the notification center, with 83% of the testers finding it on their first try.
✅ All testers could identify from who the notification was from.
✅ 83% of the testers tapped on the notification and were able to open the content of the notification correctly.
✅ All testers found it easy and intuitive to navigate through the notification center.
✅ All testers were able to mark a notification as ‘unread’ and 83% thought it was very easy and intuitive.
✅ All testers found it very easy to mark 2 notifications as ‘unread’.
✅ All testers were able to quickly find the ‘filters’, use the options under the ‘read status’ and turn on/off notification types.

😑 33% of the testers were confused by the content of the notifications.
😑 33% of the testers understood that tapping on either the username or the content of the notification will direct you to different screens (either the user page or the messages content page)
😑 50% of testers found information about the editor (from whom a message was received) by going through the content of the message and then tapping on the editor’s name rather than tapping on the username in the notification.
😑 33% of testers thought of swiping to mark a notification as ‘unread’.
😑 50% of users recommended having a way to select/deselect all notification types at once on the ‘Filters’ page.

❌ 33% of the testers think that the notification cells are crowded with information.
❌ A tester thought the teaser message wasn’t helpful.
❌ 66% of the testers did not discover the ‘Mark all as read’ option under edit and instead selected each notification and chose ‘Mark’ followed by ‘Mark as read’.
❌ A tester found it extremely strange that by 'unselecting' all filter types except for one they actually turned on the filters according to the information stated in the empty state & on the bottom of the notification center (it read that 15 filters on, but they actually turned them off. That's not good.
❌ A tester thought that being able to tap on the username in the notification can be confusing, especially if you tap there accidentally and actually wanted to read the content of a certain message.
❌ 33% of the users thought that the icons were hard to understand.

Suggestions

• Updates to notifications cell design ​​T296787
• Updates to notifications filters T296788