Planning doc

https://docs.google.com/document/d/1E_hz3MqU4MUW8w1wlHY0vhnX6e0YiswvgKpM6F7lwtI/edit?usp=sharing

Testing goals

We will now be able to support all notifications and alerts in the iOS app. This update will make it easier for editors who use iOS devices to be notified about new messages and alerts that they have, without having to voluntarily open the app or website to check. As we prepare to release this feature, we want to ensure that the usability of notifications is clear and doesn’t disrupt editors' existing flows. We will also need to ensure that new editors and other logged in users who may not be used to the notifications system from web will be able to successfully use and take action on their notifications.

Notifications can be seen in two different areas: 1) when push notifications are turned on they will appear in the system notification center and lock screen, 2) tapping on a push notification or navigating to the in app notification center will show notification messages in the app itself.

Through testing we will need to ensure that all elements of the feature are understandable and actionable. We will also need to test with both seasoned editors (to ensure that the feature meets the expectations they may have from using notifications on desktop) as well as new users who may have never seen a notification from Wikipedia before.

With these two distinct groups in mind, our aim in testing is to perform an unmoderated usability test of just the in-app features with new users followed by an unmoderated longitudinal study (weeklong diary study, with a webform and follow-up questions) with current editors.

The primary goals of this tests are to gain insights on the following:

Push notifications (Diary study only)

• Do push notifications behave as expected?
• Do push notifications arrive in a timely manner?
• Do push notifications contain enough information to be useful/actionable?
• Is any information superfluous?
• Is any information missing?
• Does the tap through to the app work smoothly and as expected?

Badging (Diary study only)

• Does badging on the app icon work as expected?
• Is app icon badging helpful?
• Is app icon badging clear and easy to understand?
• Does the in-app icon badging work as expected?
• Is the in-app badging helpful?
• Is the in-app badging clear and easy to understand?

First time user experience

• Is it easy and intuitive for participants to opt into or out of push notifications?
• Are the user education and alerts clear and actionable?

Navigation and access to notifications

• Are users able to quickly and easily navigate to their notifications in the app?
• Do in-app notifications (toasts) contain enough information to be useful/actionable?

In-app notification center

• Do individual notification cells provide enough information to be useful/actionable?
• Is anything missing in individual notification cells?
• Is anything superfluous presented in individual notification cells?
• Are participants able to easily distinguish between ‘read’ and ‘unread’ messages?
• Are participants able to easily identify the filters and project inboxes?
• Are participants able to easily customize their notification center using filters and project inboxes?
• Are empty states clear, helpful and actionable?

Taking action on notifications

• Is it easy and intuitive for participants to mark their notifications as ‘read’ or ‘unread’?
• Are swipe actions discoverable and actionable?
• Does tapping on notifications navigate to expected pages?
• Does the transition between the notification center and the notification link destination work as expected?
• Are participants able to easily navigate back to their notification center after taking action on a notification?

Settings and customization

• Is it easy for non-power users to quickly and successfully receive relevant notifications?
• Is it easy for power users to quickly and successfully customize the notification types they see in their in-app notification center?
• Is it easy for all users to quickly and successfully turn on push notifications?
• Is it easy for all users to quickly and successfully customize their push notifications?

General feedback

• Any other information or feedback testers may want to share.

Testing information

Two versions of this test will be run using the same task list with slightly modified opening scenarios.

The first group of testers will be internal testers who will complete the test on their personal devices using a beta version of the app and a Google form. The second group of testers will be recruited from UserTesting.com and will not be required to have any former editing history or knowledge.

Two different groups of testers will be used for this prototype because feedback is needed from users who have experience with communication and notifications on Wikipedia as well as first time users.

Internal testers

At least 5 users will be recruited via internal email from the WMF.

These testers will need to be able to run the latest beta version of the app on their device. Participants will be provided with a link to the testing protocol as well as three Google forms, where they will be asked to complete surveys throughout the study.

The test will be an unmoderated two-week long diary study (longitudinal study). Participants will be asked to use the beta version of the app, while logged into their volunteer account with push notifications turned on (with the option to modify their preferences throughout the duration of the study).

Participants will be asked to use the app for two weeks and answer a brief XX question survey at least 5 times throughout the duration of the study. Additionally, participants will be asked to fill out an intro and an exit survey.

Users will perform the test remotely.

There are no screener questions for this group of testers.

First time editors

The test with first time editors will utilize UserTesting.com to recruit users, record sessions and ‘facilitate’ the test.

The test will be an unmoderated, task based test. Users will be given a set of tasks to complete in a 15-20 minute time frame, followed by a set of 4 written questions. Users will perform the test remotely.