Page MenuHomePhabricator

Get user feedback for iOS Watchlist
Closed, ResolvedPublic

Description

Table of contents
  1. Background
  2. Target audiences
  3. Qualitative target audiences
  4. Tasks
  5. Method
  6. Deadline
  7. References

1. Background

We are developing the Watchlist feature for the iOS app. The Watchlist is crucial for users, particularly frequent editors, who wish to monitor changes to specific Wikipedia pages. This feature allows users to keep track of changes, review them for accuracy, and, if necessary, make corrections or additions. Given the significance of the Watchlist feature, it must be user-friendly, intuitive, and effective. As such, we plan to conduct user testing of this feature on the Wikipedia iOS app, aiming to understand its current usage, identify potential areas of improvement, and gather insights to inform future development.

2. Target audiences

2.1. Quantitative target audiences

Our primary target is frequent app-editors in Northern and Western Europe, with a secondary focus on new and infrequent app-editors in the U.S. and Canada who edit on other platforms.

2.2. Qualitative target audiences

Our demographic specifications include:

  • 15% of U.S. testers should be Spanish speakers.
  • 10% of Canadian testers should be fluent in either Chinese or Punjabi.
  • Testers should represent a diversity of age groups.
  • At least 50% of Northern and Western European testers should be women.
  • There should be a representation of varying visual abilities.

2.3. How do we classify our editors?

  • Newcomer <50 edits
  • Medium 50-1000 edits
  • Expert >1000 edits

3. Qualitative target audiences

3.1. Frequent app editors (Gdoc)

These users have substantial experience with the app, so they can provide crucial insights into the feature's utility and areas for enhancement.

Location
  • Northern and Western Europe
Requirements
  • Minimum 5 users (stretch of 10)
  • Diverse in age
  • At least 50% are women
  • Strive to have someone with visual impairments
Languages
  • At least one German speaker
Questions

General:

  • Are our target users able to find the Watchlist feature? Is it discoverable?
  • What filters would they find useful?
  • How efficiently can users perform tasks with the Watchlist feature, such as adding an article or checking updates? How often would they use watchlist expiry? How easy is it for someone to update their watchlist expiry? Is it an intuitive process?
  • When users are interacting with the flow, where do they get stuck? What tasks do they get stuck on? What items or functionality is most difficult?
  • Is the Watchlist feature consistent with the overall look and feel of the Wikipedia iOS app, and does it have continuity with Watchlist features across other platforms, such as the Mobile Web and Android?

Design:

  • Can users easily find the Watchlist feature within the app, and do they understand the iconography and wording used to represent it? How are icons perceived across different languages?

3.2. New app users (Gdoc)

These (infrequent) users may have lesser familiarity with the app, but they have some understanding of Wikipedia and the Watchlist feature. The questions will focus on their comprehension and intuitive use.

Location
  • North America (Canada & US)
Requirements
  • Minimum 5 users (stretch of 10)
  • Must use an iOS device
  • Half should not have used the app before but have edited or used Watchlist
  • Half should have used the Wikipedia App but have never edited in the app but are aware you can edit on Wikipedia
  • Must be familiar with Wikipedia
  • Diverse in age
  • At least 50% are women
Languages
  • Spanish
  • Punjabi
  • Chinese
  • English
Questions

General:

  • Are our target users able to find the Watchlist feature? Is it discoverable?
  • How are they planning to use the Watchlist? Do they understand the intended functionality?
  • Do users understand the expected functionality of the Watchlist feature from the Figma files?
  • How effective is the proposed onboarding process in helping new users understand how to use the features and the goal of Watchlist?
  • Does adding the Watchlist feature change users' overall perception of the Wikipedia app?
  • How efficiently can users perform tasks with the Watchlist feature, such as adding an article or checking updates? How often would they use watchlist expiry? How easy is it for someone to update their watchlist expiry? Is it an intuitive process?
  • When users are interacting with the flow, where do they get stuck? What tasks do they get stuck on? What items or functionality is most difficult?

Design:

  • How intuitive is the process of adding an item to the Watchlist, and can users easily tell when an article is already on their Watchlist?

3.3. Guardrail group (Gdoc)

These users are communities that might experience specific difficulties when utilizing the Watchlist feature. The Watchlist feature might significantly affect certain groups, such as people in formerly colonized countries editing large language wikis. These communities also fall under emerging communities. They might be discouraged from editing or using the iOS Watchlist if their changes (based on reputable but less known sources) are frequently reverted. To mitigate this, we could increase awareness among established editors about the challenges faced by contributors from these countries and encourage these more established groups to be supportive and understanding.

Requirements
  • Minimum 5 users (stretch of 10)
  • Must be familiar with Wikipedia
  • Should be editors or organizers
  • Diverse in age
Languages
  • Spanish
  • French
Location
  • Morocco
  • Cameroon
  • Uruguay
  • DR Congo
Questions

General questions:

  • Does adding the Watchlist feature change users' overall perception of the Wikipedia app?
  • Is a required edit summary sufficient for ensuring users that click undo or revert justify, or would they like to see other interventions for user monitoring edits?
  • Do users feel the Watchlist feature will impact their work, and if so, how?
  • Do they have suggestions for countering potential negative impacts?
  • Do they think the existing features for reacting to rollbacks are sufficient to protect their edits?

Please note: Relevant link to the user groups is here.

4. Tasks

  • @scblr to run user testing with the North America group on userlytics
  • @scblr creates protocol for frequent app editors (Gdoc)
  • @scblr creates protocol for guardrail group (Gdoc)
  • @ARamadan-WMF recruits frequent app-editors from the Northern and Western Europe group to answer the questions on MediaWiki
  • @ARamadan-WMF recruits guardrail group to answer the questions on MediaWiki
  • Add synthesis to MediaWiki

5. Method

  • For now, we can use Figma designs for testing, as demonstrated in this document.

6. Deadline

  • The deadline is flexible but should ideally be by the end of June, allowing us to incorporate the feedback into developing the Watchlist feature.

7. References

Event Timeline

JTannerWMF triaged this task as Medium priority.Apr 26 2023, 2:47 PM
JTannerWMF lowered the priority of this task from Medium to Low.
RWambua-WMF raised the priority of this task from Low to Medium.May 16 2023, 1:56 PM
RWambua-WMF updated the task description. (Show Details)

Hey @RWambua-WMF , some feedback for this ticket:

General Questions

Awareness: Are users aware of the Watchlist feature?

I think the right question here might be are our target users able to find the feature. We don't need all users to be aware of the watchlist features.

Usage: What do users typically use the Watchlist feature for?

I would tweak this to say how are users planning to use the feature or understand how it works.

Value Proposition: Do users find the Watchlist feature useful, and would they use it to track changes to articles of interest?

I think the question here is do users find the feature to be useful for achieving their goals or find it useful compared to other platforms (using it on Web) AND/OR which groups find the feature useful

Functionality: Do users understand the expected functionality of the Watchlist feature from the Figma files?

This is great as is.

User Expectations: Do user expectations of the Watchlist feature align with its actual functionality?

This is similar to value proposition.

Design Questions

What form of navigation would work best for new features and functionalities in the iOS app?

I think this is out of scope

Can users easily find the Watchlist feature within the app, and do they understand the iconography and wording used to represent it?

May be helpful to specify how icons are perceived across languages

Is the Watchlist feature consistent with the overall look and feel of the Wikipedia iOS app, and is the user experience consistent across mobile web and Android platforms?

Would specify we want to understand if cross platform users feel the feature has parity not consistency. We want to stay true to the iOS platform so we it is okay if it is not visually consistent.

How quickly can users learn to use the Watchlist feature effectively?

I would modify to say we want to know if the onboarding is sufficient to help new users understand how to use the feature and the goal of the feature

How well does the Watchlist feature integrate with other features of the app?

It isn't clear what this one means

Guardrail Groups

We could increase awareness among established editors about the challenges faced by contributors from less developed countries and encourage them to be supportive and understanding.

Instead of saying "less developed countries" you should be specific and state the why. Are we thinking about small languages in specific countries, or people in specific countries editing large language wikis?

We could implement a robust feedback mechanism for users whose changes are rolled back, providing clear reasons for the rollback and concrete steps for reinstatement.

This sounds like a suggestion for a new feature? I wonder if instead we should see what features exist for reacting to rollbacks and if it is sufficient or if we should make improvements to those as a first step then consider what new features could be but this may be creating scope creep.

The feature's functionality in showing all language Wikipedias under the watchlist, its localization and cultural sensitivity, and its simplicity of translation should be evaluated.

You'll need to specify which languages you want to test in. It seems you have it in the qual but may want to put it here

Can users easily find the Watchlist feature within the app, and do they understand the iconography and wording used to represent it?

Not sure if for now the primary question should be around iconography / terminology vs. findability in the app

How efficiently can users perform tasks with the Watchlist feature, such as adding an article to the Watchlist or checking updates?

I would add something here about Watchlist expiry

How frequently do users make errors while using the Watchlist feature, what types of errors are they, and how easily can users recover from these errors?

As this is qualitative and not quantitative testing, I wouldn't place an emphasis on frequency -- we're also testing our interface and not the users, so we're looking more to see where people get stuck vs. where they make an 'error' per say.


I have some general questions about the guardrail groups and guardrail questions (apologies if I'm just having trouble understanding!):

  1. Is the concern that Watchlist will lead to a greater number of reverts for these groups based on other editors being able to more easily revert changes through the app?

OR

  1. Is the concern that these groups will have greater visibility into their edits being reverted because of Watchlist?

If the concern is the second, they should probably already be aware of these reverts in the iOS app because of the Notifications feature.

Thanks for these comments! They are super useful.

Some points of further discussion:

General Questions

Awareness: Are users aware of the Watchlist feature?

I agree with this feedback. I have changed the question to: "Awareness: Are our target users able to find the Watchlist feature? Is it discoverable?"

Usage: What do users typically use the Watchlist feature for?

We may have some users who are editors and have used the watchlist as well as users who are editors but have not used the watchlist. Because of this, I have expanded the question into two:

Part 1

Do they already use the feature and if so, what do they typically use it for? And do they find it useful?

Part 2

If not, how are they planning to use the Watchlist? Do they understand how it works? Is the following text helpful in helping them understand how it works? Are they planning to use it in achieving their goals? Does it make sense to their workflow?Do user expectations of the Watchlist feature align with its actual functionality?

Also, I have added this question under this umbrella of questions:

User Expectations: Do user expectations of the Watchlist feature align with its actual functionality?

As for this question:

Value Proposition: Do users find the Watchlist feature useful, and would they use it to track changes to articles of interest?

I've actioned the feedback by changing the question to "Value Proposition: Do users find the Watchlist feature useful compared to using Watchlist from web or from other platforms?".

As for which groups find the feature useful, I believe we can infer this when the users get back to them.

How well does the Watchlist feature integrate with other features of the app?

Removed this as it will not be relevant to this discussion.

Design Questions

What form of navigation would work best for new features and functionalities in the iOS app?

I think this is out of scope for this project but will be useful when discussing the navigation from the app. Also it will be super easy to get feedback on this as @scblr
already had some mocks on the same.

Can users easily find the Watchlist feature within the app, and do they understand the iconography and wording used to represent it?

Changed this to "Can users easily find the Watchlist feature within the app, and do they understand the iconography and wording used to represent it? How are icons perceived across different languages?"

Not sure if for now the primary question should be around iconography / terminology vs. findability in the app

I agree with your prioritization. Perhaps we can find out both is it is findable, and when they find it do users typically understand the iconography or terminology? It would be worrying if it does not become a findable feature.

Is the Watchlist feature consistent with the overall look and feel of the Wikipedia iOS app, and is the user experience consistent across mobile web and Android platforms?

I agree with the feedback. I have changed this to "Is the Watchlist feature consistent with the overall look and feel of the Wikipedia iOS app, and does it have parity with Watchlist features across the other platforms where Watchlist is on such as the mobile web and Android platforms?"

This covers two areas:

  1. Do they feel like the feature lines in well with the feel of the iOS app?
  2. Does the iOS Watchlist achieve parity with Watchlist on other platforms

How quickly can users learn to use the Watchlist feature effectively?

Changed this to "How effective is the proposed onboarding process in helping new users understand how to use the features and understand the goal of Watchlist?"

How efficiently can users perform tasks with the Watchlist feature, such as adding an article to the Watchlist or checking updates?

I agree with the feedback. I have changed the question to: "How efficiently can users perform tasks with the Watchlist feature, such as adding an article to the Watchlist or checking updates? How often would they use watchlist expiry? How easy is it to add an item to the Watchlist Expiry? Is it an intuitive process?"

How frequently do users make errors while using the Watchlist feature, what types of errors are they, and how easily can users recover from these errors?

I agree with the feedback. I have changed the question to: "When users are interacting with the flow, where do they get stuck? What tasks do they get stuck on? What items or functionality is most difficult?"

Around guardrail groups

Is the concern that Watchlist will lead to a greater number of reverts for these groups based on other editors being able to more easily revert changes through the app?

Yes that is the main point of concern. Basically, if someone who speaks Spanish from Morocco cites a news source that may be lesser known on Spanish wiki, they may get more reverts now that the feature is more available to Spanish speakers from Spain for example.

Instead of saying "less developed countries" you should be specific and state the why. Are we thinking about small languages in specific countries, or people in specific countries editing large language wikis?

Hmm.. I partically agree. I could recategorized them into minority language speakers, but unsure if this would cover it. My thought process is, using an example, Kinshasa, the capital of DRC, has more French speakers than Paris, therefore they would not necessarily be a minority language speaking city, however the news sources from Kinshasa, due to it being less developed, may not be as well known as news sources in France.

This sounds like a suggestion for a new feature? I wonder if instead we should see what features exist for reacting to rollbacks and if it is sufficient or if we should make improvements to those as a first step then consider what new features could be but this may be creating scope creep.

I agree, this is a new feature suggestion. I have removed that suggestion and changed the questions to:

Questions for Guardrail Groups:

Do users feel they will be impacted by the Watchlist feature, and if so, how?
Do they have suggestions for countering potential negative impacts?
Do they think that the features that exist for reacting to rollbacks are sufficient to protect their edits?

You'll need to specify which languages you want to test in. It seems you have it in the qual but may want to put it here

I was actually thinking in a bit of a different train of thought of the multiple language articles being shown together but I agree with the addition of languages.

I've rephrased the question to: "The feature's functionality in showing all language Wikipedias under the watchlist, its localization and cultural sensitivity, and its simplicity of translation should be evaluated.

Is it usable in Spanish, Punjabi and Chinese?"

Also I think this is a great addition as it will assist us to test usage in a LTR language (Punjabi)

Thanks for all these responses @RWambua-WMF ! I added some additional notes

I think this is out of scope for this project but will be useful when discussing the navigation from the app. Also it will be super easy to get feedback on this as @scblr
already had some mocks on the same.

The way this is written conveys where should new features generally be placed. If the intention is to get feedback of the entry point being with reading lists, then this point needs to be clearer. Also you can see an example from Olga where she asked "where do you expect to find the feature" which touches on discoverability which we covered in awareness without it blowing up the scope of the task of asking about overall navigation of the app.

As for which groups find the feature useful, I believe we can infer this when the users get back to them.

I am not clear on what you mean here. Explicitly asking which group finds the feature useful, will ensure when Robin defines the audiences of who the run the test with he knows if he should get feedback from new users or experienced users or both. Small med or large languages or all.

I agree with your prioritization. Perhaps we can find out both is it is findable, and when they find it do users typically understand the iconography or terminology? It would be worrying if it does not become a findable feature.

I think this is why its important to define who is "they". We don't want it to be so prominent that we introduce the feature to users who wouldn't find it useful at the wrong time. But we do want to make sure that the most interested customers are able to discover it. From a scoping perspective, discoverability may blow up scope but I'll leave that to Robin if he has capacity to include it.

I agree with the feedback. I have changed this to "Is the Watchlist feature consistent with the overall look and feel of the Wikipedia iOS app, and does it have parity with Watchlist features across the other platforms where Watchlist is on such as the mobile web and Android platforms?"

This covers two areas:

Do they feel like the feature lines in well with the feel of the iOS app?
Does the iOS Watchlist achieve parity with Watchlist on other platform

Just leaving an explicit note that I care more about Parity with the iOS app and continuity with other platforms. It doesn't need to look the same but we should honor user state and a cross platform user should understand functionality when switching between platforms.

How easy is it to add an item to the Watchlist Expiry?

Small change here, "how easy is it for someone to update their watchlist expiry"

Hmm.. I partically agree. I could recategorized them into minority language speakers, but unsure if this would cover it. My thought process is, using an example, Kinshasa, the capital of DRC, has more French speakers than Paris, therefore they would not necessarily be a minority language speaking city, however the news sources from Kinshasa, due to it being less developed, may not be as well known as news sources in France.

Okay so then this should be updated from "less developed countries" to something like "people in formerly colonized countries or former protectorates editing on large language wikis" OR Emerging Communities (as defined by the community relations team) editing on large language wikis (English, French, Spanish, Portuguese etc.), but its important to be specific.

Is it usable in Spanish, Punjabi and Chinese?"

While this is valid question, I was being literal, when running the test, Robin will have to select from an endless list of languages to run the test in. He won't be able to have the script translated in hundreds of languages, so you'll need to specify 3-5 languages maximum for him to run the test in. Of course when the feature is ran it should certainly be available in all languages, you're absolutely right.

scblr renamed this task from Get moderated design feedback for iOS Watchlist to Get user feedback for iOS Watchlist.May 31 2023, 3:36 PM

The way this is written conveys where should new features generally be placed. If the intention is to get feedback of the entry point being with reading lists, then this point needs to be clearer. Also you can see an example from Olga where she asked "where do you expect to find the feature" which touches on discoverability which we covered in awareness without it blowing up the scope of the task of asking about overall navigation of the app.

Ok, that definitely works. I've removed it to reflect the scope of the task.

I am not clear on what you mean here. Explicitly asking which group finds the feature useful, will ensure when Robin defines the audiences of who the run the test with he knows if he should get feedback from new users or experienced users or both. Small med or large languages or all.

I've added two sections to cover our target groups:

  1. General Target Audiences
  2. Specific Target Audiences

I think this is why it's important to define who is "they". We don't want it to be so prominent that we introduce the feature to users who wouldn't find it useful at the wrong time. But we do want to make sure that the most interested customers are able to discover it. From a scoping perspective, discoverability may blow up scope but I'll leave that to Robin if he has capacity to include it.

I'll let @scblr determine whether to scope it in or not.

Just leaving an explicit note that I care more about Parity with the iOS app and continuity with other platforms. It doesn't need to look the same but we should honor user state and a cross platform user should understand functionality when switching between platforms.

I've changed the terminology in the previous text to reflect this.

Small change here, "how easy is it for someone to update their watchlist expiry"

Yes, changed this text to read, "How easy is it for someone to update their watchlist expiry?" so that we can find out if watchlist expiry is easy to use.

Okay so then this should be updated from "less developed countries" to something like "people in formerly colonized countries or former protectorates editing on large language wikis" OR Emerging Communities (as defined by the community relations team) editing on large language wikis (English, French, Spanish, Portuguese etc.), but its important to be specific.

Defined our guardrail groups more clearly.

While this is valid question, I was being literal, when running the test, Robin will have to select from an endless list of languages to run the test in. He won't be able to have the script translated in hundreds of languages, so you'll need to specify 3-5 languages maximum for him to run the test in. Of course when the feature is ran it should certainly be available in all languages, you're absolutely right.

Agreed! I've added in the specific languages that we can look at.

Also, a couple of other additions, I've classified the questions according to the target groups and added a reference link to the work @scblr has done.

@RWambua-WMF

CleanShot 2023-06-27 at 16.16.57@2x.png (1×1 px, 290 KB)

I noticed that the languages for the guardrail group are missing in the task’s description. Could you add them, please? Thanks!

FYI: I added more structure to the task’s description in my latest update (see Diff).

Findings from Usability Testing for entry points

New Users

All users identified female, all spoke English, one user also spoke Chinese - with more lead time there could have been more diversity in the panel

  • 2 of 6 users easily found watchlist
    • One user related the watchlist to a bookmarking feature and expected for watchlist to be under the bookmark icon on first view
    • Two users expected it to be under the edit pencil
    • One user expected it to be under the find in page icon
  • 3 of 4 users (who did not initially go to the more menu) were not surprised to find it under the overflow menu once they had tried other places (all of these testers associated the icon with 'more' or 'menu')
  • 5 of 6 users found it easy to modify expiry
  • 5 of 6 users found it easy to remove an article from watchlist
Editors

All users identified as female, all users spoke English, two users also spoke Spanish - with more lead time there could have been more diversity in the panel

  • 3 of 3 users easily found watchlist
  • 3 of 3 users found it easy to modify expiry
  • 3 of 3 users found it easy to remove an article from watchlist