Table of contents
- Background
- Research questions
- Test format
- Screener questions
- Opening scenario
- Protocol
- Post-test questionnaire
- Instructions to run this test
- Test variables
- Checklist before running the test
1. Background
The Android team will enhance our communication systems with a focus on messaging interfaces and workflows. Last year we released native talk pages. We want to understand what gaps exist in our current interface as-is and, which elements of our feature are a success, and we should consider not touching.
2. Research questions
1 ) Gauge the discoverability of user and article talk pages
2 ) Identify potential improvements to the usability of the talk page:
- Do users understand how to start a new subject on an article talk page?
- Are users able to successfully follow the steps to post a new discussion?
- Are users able to respond to a user talk page notification they receive?
- Is there anything that users would like to be able to do in our native talk page experience that they can’t currently or is there anything that would make their experience more efficient?
- What is the user’s understanding of a revert notification?
- If a user posts a message on a talk page what are the expectations of how they will hear back?
3 ) Gauge user reactions to talk pages
4 ) Observe differences in how users interact with article and user talk pages.
3. Test format
This will be an unmoderated remote task-based test, conducted using usertesting.com to recruit participants and record sessions.
Link to APK:
https://github.com/wikimedia/apps-android-wikipedia/releases/download/latest/app-alpha-universal-release.apk
4. Screener questions
Usertesting.com allows for 4 screener questions. Given the constraint we are pursuing the following plan to reach our target audience:
- Use the “Rest of the World” and “India” checkbox on usertesting.com
- Use a language screener question for Arabic, French, Hindi, Indonesian, Japanese
1 ) Which of the following are you familiar with?
- Facebook [May select]
- Google Docs [May select]
- Wikipedia [Must select]
- Twitter [May select]
- Wordpress [May select]
- WeChat [May select]
- WhatsApp [May select]
- None of the above [Reject]
2 ) How often do you use Wikipedia (either on the web or in the app)?
- Multiple times a day [ Accept ]
- Daily [ Accept ]
- 2-3 times a week [ Accept ]
- Weekly [ Accept ]
- 2-3 times a month [ Accept ]
- Less than once a month [ Accept ]
- I have never used Wikipedia before [ Reject ]
3 ) Have you ever edited an article on Wikipedia?
- Yes, many times [Reject]
- Yes, once or twice [Accept]
- I don’t remember [Accept]
- No, because I didn't know it could be edited [Accept]
- No, for other reasons [Accept]
4 ) Aside from English, which language(s) do you speak at an advanced to native level?
- Arabic [Accept]
- French [Accept]
- Hindi [Accept]
- Indonesian [Accept]
- Japanese [Accept]
- Spanish [Reject]
- German [Reject]
- Russian [Reject]
- I don’t speak other languages [Reject]
5. Opening Scenario
Imagine you are a user logged into a Wikipedia account as:
Usertestingandroid3_Francais
You have just noticed the following Wikipedia article:
https://test.wikipedia.org/wiki/Chat
6. Protocol
This is the protocol for the French version.
1 )
Please set your device’s system language to French, before opening the Wikipedia app.
The following steps are for you to log in to see the parts of the app we would like you to review. After installing and opening the app, you will see the app onboarding. Please tap 'SKIP' in the bottom area of the screen.
2 )
You should now be in the first menu item of the app, which is called the 'Explore' feed. From here:
Tap the 'More' menu item at the bottom area of the screen.
Tap the 'LOG IN / JOIN WIKIPEDIA' link
Tap the 'LOG IN' button below 'Already have an account?'
Use the following credentials to log in:
Username:
Usertestingandroid3_Francais
Password:
3 )
You should now be logged in with this username and back on the home screen (the 'Explore' feed).
4 )
Go to 'Settings' via 'More' menu item in the main app navigation and tap 'Wikipedia languages'.
Tap the '+ ADD LANGUAGE' button
Tap the search icon in the top area of the screen (magnifying glass)
Please type or copy/paste the following text into the search input field:
Test
Tap the item that shows up (Test, Test)
5 )
You should now be back on the 'Wikipedia languages' screen.
Tap the three dots in the top area of the screen and tap 'Remove language'
Remove all items except for 'Test' from this list, tap the trash icon in the top area of the screen and confirm the dialog with 'OK'.
6 )
Please tap the back button (←) in the top area of the screen twice.
You should now be back on the home screen (the 'Explore' feed).
Now you are all set to start the test.
7 )
Navigate to the search field and type or copy/paste the following text into the search box:
Chat
Tap the search result and navigate to the article.
8 )
You should now see the 'Cat' Wikipedia article in a language you speak.
You want to discuss a part of this Wikipedia article and correct, expand or reference it. Talk us through where you would go to do this. What would you expect to see?
9 )
Tap on the item that you think takes you to the place where you can discuss a part of this Wikipedia article. Please move on to the next task if you're not sure what to do or are otherwise having trouble with this task.
10 )
If you haven’t already, please tap on the three dots in the top area of the screen. Then tap 'View talk page'. The Wikipedia 'Talk page' is the place where you can discuss a part of this Wikipedia article and correct, expand or reference it. What do you notice on this screen? Please describe your thoughts out loud.
11 )
You are still in the mindset of someone that wants to collaborate with other editors on Wikipedia.
Start a new discussion thread to ask if anyone would want to collaborate on improving the article. Please narrate your thoughts through this process. Then, please publish the new discussion thread.
12 )
If you have not already, please tap '+ NEW TOPIC' button in the bottom area of the screen to start a new discussion thread. Publish the new topic you created by tapping the blue tick icon in the top area of the screen.
13 )
Now that you have successfully published a discussion thread on the talk page, please narrate how you expect to receive a reply and from whom?
14 )
On the talk page of this article, select a conversation thread that you did not start.
15 )
Narrate your understanding of the discussion on this page.
16 )
How would you reply in the discussion thread? Please narrate your thoughts out loud, then go ahead and do it.
17 )
Where do you think you would get notifications about your activities on Wikipedia? Please narrate where you think you would go to notifications. Then go ahead and navigate there. Please move on to the next task if you're not sure what to do or are otherwise having trouble with this task.
18 )
If you don’t find notifications:
Navigate to your notifications via the bell icon in the top area of the screen.
19 )
What do you think the 'Your edit on...was reverted' notification means? Please narrate your thoughts out loud.
Then tap a notification item that says 'Your edit on ... was reverted'.
20 )
Narrate your understanding of this screen.
21 )
Navigate back to your notifications.
22 )
What do you think the '...left a message on your talk page' notification means? Please narrate your thoughts out loud.
Then tap a notification item that says '...left a message on your talk page'.
23 )
Narrate your understanding of this screen. Then reply to the message you just read in English or another language you speak.
24 )
Please tap the back button (←) in the top area of the screen twice.
You should now be back on the 'Explore' feed.
Go to 'Settings' via 'More' menu item in the main app navigation.
Scroll to the bottom of the 'Settings' screen.
Tap 'LOG OUT'.
7. Post-test questionnaire
1 ) What are your thoughts on the talk page experience you just reviewed?
2 ) What worked well for you?
3 ) What did not work or was confusing for you? Is there anything you'd like to be able to do but couldn't?
8. Instructions to run this test
Create an article page with your user account on test.wikipedia.org in the language you want to test.
Copy the source text of test.wikipedia.org/wiki/Cat and run it through Google for the target language.
With the translation at hand, create or update the article in the target language, e.g. test.wikipedia.org/wiki/Chat for the French protocol.
Create new topics on this article’s talk page with your username, here’s an English template:
== Welcome == This talk page has been created to perform usability tests for the official Wikipedia Android App. The various posts on this page are fictitious. == Criticism == The article lacks important sources. I'm not sure if the readers get a complete picture of cats when reading this article. == Source for weight? == I do not have a high-quality source on hand at the moment, but large cats can definitely weigh a lot! == Clarification == Domestic cats do not count among the species that may be hunted, unless otherwise specified at state level. == Historical figures? == Does anyone know a source where historical domestic cat populations are estimated compared to today? Couldn't find anything in a quick search. Would maybe be interesting to mention here? == Indoor cats == The section "Indoor cats" has no valid sources but makes recommendations. However, I have not found any scientific text that proves that cats should only be kept in pairs. On the contrary, the vast majority of cats kept alone do not show any abnormalities. == Praise == This article is very extensive and informative. Thanks to all authors! == Vandalism == I've noticed that many users have been editing the article at the same time lately. The edits are of a low quality. Could it be vandalism?
Navigate to the user’ talk page in the corresponding language with your user account and create the following topics (run it through Google translate):
== Hi there == I noticed you are new to Wikipedia. Let me know if I could be of any help, okay? == Thanks == I really enjoyed your contributions to the dog article. Great work! == Your recent edits == You missed adding reliable sources to the dog article, so I reverted the edit.
Make an edit (change cat breeds from 60 to 50) with the participant’s user account in the corresponding language to the cat article with the following edit summary:
I believe it is 50 not 60
Revert this edit and provide the following summary with your user account to trigger a revert notification for the participant:
A reliable source was missing
9. Test variables
Arabic
- Article name:قط
- Link: https://test.wikipedia.org/wiki/قط
- Username: QTE-Test6-WMF
French
- Article name: Chat
- Link: https://test.wikipedia.org/wiki/Chat
- Username: QTE-Test5-WMF
Hindi
- Article name: बिल्ली
- Link: https://test.wikipedia.org/wiki/बिल्ली
- Username: QTE-Test4-WMF
Indonesian
- Article name: Kucing
- Link: https://test.wikipedia.org/wiki/Kucing
- Username: QTE-Test7-WMF
Japanese
- Article name: ネコ
- Link to article: https://test.wikipedia.org/wiki/ネコ
- Username: QTE-Test8-WMF
10. Checklist before running the test
- Check if article page exists
- Check if article talk page contains topics
- Check if user talk page contains topics
- Has an edit with the participant’s user account been made?
- Has the edit been reverted with your user account?
- Log into the app with the participant’s user account for a last check.
- Make sure the protocol contains all the variables from
- Screener #4 (language)
- Opening scenario (username)
- Protocol #2 (username + password)
- Protocol #7 (article name)
Please find the analysis of this work here: T294078