Why are we doing this?
A revised search experience for the Wikipedia Android app has been released on September 29, 2020 (T256819).
Our initial hypothesis:
- Search becomes more consistent by having it in the same place on both the bottom navigation and article toolbar.
- Searching becomes more ergonomic by moving it from the top to the bottom in the article page.
What did we change specifically?
 |  |
Before (left)
- Search icon is positioned at the top of the article.
- Find in page is accessible via bottom navigation.
After (right)
- Article search is accessible via toolbar (bottom)
- Find in page has been moved to the overflow menu (top right)
- Bottom toolbar has been changed to accent50 for better distinction from bottom navigation.
Problem
Based on user feedback on Google Play and a decline in search usage, we have reason to believe that repositioning the search button from the top to the bottom on the article page did not have the desired effect of our hypotheses, stated above.
What are we testing?
We created three fully functional concepts and want to find out about pros and cons of each variant. What are the differences and how do they look like? 👇
Variant A (Link)
- Search input top
- Find in article bottom
Variant B (Link)
- Search icon top
- Find in article bottom
Variant C (Link)
- Search icon bottom
- Find in article in overflow menu (top right)
Variant A: This is new design of the search that features a search input at the top of the article. Why did we come up with it? Android search data indicates, that the majority of search queries within the app (~ 60% come from) search input field on Explore. This concept is based on this known behavior.
More details about the variants
Variant B: This has been the search design in the Android app until Sep 29.
Variant C: This is the current search design of the app. It is rolled out since Sep 29 and can be downloaded and installed via Google Play Store.
What are the goals of the test?
- Through close observation, we want to find out about pros and cons of each variant, with a specific focus on discoverability and speed of access.
- How discoverable is Search Wikipedia, how discoverable is Find in page in each variant.
- How can the specific variants be improved?
- Bonus: do people find their History after visiting articles?
How are we testing?
We will utilize UserTesting.com to recruit participants, record sessions and facilitate the test.
The test will be an unmoderated, task based test. Participants will be given a set of tasks to complete in a 15-20 minute time frame. Participants will perform the test remotely.
Test
Screener
5 participants per variant will be recruited using UserTesting.com.
Question 1: Have you ever used the Wikipedia mobile app before (either on Android or iOS)?
- Yes : [ Accept ]
- No : [ Accept ]
- Not sure : [ Reject ]
Question 2: How often do you use Wikipedia (either on the web or in the app)?
- Multiple times a day : [ Accept ]
- Daily : [ Accept ]
- 2-3 times a week : [ Accept ]
- Weekly : [ Accept ]
- 2-3 times a month : [ Accept ]
- Less than once a month : [ Accept ]
- I have never used Wikipedia before : [ Reject ]
Instructions
For Variant A:
Please download and install the APK of the Wikipedia app with the following link:
https://releases.wikimedia.org/mobile/android/wikipedia/scratch/app-alpha-release_searchproto1.apk
For Variant B:
Please download and install the APK of the Wikipedia app with the following link:
https://releases.wikimedia.org/mobile/android/wikipedia/scratch/app-alpha-release_searchproto2.apk
For Variant C:
Please download and install the latest version of the Wikipedia app from Google Play.
https://play.google.com/store/apps/details?id=org.wikipedia&hl=en
- After installing and opening the app, you will see the app onboarding.
- Please tap 'Skip' at the bottom left.
- You should find yourself in the first tab of the app (the so called Explore feed).
- You’re all set to start the test.
Opening scenario
Imagine that you just downloaded the Wikipedia app.
Script
01)
Please open the Wikipedia app you just downloaded on your Android device.
You will see the app onboarding. Please tap 'Skip' at the bottom left.
02)
You can dismiss the 'Customize your Explore feed' card here.
You will now see a feed that features content of Wikipedia. This is called the 'Explore' feed.
Here you are at the right place, this is your starting point.
03)
Imagine you’ve seen a documentary film about life in the arctic last night.
Now you would like to learn more about polar bears. What would you do to find more information about the polar bear on Wikipedia?
04)
Please search the Wikipedia article for Polar bear.
If you have done this already, please move on to the next task.
05)
If you have not already done so, please tap on the search bar at the top of the 'Explore' feed and search for 'polar bear,' then tap through to the Polar bear article.
06)
Imagine that you are interested in how climate change affects the lives of polar bears, how would you find information about climate change in this article?
07)
Look for the words 'climate change' in this article.
If you have done this already, please move on to the next task.
08)
If you have not already done so, please locate the 'Find in article' feature and search for the words 'climate change'. If you are unable to find this feature after 1 minute, please move on to the next task.
09)
If the 'Find in article' feature is still open and active, please close it.
10)
You are now back on the polar bear article.
Imagine you are fascinated by bears and you want to find out more about another bear species, the 'brown bear'.
11)
If you have not already done so, please locate the 'Search' feature and search for 'brown bear', then tap through to the Brown bear article.
12)
From the brown bear article, please navigate back to where you started this usability test.
13)
If you have not already done so, tap back (top left) until you are back in the 'Explore' feed, where Wikipedia content is featured.
14)
Imagine you are accessing the Wikipedia Android app a week later. You want to find out about articles you have visited last week, because you want to learn even more about bears.
Please navigate to articles you have visited in the past (History).
15)
You were successful if you found the 'History' feature that is part of the 'Search' tab in the main navigation.
If you haven’t visited the 'Search' tab (next to 'My lists' and 'Edits') yet, please navigate to it.
16)
Did you notice the blue tooltip when tapping search in the bottom navigation?
Do you remember what it said? What did you learn from it?
Please describe your thoughts out loud.
17)
Was the search functionality easy to find and quick to use?
Please describe your thoughts out loud.
18)
What was your general experience in searching articles?
What did work? What did not work? What would you improve?
Please describe your thoughts out loud.
19)
Was it clear to you which type of search to use in each case? Was the distinction between searching articles and find words in articles clear to you?
Please describe your thoughts out loud.
20)
That is it. Thank you.