Background

The Growth team has created a "welcome" survey to show users just after they create a new account, with the aim to capture information that may be useful in future in helping onboard and promote new editor retention.

Goals of this user test

1. Identify improvements to Understanding of the survey (update copy depending on users' comprehension of the instructions and questions on the form).
2. Identify improvements to the Usability of the survey (check whether users are able to correctly input and submit their intended responses).
   • Do users know how the visual design and layout works? (Eg., do they understand they should be selecting one answer from a radio button group)
   • Are users able to navigate through the form?
   • Any feedback missing that could help users complete the form?
3. Gauge user Reactions to the survey and Expectations of how the information will be used.

Testing format

UserTesting.com will be used to recruit users and run the test.
It will be a remote, unmoderated test where users will be asked to fill out the survey as series of tasks, while "talking aloud" their expectations and understanding of each question. It will be followed by a few written questions. Each session test should take no longer than 5-10min to complete.

Testers

6 users will be recruited using UserTesting.com. The intention will be to screen for respondents who likely do not already have a Wikipedia account.

Test protocol

https://docs.google.com/document/d/1peIHQ2k8aKiPfXitTMWZ_N1cYhfVnODhCTh8s9XGPT4

Findings deck

https://docs.google.com/presentation/d/1u4mQ9tkHn2JiK4rIPJrSqXixfCPQOzjPgp6AlIe36qg/

Summary of findings

• Survey was clearly optional, and seen as short, low-effort, and non-intrusive to complete
• Generally seen as data capture for research (though a few testers thought more specific explanation about how responses may be used might make them more inclined to fill it in)
• "Mentorship program" was the one aspect not clearly understood, with about half of participants mistakenly assuming they would be providing the help rather than receiving it.
• A couple of users had misgivings about providing email and feared it may be misused for marketing or given to 3rd parties (also as they expected it to be mandatory for account creation if it was needed for recovery)
• Users liked the post-submission message with more information about "Getting started with editing"

Recommendations

• Add a full Thanks message after survey completion
• Rephrase “Mentor” checkbox question and clarify expectations for how users may be contacted if they select the mentor checkbox
• Revise phrasing of Q3 (Wordsmithing so that the the second sentence “We've listed a few below popular for editing” is more easily understood.)
• Add a tooltip beside or assistive text under the “Add more topics” field for no-js users explaining they can comma-separate multiple topics
• Add more information about how the email is used
  • Potentially be extra clear by placing a tooltip with the following message (taken from Help:Account_management): “If you choose to give an email address, other users will be able to contact you by email. This feature is anonymous—the user who emails you will not know your email address. You don't have to give your email address if you don't want to (but doing so is required to reset your password if you forget it)”