Draft a survey for Content Translation users, but do not send it out. Work in a new etherpad, using whatever language you are most comfortable writing.
The goal of the survey is to learn more about how this software is used, and how translation languages are chosen.
Please note: Several people can work on this Phabricator task - please do not claim / assign this task to yourself. Thanks!
- Write a short introduction to the survey, explaining what data we want to collect.
- Quantitative
- first language
- most common source and target language (Give examples of how to write this clearly, eg. "en -> es".)
- …
- Qualitative
- biggest difficulties encountered translating
- what should newcomers be aware of
- …
NOTES (You do not need to read them all to get started, I just could not pin the post with them, so I pasted them here!)
- General concept: It helps to know what you want to find out and particularly, what your assumptions are. If you know your assumptions, you can put them to a test. We broadly wrote "The goal of the survey is to learn more about how this software is used" but that is pretty vague, maybe actually a question for a qualitative study. How do you think people translate? Can you learn something about translations from published research that informs your assumptions? With assumptions and questions about them you can focus your survey questions and the survey design.
- Introduction: This is very important as it is what participants can use to see if they want to participate or not. Give the purpose of the survey but avoid merely repeating questions you are going to ask. Check if the language is clear. In this case, it might be tempting to use e.g. "qualitative" and "quantitative" here, but are reseracher jargon that might be unfamiliar to the participants.
- Questions:
- Aim to make them both a) short and clear b) self-contained, so they can be understood without reading previous questions
- Check if they make sense in the context of the tool. E.g. asking what people translate with the tool or if they used customer service might generally be good ideas, but they do not apply to the tool which is usually used to translate wikipedia articles and has no customer support in the conventional sense.
- Answers: As important as the questions you ask are the answers you allow…
- Age: People might not like to give their age in years for various reasons, so age ranges are a good practice. The need to be unambigous: 20-30, 30-40 are ambigous – what do I check when I am 30? Better: 20-29, 30-39 or 21-30, 31-40, …
- Gender: "How to Do Better with Gender on Surveys: A Guide for HCI Researchers" is my go-to resource (They recommend a man/woman/non-binary/prefer to self define/prefer not to answer). However, this does not mean that you need to take this route; some people might also opt for an open text field (in which case you need to think about how to analyse that) or not ask gender at all.
- Nationality/Country: Important to keep in mind that the state controlling the territorry they are on might be different than the nation they see themselves belonging to.
- Language (proficiency): Not easy, because it is unclear what skill level is meant. You could also set a definition like "good enough to write a Wikipedia article in that language" (which is not a perfect criterion, but at least something). If it is important to you, you could ask for their proficiency level. If that makes sense depends on your research interest (you can use more powerful analysis methods with rank-able items like a proficiency scale rather than a binary competent/not-competent scale, but it is more difficult to answer, so the question if it is worth it)
- Asking for frequencies: To better compare these, give some hints what you mean: "In the last month, how often did you…": Never/ 1-2 times/3-10 times/11-50 times/ more than 51 times (there are different scales you can use – look at some examples)
- Reasons for translations, motivations, other qualitative questions: To ask this you should have some good hypothesis of what you can to do with the data. As merely descriptive data of how participants are like they are not very informative (they depend on how people interpret them, how desireable the answers are etc.). They can be useful if you have an hypothesis like "people who are motivated by increasing coverage in their native language edit smaller wikipedias" (which would not be surprising, but can be tested)
Here are some more resources to learn about survey design:
- 12 Tips For Writing Better Survey Questions, Jeff Sauro, measuringu
- Writing surveys that work, Rebecca Weiss, Mozilla