Since deploying the welcome survey and seeing initial data coming in, we have started talking about how the questions and responses could be rephrased so that responses match our intent.
We want to be deliberate about when we make any changes so that our data analysis doesn't get mixed up. We should not make any changes before Variation A has run for a month.
@RHo has been collecting some of these thoughts.
In the question, "Have you ever edited Wikipedia?", the response "No, I didn't know I could edit Wikipedia" is getting a higher response rate than expected. This might be because it includes both people who mean, "No, I didn't know I could edit Wikipedia until I read this question" and people who mean, "No, I didn't know I could edit Wikipedia until earlier today, which is why I am currently creating my account." One idea to help here might be to simply exclude the word "No", so that it says, "I didn't know I could edit Wikipedia." And the response that currently reads, "No, for other reasons" would read, "No, I have never edited".