EN:VP post for pilot (posted 22 Feb 2019): https://en.wikipedia.org/wiki/Wikipedia:Village_pump_(miscellaneous)#Notification_for_a_survey

Current status and findings:

• We are concerned about how much selection bias we may be seeing in the pilot results (i.e. is the survey reaching a representative sample of readers or not).
  • We are seeing a much higher proportion of younger users and users who identify as men than we expected. This could be because these users truly read Wikipedia more frequently (and so are more likely to be included in the survey) or it could be due to higher rates of self-selection into the survey. We will evaluate whether these trends are consistent by country and other demographics.
• We are concerned about whether certain bugs are affecting the QuickSurvey sampling and logging and would like to address these (or at least better understand them) before moving on:
  • T218243 which would mean mobile is being undersampled. This matches what we see in our survey, which is that while e.g., ~20% of English Wikipedia readers use Chrome, 34% of the devices that saw the survey used Chrome
  • Approximately 12% of our survey responses cannot be matched to QuickSurveyInitiation EventLogging. The reason for this is unclear: almost all of the survey codes from these responses look normal and the timestamps are relatively evenly distributed across the survey deployment.
  • Approximately 18% of our survey responses cannot be matched to QuickSurveysResponses EventLogging: this higher percentage is possibly due to QuickSurveysResponses EL not being captured if the user explicitly right-clicks and opens the survey in a new tab (verified by me and see T131315#2311065 for related issue)

While we are still waiting on findings related to QuickSurveys (T218243 and T220627), we have decided to move forward w/ the survey in other languages. A few notes:

• Found supporting evidence that men do indeed read Wikipedia more frequently than women in the United States: https://papers.ssrn.com/sol3/papers.cfm?abstract_id=2617021
  • Based on survey of 1000 AMT workers from US: "Second, men use Wikipedia more often — they are twice as likely than women to use Wikipedia daily"
• While younger respondents were consistently more likely to read Wikipedia frequently, mixed evidence from Global Insights phone surveys on gender:
  • India: women more likely to be frequent readers of Wikipedia
  • Mexico: men more likely to be frequent readers of Wikipedia
  • Nigeria: men slightly more likely to be frequent readers of Wikipedia
  • Iraq: ~equal likelihood by gender of being frequent readers of Wikipedia
• Though we're missing EventLogging for ~10% of our responses, it actually is more likely to be missing from our younger, male readers (T220627#5113109), so if there are issues with QuickSurveys loading, this would suggest that if anything it would lead to greater skew in the demographics results