Page MenuHomePhabricator

Repeat demographics surveys for longer time period
Open, Needs TriagePublic

Description

One of the major questions around interpreting the first round of demographics surveys has been what readers are our results reflective of? That is, do the results best represent the population of readers who reads daily? Or the population who reads weekly? monthly?

Why might a longer survey give different results?

Reasons why a longer survey (one month) might change the results, especially with respect to gender:

  • External surveys have shown that the gender distribution of Wikipedia readers can depend on whether you ask something like "Do you read daily?" or "Do you read weekly?"
  • For several languages in the first round of surveys, a trend was seen where the further into the week that the survey went, the more balanced the proportion of men/women in the respondents.
  • From debiasing, women were consistently less likely to respond to the surveys than men
  • It is known that, especially on desktop, it is incredibly easy to miss the survey widget. For readers who visit Wikipedia once a week or less, this means that it would be incredibly unlikely for them to see the widget to respond.

Ethical considerations

We are always hesitant to survey readers unnecessarily because however small the survey widget, it does still have the potential to disrupt reading. One of the benefits of running a survey for longer is that we can drastically lower the sampling rate (e.g., in the realm of 1:100). This ensures that only a small proportion of readers will see this survey and therefore readers with many browsers or who clear cookies will still be unlikely to be resampled into the survey. Repeatedly resampling the same user is mainly problematic from the perspective of respecting readers as opposed to statistics / results.

We also will not run these surveys in all 13 languages but limit them to new languages and a few languages whose sampling rate was lower than 1 in 20 in the first round (to minimize overlap between surveys). In this way, we do not overburden every language community but also get some sense of whether any patterns we see are language specific. We hope that the results will provide some indication of how much the results of the other languages might shift with a longer survey period.

Details

The exact same survey will be used to maximize the comparability of the June round of surveys and this round (though notably seasonality will make direct comparison impossible as it will likely have a large impact).

  • English was selected due to its wide coverage and availability of survey data in America
  • Polish was not part of the first round so they are being included this round
  • Russian is included because it is a language that had a low sampling rate the first round (less likely for readers to have already seen the survey) and displayed the narrowing gender gap throughout the week.

Sampling rates are still more an art than a science. In this case, the survey is expected to go 4x longer than the June survey, but with more time to see and respond to the survey, we expect response rate will increase. This would argue for at least a 6-8x reduction in sampling rate. For both Russian and English, we also do not need as many responses as were gathered in the first round (ideally more like 1000-2000 responses), so we can further reduce to 12-16x.

LanguageSurvey ReadyPrivacy Policy UploadedInterface Pages CreatedVillage Pump NotifiedJune Sampling RateJune Responses*Proposed Sampling Rate
en -- English (world)YesYesT232525#550736623-Sept-191 of 9861811 of 1200
pl -- PolishYesYesT232525#551628323-Sept-19N/AN/A1 of 200
ru -- RussianYesYesT232525#551205023-Sept-191 of 5945651 of 720

* This is the total number of responses after removing those under 18 and responses that could not be linked to EventLogging, so it's a lower number than others reported. But it's the number that matters when analyzing the gender of the reading population.

Details

Event Timeline

Isaac created this task.Sep 10 2019, 7:57 PM
Isaac added a comment.Sep 11 2019, 7:20 PM

Regarding the sampling rate for Polish Wikipedia, their page views are generally about one third of those to Russian Wikipedia (https://tools.wmflabs.org/siteviews/?platform=all-access&source=pageviews&agent=user&range=this-year&sites=ru.wikipedia.org|pl.wikipedia.org) and unique devices are also a bit under one third (https://stats.wikimedia.org/v2/#/pl.wikipedia.org/reading/unique-devices/normal|line|2-year|~total|monthly). Based on this, I will recommend setting the sampling rate to three times that of Russian Wikipedia.

Isaac updated the task description. (Show Details)Sep 19 2019, 3:27 PM
Isaac added a comment.EditedSep 19 2019, 6:53 PM

English (worldwide)

SurveyDurationGoalStart Date and timeEnd Date
reader-demographics-en28 days2000 responses2019/9/24 @ 1100 UTC (0700 EST)2019/10/15 @ 1100 UTC

Interface pages

enwiki => [
    'enabled' => true,
    "name" => "reader-demographics-en",
    "type" => "external",
    "description" => "Reader-demographics-1-description",
    "link" => "Reader-demographics-2-link",
    "question" => "Reader-demographics-1-message",
    "privacyPolicy" => "Reader-demographics-1-privacy",
    "coverage" => 0.000833, // 1 out of 1200
    "instanceTokenParameterName" => "entry.1791119923",
    "platforms" => [
        "desktop"=> ["stable"],
        "mobile"=> ["stable"]
    ],
]
Isaac added a comment.EditedSep 20 2019, 9:08 PM

Russian

SurveyDurationGoalStart Date and timeEnd Date
reader-demographics-ru28 days2000 responses2019/9/24 @ 1100 UTC (0700 EST)2019/10/15 @ 1100 UTC

Interface pages

ruwiki => [
    'enabled' => true,
    "name" => "reader-demographics-ru",
    "type" => "external",
    "description" => "Reader-segmentation-1-description",
    "link" => "Reader-demographics-2-link",
    "question" => "Reader-demographics-1-message",
    "privacyPolicy" => "Reader-demographics-1-privacy",
    "coverage" => 0.00167, // 1 out of 600
    "instanceTokenParameterName" => "entry.1791119923",
    "platforms" => [
        "desktop"=> ["stable"],
        "mobile"=> ["stable"]
    ],
]
Isaac added a comment.EditedSep 23 2019, 2:19 PM

Polish

SurveyDurationGoalStart Date and timeEnd Date
reader-demographics-pl28 days2000 responses2019/9/26 @ 1100 UTC (0700 EST)2019/10/17 @ 1100 UTC

Interface pages

plwiki => [
    'enabled' => true,
    "name" => "reader-demographics-pl",
    "type" => "external",
    "description" => "Reader-demographics-1-description",
    "link" => "Reader-demographics-1-link",
    "question" => "Reader-demographics-1-message",
    "privacyPolicy" => "Reader-demographics-1-privacy",
    "coverage" => 0.005, // 1 out of 200
    "instanceTokenParameterName" => "entry.1791119923",
    "platforms" => [
        "desktop"=> ["stable"],
        "mobile"=> ["stable"]
    ],
]
Isaac updated the task description. (Show Details)Sep 23 2019, 3:04 PM

Change 539183 had a related patch set uploaded (by Isaac Johnson; owner: Isaac Johnson):
[operations/mediawiki-config@master] Enable reader demographic surveys in English, Polish, and Russian.

https://gerrit.wikimedia.org/r/539183

Change 539183 merged by jenkins-bot:
[operations/mediawiki-config@master] Enable reader demographic surveys in English, Polish, and Russian. With proper links now.

https://gerrit.wikimedia.org/r/539183

Mentioned in SAL (#wikimedia-operations) [2019-09-26T11:10:47Z] <urbanecm@deploy1001> Synchronized wmf-config/InitialiseSettings.php: SWAT: 7645e55: Enable reader demographic surveys in English, Polish, and Russian (T232525) (duration: 01m 06s)

Isaac updated the task description. (Show Details)Sep 26 2019, 11:27 AM

Had no responses to the Village Pump posts or on the meta page so deployed the surveys this morning. The intent is for one month but we will monitor response count / feedback and adjust as needed.

Isaac moved this task from Staged to In Progress on the Research board.Sep 30 2019, 4:24 PM

Change 547339 had a related patch set uploaded (by Isaac Johnson; owner: Isaac Johnson):
[operations/mediawiki-config@master] Undeploy reader surveys in English, Polish, and Russian.

https://gerrit.wikimedia.org/r/547339

Change 547339 merged by jenkins-bot:
[operations/mediawiki-config@master] Undeploy reader surveys in English, Polish, and Russian.

https://gerrit.wikimedia.org/r/547339

Mentioned in SAL (#wikimedia-operations) [2019-10-31T18:12:33Z] <urbanecm@deploy1001> Synchronized wmf-config/InitialiseSettings.php: SWAT: fe08fbb: Undeploy reader surveys in English, Polish, and Russian (T232525) (duration: 01m 02s)