Surveys implement a coverage rate, allowing eg. 10% of editors to be selected to see a survey and the other 90% never see it. I believe this is a sticky association which relies on a cookie set by mw.experiments.getBucket. This cookie seems to not be invalidated by a change to the coverage rate for a survey, which means that changing the rate partway through a campaign will have a surprising impact which is difficult to predict. For example, if the campaign has been running for many weeks and targets logged-in users, then the pool of users without a cookie will be small and changing coverage will have little effect. If the campaign targets anonymous users and was only running for a few days, then changing coverage will have a large and expected effect.

This task is complete once we've documented the behavior of changing coverage in the manual.

Depending on what we find, follow-up work might include a behavior change such as invalidating "don't show" cookies when the coverage is changed (but leaving "have seen / dismissed / responded" cookies untouched).

Questions:

• Is the coverage rate used only on the first visit, or for each page view?
  • Participation is recalculated for every page view, so changes to coverage have immediate (up to the ResourceLoader cache lag of 15 minutes) effect and will change which users are participating.
  • Note that if coverage is increased, the users who were already participating in the survey are a strict subset of the new participants. In other words, if the coverage is increased from 5% to 10%, half of the users now participating are the same 5% who were included under the original coverage.
• Is bucket assignment reset for every new session, or sticky for logged-in users?
  • Although the bucket assignment is recalculated for every pageview, it's done using a sticky token. Each individual will receive the same assignment until their browser's local storage is purged (a rare case).
• Is there a maximum number of impressions per user, or will the survey show up on every page view until dismissed?
  • You will keep seeing the survey until it's dismissed. Even if the coverage is low, eg. 1%, a user chosen to participate in the survey will see it on every page view. If an individual is chosen for multiple surveys, you'll see a random choice of survey.
• How does the dismissal feature work? Is there always a button to dismiss, what does it look like? Is dismissal done via cookie (so would not have an effect for high-privacy browsing?)
  • Dismissal can only be done via the close box "X" in the top-right corner, or the "no, thanks" button.
  • The close box is standard on every survey dialog.
  • Once it's clicked, the user gets a sticky, local storage token which prevents them from seeing the survey again.
  • If local storage is purged then they are eligible to see the survey again.