Change Details

Surveys implement a `coverage` rate, allowing eg. 10% of editors to be selected to see a survey and the other 90% never see it. I believe this is a sticky association which relies on a cookie set by `mw.experiments.getBucket`. This cookie seems to not be invalidated by a change to the coverage rate for a survey, which means that changing the rate partway through a campaign will have a surprising impact which is difficult to predict. For example, if the campaign has been running for many weeks and targets logged-in users, then the pool of users without a cookie will be small and changing coverage will have little effect. If the campaign targets anonymous users and was only running for a few days, then changing coverage will have a large and expected effect. This task is complete once we've documented the behavior of changing `coverage` [[ https://www.mediawiki.org/wiki/Extension:QuickSurveys | in the manual ]]. Depending on what we find, follow-up work might include a behavior change such as invalidating "don't show" cookies when the coverage is changed (but leaving "have seen / dismissed / responded" cookies untouched). More questions we want clarify on: * Is bucket assignment reset for every new session, or sticky for logged-in users? * Is there a maximum number of impressions per user, or will the survey show up on every page view until dismissed? * How does the dismissal feature work? Is there always a button to dismiss, what does it look like? Is dismissal done via cookie (so would not have an effect for high-privacy browsing?)

Surveys implement a `coverage` rate, allowing eg. 10% of editors to be selected to see a survey and the other 90% never see it. I believe this is a sticky association which relies on a cookie set by `mw.experiments.getBucket`. This cookie seems to not be invalidated by a change to the coverage rate for a survey, which means that changing the rate partway through a campaign will have a surprising impact which is difficult to predict. For example, if the campaign has been running for many weeks and targets logged-in users, then the pool of users without a cookie will be small and changing coverage will have little effect. If the campaign targets anonymous users and was only running for a few days, then changing coverage will have a large and expected effect. This task is complete once we've documented the behavior of changing `coverage` [[ https://www.mediawiki.org/wiki/Extension:QuickSurveys | in the manual ]]. Depending on what we find, follow-up work might include a behavior change such as invalidating "don't show" cookies when the coverage is changed (but leaving "have seen / dismissed / responded" cookies untouched). Questions: * Is the `coverage` rate used only on the first visit, or for each page view? * Participation is recalculated for every page view, so changes to `coverage` have immediate effect and will change which users are participating. * Note that if coverage is increased, the users who were already participating in the survey are a strict subset of the new participants. In other words, if the coverage is increased from 5% to 10%, half of the users now participating are the same 5% who were included under the original coverage. * Is bucket assignment reset for every new session, or sticky for logged-in users? * Although the bucket assignment is recalculated for every pageview, it's done using a sticky token. Each individual will receive the same assignment until their browser's local storage is purged (a rare case). * Is there a maximum number of impressions per user, or will the survey show up on every page view until dismissed? * You will keep seeing the survey until it's dismissed. Even if the coverage is low, eg. 1%, a user chosen to participate in the survey will see it on every page view. If an individual is chosen for multiple surveys, you'll see a random choice of survey. * How does the dismissal feature work? Is there always a button to dismiss, what does it look like? Is dismissal done via cookie (so would not have an effect for high-privacy browsing?) * Dismissal can only be done via the close box. * Once it's clicked, the user gets a sticky, local storage token which prevents them from seeing the survey again. * If local storage is purged then they are eligible to see the survey again.

Surveys implement a `coverage` rate, allowing eg. 10% of editors to be selected to see a survey and the other 90% never see it. I believe this is a sticky association which relies on a cookie set by `mw.experiments.getBucket`. This cookie seems to not be invalidated by a change to the coverage rate for a survey, which means that changing the rate partway through a campaign will have a surprising impact which is difficult to predict. For example, if the campaign has been running for many weeks and targets logged-in users, then the pool of users without a cookie will be small and changing coverage will have little effect. If the campaign targets anonymous users and was only running for a few days, then changing coverage will have a large and expected effect. This task is complete once we've documented the behavior of changing `coverage` [[ https://www.mediawiki.org/wiki/Extension:QuickSurveys | in the manual ]]. Depending on what we find, follow-up work might include a behavior change such as invalidating "don't show" cookies when the coverage is changed (but leaving "have seen / dismissed / responded" cookies untouched). More questions we want clarify on:Questions: * Is the `coverage` rate used only on the first visit, or for each page view? * Participation is recalculated for every page view, so changes to `coverage` have immediate effect and will change which users are participating. * Note that if coverage is increased, the users who were already participating in the survey are a strict subset of the new participants. In other words, if the coverage is increased from 5% to 10%, half of the users now participating are the same 5% who were included under the original coverage. * Is bucket assignment reset for every new session, or sticky for logged-in users? * Although the bucket assignment is recalculated for every pageview, it's done using a sticky token. Each individual will receive the same assignment until their browser's local storage is purged (a rare case). * Is there a maximum number of impressions per user, or will the survey show up on every page view until dismissed? * You will keep seeing the survey until it's dismissed. Even if the coverage is low, eg. 1%, a user chosen to participate in the survey will see it on every page view. If an individual is chosen for multiple surveys, you'll see a random choice of survey. * How does the dismissal feature work? Is there always a button to dismiss, what does it look like? Is dismissal done via cookie (so would not have an effect for high-privacy browsing?) * Dismissal can only be done via the close box. * Once it's clicked, the user gets a sticky, local storage token which prevents them from seeing the survey again. * If local storage is purged then they are eligible to see the survey again.