@ovasileva
@phuedx

Is it possible to show this to just account/IP users who make edit(s)? If not, go ahead and show to everyone. Thanks!

In T248421#6002641, @dchen wrote:

@ovasileva
@phuedx

Is it possible to show this to just account/IP users who make edit(s)? If not, go ahead and show to everyone. Thanks!

@dchen - afraid this isn't possible - we'd have to show it to everyone. Perhaps we can edit this within the copy and add something like "Do you edit Wikipedia"? Another question - do you have a rough idea on how many responses you would like to receive/what the sampling rate for the survey should be?

@ovasileva ok got it. I'll just ask that question in the linked survey.

Re responses: not really... I need as many as possible to fill approx 10-15 interview slots. Hopefully getting as many subcohorts of user types filled as well. Hard to quantify that in total # of responses, especially if we can't target only those making edits...

thanks & let me know if you have any other questions!

In T248421#6003035, @dchen wrote:

@ovasileva ok got it. I'll just ask that question in the linked survey.

Re responses: not really... I need as many as possible to fill approx 10-15 interview slots. Hopefully getting as many subcohorts of user types filled as well. Hard to quantify that in total # of responses, especially if we can't target only those making edits...

thanks & let me know if you have any other questions!

@dchen - it might be more efficient to ask from the very beginning (that way we're not getting people to click through to the survey that are not part of the group we're trying to test for). Totally up to you though! In terms of sampling rate, what would be the best number of responses for the 15 interview slots? Around 100? (We're currently getting around 11K anon edits on all wikis per day, around 1.5 - 2K for enwiki only)

@ovasileva

hmm good point. possible to add this question and 'activate' the survey link only for the two Yes answers?

How often have you edited Wikipedia?

• Yes, many times --> GETS SURVEY LINK
• Yes, once or twice --> GETS SURVEY LINK
• No
• I didn't know I could edit Wikipedia
• I don't remember

hmm i'm not really sure as it is totally dependent on response rates, availability, follow through... i guess we can start with 100 and see how it goes?

@ovasileva hey olga, just wondering if you have an ETA on getting this out? tx!

Tip for mitigating risk with running this survey - do the deploy in 2 steps - 1 to enable the survey on the wiki to a sample size of 0 and then 2 to dispatch it to users. Doing this in two steps would allow us to check the survey itself using a query string parameter prior to deploying.

In T248421#6040361, @dchen wrote:

@ovasileva hey olga, just wondering if you have an ETA on getting this out? tx!

@dchen - sorry for the delay here. We just estimated this and it's all ready to go. We'd like to test it first to make sure everything is okay, and then we'll be ready to deploy everywhere. Due to testing, it will probably be early next week.

In T248421#6040427, @Jdlrobson wrote:

Tip for mitigating risk with running this survey - do the deploy in 2 steps - 1 to enable the survey on the wiki to a sample size of 0 and then 2 to dispatch it to users. Doing this in two steps would allow us to check the survey itself using a query string parameter prior to deploying.

Now added this to the acceptance criteria

Change 592787 had a related patch set uploaded (by Nray; owner: Nray):
[mediawiki/extensions/WikimediaMessages@master] Add Messages for Growth Study Quick Survey

https://gerrit.wikimedia.org/r/592787

Change 592788 had a related patch set uploaded (by Nray; owner: Nray):
[operations/mediawiki-config@master] Add Config for Growth Study Quick Survey

https://gerrit.wikimedia.org/r/592788

@dchen @ovasileva I currently have the survey setup locally, but ran into a few questions I was hoping you could clarify. It currently looks like:

As you can see, this deviates from the AC in a couple ways.

1. The "...invite you to chat with us about your experiences." text and "Thank you in advance for your interest and for visiting Wikipedia!" text are not separated out by any spacing. I think this is a limitation of QuickSurveys which escapes HTML. If the spacing is desired, we'd probably need to tweak how QuickSurveys renders. Relatedly, I placed the "Wikimedia User Research team" line at the bottom near the privacy policy text to achieve separation from the description and make it more like the AC. Does this suffice?

2. The AC links the "screener survey" text to the survey, but from what I can tell, QuickSurveys assumes that link will be a "Visit survey" button combined with a "No thanks" button. One of the reasons for this is that when the user clicks either button, QuickSurveys persists that state to the user's browser so when they refresh the page, they won't be bothered by the survey again. However, QuickSurvey's is not setup to do that magic with a link in the description. Is that okay?

Additionally:

• Just FYI, I've setup the config to work on both desktop and mobile assuming that is what is wanted. (Please let me know if not!)
• I took the liberty to make the privacy policy text read as "For more information, see the survey's privacy statement", but that can easily be changed.

From task description:

I'd like to request a quicksurveys to be deployed as soon as possible on all language Wikipedia sites (the CTA itself will be in English as the research needs to be conducted in English. If this means that we should only deploy this on English-language wikis, that is fine, but ideally all lang wikis can still see it).

From AC:

Wikis: enwiki

@ovasileva @dchen Also, should we deploy this to all wikipedias or only English wikipedia? The task description mentions all wikipedia would be preferable, but the AC only lists English wikipedia.

If deploying to all wikipedias is preferable, is it okay for the text in the survey to be translated in each wiki's language? I'm asking because I think QuickSurveys depends on a translation key for the text.

In T248421#6087395, @nray wrote:

@dchen @ovasileva I currently have the survey setup locally, but ran into a few questions I was hoping you could clarify. It currently looks like:

As you can see, this deviates from the AC in a couple ways.

1. The "...invite you to chat with us about your experiences." text and "Thank you in advance for your interest and for visiting Wikipedia!" text are not separated out by any spacing. I think this is a limitation of QuickSurveys which escapes HTML. If the spacing is desired, we'd probably need to tweak how QuickSurveys renders. Relatedly, I placed the "Wikimedia User Research team" line at the bottom near the privacy policy text to achieve separation from the description and make it more like the AC. Does this suffice?

2. The AC links the "screener survey" text to the survey, but from what I can tell, QuickSurveys assumes that link will be a "Visit survey" button combined with a "No thanks" button. One of the reasons for this is that when the user clicks either button, QuickSurveys persists that state to the user's browser so when they refresh the page, they won't be bothered by the survey again. However, QuickSurvey's is not setup to do that magic with a link in the description. Is that okay?

Additionally:

• Just FYI, I've setup the config to work on both desktop and mobile assuming that is what is wanted. (Please let me know if not!)
• I took the liberty to make the privacy policy text read as "For more information, see the survey's privacy statement", but that can easily be changed.

Sounds good @nray

In T248421#6089627, @nray wrote:

From task description:

I'd like to request a quicksurveys to be deployed as soon as possible on all language Wikipedia sites (the CTA itself will be in English as the research needs to be conducted in English. If this means that we should only deploy this on English-language wikis, that is fine, but ideally all lang wikis can still see it).

From AC:

Wikis: enwiki

@ovasileva @dchen Also, should we deploy this to all wikipedias or only English wikipedia? The task description mentions all wikipedia would be preferable, but the AC only lists English wikipedia.

If deploying to all wikipedias is preferable, is it okay for the text in the survey to be translated in each wiki's language? I'm asking because I think QuickSurveys depends on a translation key for the text.

Since there's no translations, I think enwiki is best, since we usually try to avoid putting english text on the other wikis

Change 593002 had a related patch set uploaded (by Nray; owner: Nray):
[translatewiki@master] Ignore Growth Screener QuickSurvey Messages

https://gerrit.wikimedia.org/r/593002

Thanks @ovasileva . This should be ready to be swatted then pending code review. The relevant patches are:

https://gerrit.wikimedia.org/r/592787 (Adds messages to WikimediaMessages)
https://gerrit.wikimedia.org/r/593002 (Ignores messages in translatewiki since we only want survey to run on enwiki)
https://gerrit.wikimedia.org/r/592788 (Turns on survey in mediawiki-config with sample size 0 per AC).

@nray @ovasileva
these are all fine. thank you!

Change 593002 merged by jenkins-bot:
[translatewiki@master] Ignore Growth Screener QuickSurvey Messages

https://gerrit.wikimedia.org/r/593002

Change 592787 merged by Nray:
[mediawiki/extensions/WikimediaMessages@master] Add Messages for Growth Study Quick Survey

https://gerrit.wikimedia.org/r/592787

Change 593064 had a related patch set uploaded (by Nray; owner: Nray):
[mediawiki/extensions/WikimediaMessages@wmf/1.35.0-wmf.30] Add Messages for Growth Study Quick Survey

https://gerrit.wikimedia.org/r/593064

Change 593081 had a related patch set uploaded (by Nray; owner: Nray):
[mediawiki/extensions/WikimediaMessages@wmf/1.35.0-wmf.28] Add Messages for Growth Study Quick Survey

https://gerrit.wikimedia.org/r/593081

Change 593064 abandoned by Nray:
Add Messages for Growth Study Quick Survey

Reason:
wrong branch

https://gerrit.wikimedia.org/r/593064

Change 593081 merged by jenkins-bot:
[mediawiki/extensions/WikimediaMessages@wmf/1.35.0-wmf.28] Add Messages for Growth Study Quick Survey

https://gerrit.wikimedia.org/r/593081

Change 593064 restored by Catrope:
Add Messages for Growth Study Quick Survey

https://gerrit.wikimedia.org/r/593064

Change 593064 merged by jenkins-bot:
[mediawiki/extensions/WikimediaMessages@wmf/1.35.0-wmf.30] Add Messages for Growth Study Quick Survey

https://gerrit.wikimedia.org/r/593064

Mentioned in SAL (#wikimedia-operations) [2020-04-28T23:48:21Z] <catrope@deploy1001> Started scap: Update WikimediaMessages with new i18n messages for T248421

Mentioned in SAL (#wikimedia-operations) [2020-04-29T00:43:44Z] <catrope@deploy1001> Finished scap: Update WikimediaMessages with new i18n messages for T248421 (duration: 55m 23s)

Change 592788 merged by jenkins-bot:
[operations/mediawiki-config@master] Add Config for Growth Study Quick Survey

https://gerrit.wikimedia.org/r/592788

Mentioned in SAL (#wikimedia-operations) [2020-04-29T00:56:30Z] <catrope@deploy1001> Synchronized wmf-config/InitialiseSettings.php: Enable Growth Study QuickSurvey on enwiki (with sample size 0, for testing) (T248421) (duration: 01m 10s)

@ovasileva @dchen The survey has been swatted to enwiki with a sample size of 0 to test per the AC. To see the survey, you may visit any article as long as the quicksurvey=true query param is appended at the end. For example:

https://en.wikipedia.org/wiki/Tree?quicksurvey=true

In T248421#6091875, @nray wrote:

@ovasileva @dchen The survey has been swatted to enwiki with a sample size of 0 to test per the AC. To see the survey, you may visit any article as long as the quicksurvey=true query param is appended at the end. For example:

https://en.wikipedia.org/wiki/Tree?quicksurvey=true

@nray - I don't seem to see the survey when I'm logged-out. Did we turn it on only for logged-in users? I think it's supposed to be the reverse (sorry if that wasn't clear in the AC) - I'll make a note right now

Change 593265 had a related patch set uploaded (by Nray; owner: Nray):
[operations/mediawiki-config@master] Set Growth Screener Survey sample rate to 0.1% and limit to anon audience

https://gerrit.wikimedia.org/r/593265

@ovasileva Thanks for catching that. The survey is currently set to show to everyone (logged in and anon), but I've created a new patch to limit it to anons only as well as to set the sampling rate to 0.1%. You might not be seeing it for anons if you've already clicked one of the buttons on the survey ("Visit survey" or "No thanks") as it won't show again unless you open a new incognito window .

Update: I'm planning on swatting the following revisions at 16:00 PDT today:

1. Setting the sampling rate to 0.1%
2. Making the survey visible to anons only

https://gerrit.wikimedia.org/r/#/c/operations/mediawiki-config/+/593265/

Change 593265 merged by jenkins-bot:
[operations/mediawiki-config@master] Set Growth Screener Survey sample rate to 0.1% and limit to anon audience

https://gerrit.wikimedia.org/r/593265

Mentioned in SAL (#wikimedia-operations) [2020-04-29T23:37:40Z] <catrope@deploy1001> Synchronized wmf-config/InitialiseSettings.php: Set Growth Screener survey sample rate to 0.1% and limit to anons only (T248421) (duration: 01m 05s)

@ovasileva @dchen The swat was a spectacular success story. The changes are live now and the survey will only show to anon users on english wikipedia with a sampling rate of 0.1%.

You can see the survey on english wikipedia again by using the quicksurvey=true query param e.g. (https://en.wikipedia.org/wiki/Barack_Obama?quicksurvey=true) in an incognito window. Please note that by using this query param, it will bypass the audience/sampling rate filter that QuickSurveys uses making it possible to see the survey even when logged in. Don't be alarmed, however, as without this query param only anon users will be able to see the survey.

Also, note that although the AC states that this survey should run for 2 weeks, I don't think QuickSurveys provides any way to automatically turn it off so we'll have to turn it off manually via another swat (which I guess should be around May 13).

Yayy! thank you @nray!

@nray - I think this might just be my browser, but I still can't get the survey to show in Chrome (incognito, with https://en.wikipedia.org/wiki/Barack_Obama?quicksurvey=true). Works fine on Firefox and Safari though

@ovasileva: I'd be interested to see if there's anything showing up in your browser's console. I can confirm that I'm seeing the survey in a fresh incognito window in Google Chrome (81.0.4044.129) on macOS Catalina (10.15.4).

There've been impressions > 57000 [0] and nearly 1000 responses [1] for the survey as of now:

57108


select
    count(*)
from event.quicksurveyinitiation
where
    year = 2020
    and month = 5
    and day = 1

    and event.surveycodename = 'growth-study-screener-survey'
    and event.eventname = 'impression'
;

998


select
    count(*)
from event.quicksurveysresponses
where
    year = 2020
    and month = 5
    and day = 1

    and event.surveycodename = 'growth-study-screener-survey'
;

In T248421#6099756, @phuedx wrote:

@ovasileva: I'd be interested to see if there's anything showing up in your browser's console. I can confirm that I'm seeing the survey in a fresh incognito window in Google Chrome (81.0.4044.129) on macOS Catalina (10.15.4).

What should I look for? I'm also Chrome (81.0.4044.129) on macOS Catalina (10.15.4).

@ovasileva Does your Chrome's browser console differ in any way from my screenshot?:

@nray - seems it's DNT although it shouldn't be on in incognito

👍👍👍 DNT strikes again! Makes sense, I see in the QuickSurveys code we abort early if that is enabled

Change 593939 had a related patch set uploaded (by Phuedx; owner: Phuedx):
[mediawiki/extensions/QuickSurveys@master] Log when the user has enabled Do Not Track

https://gerrit.wikimedia.org/r/593939

☝ Hopefully the above is a super-quick-to-merge QoL improvement. It'll log a message to the web console if the user has enabled DNT.

Change 593939 merged by jenkins-bot:
[mediawiki/extensions/QuickSurveys@master] Log when the user has enabled Do Not Track

https://gerrit.wikimedia.org/r/593939

Not sure who to tell this to, but came across this task and I wanted to say that if there are people working on documentation for how to run QuickSurveys, I have some knowledge in my head that I would be glad to contribute. Of note, I wanted to say that for the future, we generally host the privacy policy on foundation.wikimedia.org (example) so that people can read it without worrying about Google (or other) trackers.

I'm a little confused as what we're asking now, gjven the original scope was deployment and we don't tend to reopen tasks.

Can we open a new bug with the remaining work?
Note in terms of documentation we are not working on this right now, but https://meta.wikimedia.org/wiki/QuickSurveys#How_can_I_create_a_Quick_Survey and associated talk page would be the place to do it.

@Jdlrobson i wasn't sure if reopening or making a new task was the best move... so i did both... here is the other task: https://phabricator.wikimedia.org/T257015

Feel free to move/close as needed, thanks!