Page MenuHomePhabricator

Deploy Quicksurveys extension on all Wikipedias (for a Growth study)
Closed, ResolvedPublic3 Estimated Story Points

Description

Hi all - I'm reopening this task and putting in 'needs triage' to get another round of this survey going (for ~3 weeks, per my convo with Olga). Not sure if this is the best SOP? Let me know if you need a new ticket entirely and if you have any other questions/concerns. All CTA/survey elements remain the same.


Background

Hi team,

I'd like to request a quicksurveys to be deployed as soon as possible on all language Wikipedia sites (the CTA itself will be in English as the research needs to be conducted in English. If this means that we should only deploy this on English-language wikis, that is fine, but ideally all lang wikis can still see it).

Would like this to run for 2 weeks for now, and will reassess at the end of that period.

If applicable/possible, trigger would be:
XXX1) a logged out IP user makes an edit
Ideal option-->2) if not possible, show to any user who makes an edit

  1. if not possible, show to everyone

Acceptance criteria

Hello Wikipedian!

We are looking to learn from our users and how we can best support your experiences on Wikipedia. Please fill out a short screener survey and we may invite you to chat with us about your experiences.

Thank you in advance for your interest and for visiting Wikipedia!
The Wikimedia User Research team


Event Timeline

There are a very large number of changes, so older changes are hidden. Show Older Changes
dchen updated the task description. (Show Details)Mar 26 2020, 4:56 PM
dchen updated the task description. (Show Details)EditedMar 26 2020, 5:14 PM
dchen added subscribers: ovasileva, phuedx.

@ovasileva
@phuedx

Is it possible to show this to just account/IP users who make edit(s)? If not, go ahead and show to everyone. Thanks!

@ovasileva
@phuedx

Is it possible to show this to just account/IP users who make edit(s)? If not, go ahead and show to everyone. Thanks!

@dchen - afraid this isn't possible - we'd have to show it to everyone. Perhaps we can edit this within the copy and add something like "Do you edit Wikipedia"? Another question - do you have a rough idea on how many responses you would like to receive/what the sampling rate for the survey should be?

dchen added a comment.Mar 26 2020, 7:01 PM

@ovasileva ok got it. I'll just ask that question in the linked survey.

Re responses: not really... I need as many as possible to fill approx 10-15 interview slots. Hopefully getting as many subcohorts of user types filled as well. Hard to quantify that in total # of responses, especially if we can't target only those making edits...

thanks & let me know if you have any other questions!

ovasileva added a comment.EditedMar 30 2020, 10:03 AM

@ovasileva ok got it. I'll just ask that question in the linked survey.

Re responses: not really... I need as many as possible to fill approx 10-15 interview slots. Hopefully getting as many subcohorts of user types filled as well. Hard to quantify that in total # of responses, especially if we can't target only those making edits...

thanks & let me know if you have any other questions!

@dchen - it might be more efficient to ask from the very beginning (that way we're not getting people to click through to the survey that are not part of the group we're trying to test for). Totally up to you though! In terms of sampling rate, what would be the best number of responses for the 15 interview slots? Around 100? (We're currently getting around 11K anon edits on all wikis per day, around 1.5 - 2K for enwiki only)

dchen added a comment.Mar 31 2020, 7:56 AM

@ovasileva

hmm good point. possible to add this question and 'activate' the survey link only for the two Yes answers?

How often have you edited Wikipedia?

  • Yes, many times --> GETS SURVEY LINK
  • Yes, once or twice --> GETS SURVEY LINK
  • No
  • I didn't know I could edit Wikipedia
  • I don't remember

hmm i'm not really sure as it is totally dependent on response rates, availability, follow through... i guess we can start with 100 and see how it goes?

dchen added a comment.Apr 2 2020, 4:24 AM
This comment was removed by dchen.
ovasileva raised the priority of this task from Medium to High.Apr 8 2020, 3:39 PM
ovasileva updated the task description. (Show Details)
Restricted Application added a subscriber: Dereckson. · View Herald TranscriptApr 8 2020, 3:40 PM
dchen added a comment.Apr 8 2020, 4:07 PM

@ovasileva hey olga, just wondering if you have an ETA on getting this out? tx!

ovasileva set the point value for this task to 3.Apr 8 2020, 4:16 PM

Tip for mitigating risk with running this survey - do the deploy in 2 steps - 1 to enable the survey on the wiki to a sample size of 0 and then 2 to dispatch it to users. Doing this in two steps would allow us to check the survey itself using a query string parameter prior to deploying.

@ovasileva hey olga, just wondering if you have an ETA on getting this out? tx!

@dchen - sorry for the delay here. We just estimated this and it's all ready to go. We'd like to test it first to make sure everything is okay, and then we'll be ready to deploy everywhere. Due to testing, it will probably be early next week.

ovasileva updated the task description. (Show Details)Apr 9 2020, 8:44 AM

Tip for mitigating risk with running this survey - do the deploy in 2 steps - 1 to enable the survey on the wiki to a sample size of 0 and then 2 to dispatch it to users. Doing this in two steps would allow us to check the survey itself using a query string parameter prior to deploying.

Now added this to the acceptance criteria

Jdlrobson updated the task description. (Show Details)Apr 9 2020, 4:22 PM
Jdlrobson updated the task description. (Show Details)

Change 592787 had a related patch set uploaded (by Nray; owner: Nray):
[mediawiki/extensions/WikimediaMessages@master] Add Messages for Growth Study Quick Survey

https://gerrit.wikimedia.org/r/592787

Change 592788 had a related patch set uploaded (by Nray; owner: Nray):
[operations/mediawiki-config@master] Add Config for Growth Study Quick Survey

https://gerrit.wikimedia.org/r/592788

nray added a subscriber: nray.EditedApr 28 2020, 12:07 AM

@dchen @ovasileva I currently have the survey setup locally, but ran into a few questions I was hoping you could clarify. It currently looks like:

As you can see, this deviates from the AC in a couple ways.

  1. The "...invite you to chat with us about your experiences." text and "Thank you in advance for your interest and for visiting Wikipedia!" text are not separated out by any spacing. I think this is a limitation of QuickSurveys which escapes HTML. If the spacing is desired, we'd probably need to tweak how QuickSurveys renders. Relatedly, I placed the "Wikimedia User Research team" line at the bottom near the privacy policy text to achieve separation from the description and make it more like the AC. Does this suffice?
  1. The AC links the "screener survey" text to the survey, but from what I can tell, QuickSurveys assumes that link will be a "Visit survey" button combined with a "No thanks" button. One of the reasons for this is that when the user clicks either button, QuickSurveys persists that state to the user's browser so when they refresh the page, they won't be bothered by the survey again. However, QuickSurvey's is not setup to do that magic with a link in the description. Is that okay?

Additionally:

  • Just FYI, I've setup the config to work on both desktop and mobile assuming that is what is wanted. (Please let me know if not!)
  • I took the liberty to make the privacy policy text read as "For more information, see the survey's privacy statement", but that can easily be changed.
nray added a comment.EditedApr 28 2020, 4:13 PM

From task description:

I'd like to request a quicksurveys to be deployed as soon as possible on all language Wikipedia sites (the CTA itself will be in English as the research needs to be conducted in English. If this means that we should only deploy this on English-language wikis, that is fine, but ideally all lang wikis can still see it).

From AC:

Wikis: enwiki

@ovasileva @dchen Also, should we deploy this to all wikipedias or only English wikipedia? The task description mentions all wikipedia would be preferable, but the AC only lists English wikipedia.

If deploying to all wikipedias is preferable, is it okay for the text in the survey to be translated in each wiki's language? I'm asking because I think QuickSurveys depends on a translation key for the text.

@dchen @ovasileva I currently have the survey setup locally, but ran into a few questions I was hoping you could clarify. It currently looks like:

As you can see, this deviates from the AC in a couple ways.

  1. The "...invite you to chat with us about your experiences." text and "Thank you in advance for your interest and for visiting Wikipedia!" text are not separated out by any spacing. I think this is a limitation of QuickSurveys which escapes HTML. If the spacing is desired, we'd probably need to tweak how QuickSurveys renders. Relatedly, I placed the "Wikimedia User Research team" line at the bottom near the privacy policy text to achieve separation from the description and make it more like the AC. Does this suffice?
  1. The AC links the "screener survey" text to the survey, but from what I can tell, QuickSurveys assumes that link will be a "Visit survey" button combined with a "No thanks" button. One of the reasons for this is that when the user clicks either button, QuickSurveys persists that state to the user's browser so when they refresh the page, they won't be bothered by the survey again. However, QuickSurvey's is not setup to do that magic with a link in the description. Is that okay?

Additionally:

  • Just FYI, I've setup the config to work on both desktop and mobile assuming that is what is wanted. (Please let me know if not!)
  • I took the liberty to make the privacy policy text read as "For more information, see the survey's privacy statement", but that can easily be changed.

Sounds good @nray

From task description:

I'd like to request a quicksurveys to be deployed as soon as possible on all language Wikipedia sites (the CTA itself will be in English as the research needs to be conducted in English. If this means that we should only deploy this on English-language wikis, that is fine, but ideally all lang wikis can still see it).

From AC:

Wikis: enwiki

@ovasileva @dchen Also, should we deploy this to all wikipedias or only English wikipedia? The task description mentions all wikipedia would be preferable, but the AC only lists English wikipedia.

If deploying to all wikipedias is preferable, is it okay for the text in the survey to be translated in each wiki's language? I'm asking because I think QuickSurveys depends on a translation key for the text.

Since there's no translations, I think enwiki is best, since we usually try to avoid putting english text on the other wikis

Change 593002 had a related patch set uploaded (by Nray; owner: Nray):
[translatewiki@master] Ignore Growth Screener QuickSurvey Messages

https://gerrit.wikimedia.org/r/593002

nray added a comment.EditedApr 28 2020, 4:58 PM

Thanks @ovasileva . This should be ready to be swatted then pending code review. The relevant patches are:

https://gerrit.wikimedia.org/r/592787 (Adds messages to WikimediaMessages)
https://gerrit.wikimedia.org/r/593002 (Ignores messages in translatewiki since we only want survey to run on enwiki)
https://gerrit.wikimedia.org/r/592788 (Turns on survey in mediawiki-config with sample size 0 per AC).

dchen added a comment.Apr 28 2020, 5:57 PM

@nray @ovasileva
these are all fine. thank you!

Change 593002 merged by jenkins-bot:
[translatewiki@master] Ignore Growth Screener QuickSurvey Messages

https://gerrit.wikimedia.org/r/593002

Change 592787 merged by Nray:
[mediawiki/extensions/WikimediaMessages@master] Add Messages for Growth Study Quick Survey

https://gerrit.wikimedia.org/r/592787

Change 593064 had a related patch set uploaded (by Nray; owner: Nray):
[mediawiki/extensions/WikimediaMessages@wmf/1.35.0-wmf.30] Add Messages for Growth Study Quick Survey

https://gerrit.wikimedia.org/r/593064

Change 593081 had a related patch set uploaded (by Nray; owner: Nray):
[mediawiki/extensions/WikimediaMessages@wmf/1.35.0-wmf.28] Add Messages for Growth Study Quick Survey

https://gerrit.wikimedia.org/r/593081

Change 593064 abandoned by Nray:
Add Messages for Growth Study Quick Survey

Reason:
wrong branch

https://gerrit.wikimedia.org/r/593064

Change 593081 merged by jenkins-bot:
[mediawiki/extensions/WikimediaMessages@wmf/1.35.0-wmf.28] Add Messages for Growth Study Quick Survey

https://gerrit.wikimedia.org/r/593081

Change 593064 restored by Catrope:
Add Messages for Growth Study Quick Survey

https://gerrit.wikimedia.org/r/593064

Change 593064 merged by jenkins-bot:
[mediawiki/extensions/WikimediaMessages@wmf/1.35.0-wmf.30] Add Messages for Growth Study Quick Survey

https://gerrit.wikimedia.org/r/593064

Mentioned in SAL (#wikimedia-operations) [2020-04-28T23:48:21Z] <catrope@deploy1001> Started scap: Update WikimediaMessages with new i18n messages for T248421

Mentioned in SAL (#wikimedia-operations) [2020-04-29T00:43:44Z] <catrope@deploy1001> Finished scap: Update WikimediaMessages with new i18n messages for T248421 (duration: 55m 23s)

Change 592788 merged by jenkins-bot:
[operations/mediawiki-config@master] Add Config for Growth Study Quick Survey

https://gerrit.wikimedia.org/r/592788

Mentioned in SAL (#wikimedia-operations) [2020-04-29T00:56:30Z] <catrope@deploy1001> Synchronized wmf-config/InitialiseSettings.php: Enable Growth Study QuickSurvey on enwiki (with sample size 0, for testing) (T248421) (duration: 01m 10s)

nray added a comment.Apr 29 2020, 1:08 AM

@ovasileva @dchen The survey has been swatted to enwiki with a sample size of 0 to test per the AC. To see the survey, you may visit any article as long as the quicksurvey=true query param is appended at the end. For example:

https://en.wikipedia.org/wiki/Tree?quicksurvey=true

@ovasileva @dchen The survey has been swatted to enwiki with a sample size of 0 to test per the AC. To see the survey, you may visit any article as long as the quicksurvey=true query param is appended at the end. For example:

https://en.wikipedia.org/wiki/Tree?quicksurvey=true

@nray - I don't seem to see the survey when I'm logged-out. Did we turn it on only for logged-in users? I think it's supposed to be the reverse (sorry if that wasn't clear in the AC) - I'll make a note right now

ovasileva updated the task description. (Show Details)Apr 29 2020, 8:32 AM

Change 593265 had a related patch set uploaded (by Nray; owner: Nray):
[operations/mediawiki-config@master] Set Growth Screener Survey sample rate to 0.1% and limit to anon audience

https://gerrit.wikimedia.org/r/593265

nray added a comment.Apr 29 2020, 3:59 PM

@ovasileva Thanks for catching that. The survey is currently set to show to everyone (logged in and anon), but I've created a new patch to limit it to anons only as well as to set the sampling rate to 0.1%. You might not be seeing it for anons if you've already clicked one of the buttons on the survey ("Visit survey" or "No thanks") as it won't show again unless you open a new incognito window .

nray added a comment.EditedApr 29 2020, 5:58 PM

Update: I'm planning on swatting the following revisions at 16:00 PDT today:

  1. Setting the sampling rate to 0.1%
  2. Making the survey visible to anons only

https://gerrit.wikimedia.org/r/#/c/operations/mediawiki-config/+/593265/

Change 593265 merged by jenkins-bot:
[operations/mediawiki-config@master] Set Growth Screener Survey sample rate to 0.1% and limit to anon audience

https://gerrit.wikimedia.org/r/593265

Mentioned in SAL (#wikimedia-operations) [2020-04-29T23:37:40Z] <catrope@deploy1001> Synchronized wmf-config/InitialiseSettings.php: Set Growth Screener survey sample rate to 0.1% and limit to anons only (T248421) (duration: 01m 05s)

nray added a comment.EditedApr 30 2020, 12:02 AM

@ovasileva @dchen The swat was a spectacular success story. The changes are live now and the survey will only show to anon users on english wikipedia with a sampling rate of 0.1%.

You can see the survey on english wikipedia again by using the quicksurvey=true query param e.g. (https://en.wikipedia.org/wiki/Barack_Obama?quicksurvey=true) in an incognito window. Please note that by using this query param, it will bypass the audience/sampling rate filter that QuickSurveys uses making it possible to see the survey even when logged in. Don't be alarmed, however, as without this query param only anon users will be able to see the survey.

Also, note that although the AC states that this survey should run for 2 weeks, I don't think QuickSurveys provides any way to automatically turn it off so we'll have to turn it off manually via another swat (which I guess should be around May 13).

nray removed nray as the assignee of this task.Apr 30 2020, 9:24 PM
ovasileva added a comment.EditedMay 1 2020, 8:57 AM

@nray - I think this might just be my browser, but I still can't get the survey to show in Chrome (incognito, with https://en.wikipedia.org/wiki/Barack_Obama?quicksurvey=true). Works fine on Firefox and Safari though

@ovasileva: I'd be interested to see if there's anything showing up in your browser's console. I can confirm that I'm seeing the survey in a fresh incognito window in Google Chrome (81.0.4044.129) on macOS Catalina (10.15.4).

phuedx added a comment.May 1 2020, 1:33 PM

There've been impressions > 57000 [0] and nearly 1000 responses [1] for the survey as of now:

[0]
57108


select
    count(*)
from event.quicksurveyinitiation
where
    year = 2020
    and month = 5
    and day = 1

    and event.surveycodename = 'growth-study-screener-survey'
    and event.eventname = 'impression'
;
[1]
998


select
    count(*)
from event.quicksurveysresponses
where
    year = 2020
    and month = 5
    and day = 1

    and event.surveycodename = 'growth-study-screener-survey'
;

@ovasileva: I'd be interested to see if there's anything showing up in your browser's console. I can confirm that I'm seeing the survey in a fresh incognito window in Google Chrome (81.0.4044.129) on macOS Catalina (10.15.4).

What should I look for? I'm also Chrome (81.0.4044.129) on macOS Catalina (10.15.4).

nray added a comment.May 1 2020, 4:28 PM

@ovasileva Does your Chrome's browser console differ in any way from my screenshot?:

@nray - seems it's DNT although it shouldn't be on in incognito

nray added a comment.May 1 2020, 5:09 PM

👍👍👍 DNT strikes again! Makes sense, I see in the QuickSurveys code we abort early if that is enabled

Change 593939 had a related patch set uploaded (by Phuedx; owner: Phuedx):
[mediawiki/extensions/QuickSurveys@master] Log when the user has enabled Do Not Track

https://gerrit.wikimedia.org/r/593939

phuedx added a comment.May 3 2020, 3:51 AM

☝ Hopefully the above is a super-quick-to-merge QoL improvement. It'll log a message to the web console if the user has enabled DNT.

Change 593939 merged by jenkins-bot:
[mediawiki/extensions/QuickSurveys@master] Log when the user has enabled Do Not Track

https://gerrit.wikimedia.org/r/593939

ovasileva closed this task as Resolved.May 6 2020, 9:56 AM
ovasileva claimed this task.

@dchen - resolving the task as the survey is out and has gone through QA. Feel free to reopen if you see any issues.

Isaac added a subscriber: Isaac.May 7 2020, 8:08 PM

Not sure who to tell this to, but came across this task and I wanted to say that if there are people working on documentation for how to run QuickSurveys, I have some knowledge in my head that I would be glad to contribute. Of note, I wanted to say that for the future, we generally host the privacy policy on foundation.wikimedia.org (example) so that people can read it without worrying about Google (or other) trackers.

dchen reopened this task as Open.Wed, Jul 1, 6:55 PM
dchen raised the priority of this task from High to Needs Triage.
dchen updated the task description. (Show Details)

I'm a little confused as what we're asking now, gjven the original scope was deployment and we don't tend to reopen tasks.

Can we open a new bug with the remaining work?
Note in terms of documentation we are not working on this right now, but https://meta.wikimedia.org/wiki/QuickSurveys#How_can_I_create_a_Quick_Survey and associated talk page would be the place to do it.

dchen added a comment.Tue, Jul 7, 7:46 AM

@Jdlrobson i wasn't sure if reopening or making a new task was the best move... so i did both... here is the other task: https://phabricator.wikimedia.org/T257015

Feel free to move/close as needed, thanks!

ovasileva closed this task as Resolved.Tue, Jul 7, 9:58 AM

Thanks @dchen - resolving this again, and we'll use the new one.