Page MenuHomePhabricator

Second QuickSurvey for reader segmentation research - mobile
Closed, ResolvedPublic

Description

Array values correspond to page names within the Mediawiki namespace: e.g. "Reader-segmentation-1-description" corresponds to Mediawiki:Reader-segmentation-1-description

Please include the actual value of a message as a comment after the key: e.g. // Please answer our survey

array(
    "@description"=> "message key for description of the survey",
    "description" => "Reader-segmentation-1-description", // (blank)

    "@link"=> "message key for external link to the survey (must be https)",
    "link"=> "Reader-segmentation-1-link", // https://docs.google.com/forms/d/17vKKxhVF4A3iO71JNQ4xbvc42Md3U87WgnYwmXISqoM/viewform

    "@question"=> "survey question message key",
    "question"=> "Reader-segmentation-1-message", // Answer one question and help us improve Wikipedia.

    "@privacyPolicy"=> "message key for privacy policy. May contain links.",
    "privacyPolicy"=> "Reader-segmentation-1-privacy", // Survey data handled by a third party. [https://wikimediafoundation.org/wiki/Survey_Privacy_Statement Privacy].


    "@coverage"=> "percentage of users that will see the survey/100",
    "coverage"=> "0.0005", // .05%

    "@platform"=> "for each platform (desktop, mobile), which version of it is targeted (stable, beta, alpha)",
    "platform"=> array(
        "desktop"=> array(),
        "mobile"=> array( "stable" )
    )
)

Please copy this description for future surveys and modify the appropriate values (message keys) and comments (messages).

Event Timeline

JKatzWMF created this task.Nov 17 2015, 6:16 PM
JKatzWMF raised the priority of this task from to Needs Triage.
JKatzWMF updated the task description. (Show Details)
JKatzWMF added subscribers: JKatzWMF, leila.
Restricted Application added subscribers: StudiesWorld, Aklapper. · View Herald TranscriptNov 17 2015, 6:16 PM
leila updated the task description. (Show Details)Nov 18 2015, 4:43 PM
leila set Security to None.
leila added a comment.Nov 18 2015, 4:48 PM

@JKatzWMF et al., please let me know if we are missing something we should provide. If we can start the survey on Thursday morning, that'd be great. There is a slight chance we need to stop the survey on Thursday afternoon. it would be great if we can be ready for that. If things go as expected in the first hours after the test is live on Thursday, we will leave the test run until Monday morning.

@leila what's different about this survey besides targeting mobile? If it's the link, has the enwiki key been updated with the new link? If it's nothing, are we just running the exact same survey again, but this time on mobile?

@jhobs: we're running the exact same survey on mobile, and with a different sampling rate. I expect all the keys to stay the same.

@leila @atgo owns the answers to these questions -- I was just creating the ticket. Sorry I can't be more helpful, but it is definitely for the best :)

Working on the answers. @jhobs is checking if we can do deployments at all in this window - apparently it's been marked as "high priority SWATs only"

If you look at this page https://wikitech.wikimedia.org/wiki/Deployments#End_of_year.2Fbeginning_of_new_year_.27code_freezes.27, it looks like if we tried to run a survey tomorrow, we wouldn't be able to bring it down until Monday the 30th... I'm trying to get clarity on what qualifies as a "high priority SWAT," but I highly doubt this does.

atgo added a subscriber: greg.Nov 19 2015, 12:06 AM

I'll follow up with Megan Hernandez (off phab) about this timing with the
campaign and get back to you. Maybe @greg can weigh in on the deployment
schedule?

For loop-closedness, I've emailed @MeganHernandez_WMF and @greg and asked them to follow up here:

Hey team-

Reading/Research would like to run another round of quicksurvey before the campaign goes live if possible. It requires a SWAT deploy to go up and down.

There's discussion in this ticket: https://phabricator.wikimedia.org/T118881

2 questions:
- Can we take it down during the end of next week (thanksgiving) with someone who's not US based?
- Would taking it down Monday 11/30 be OK from a campaign perspective?

If neither of these are possible, we'll table it. Could you follow up on the phab task?

Thanks,
Anne
greg added a subscriber: K4-713.Nov 19 2015, 6:20 PM

2 questions:

  • Can we take it down during the end of next week (thanksgiving) with someone who's not US based?

Can you do it on Wednesday? That's mostly OK with me (turning something off is less risky than turning it on, usually).
@K4-713: that ok with you?

  • Would taking it down Monday 11/30 be OK from a campaign perspective?

This is an even better solution in my mind :)

Hi, what's the chance there could be a problem with the deployment once the survey ends? Could we end the survey on Wednesday before Thanksgiving so that we're not dealing with any potential issues with the deployment the day before the fundraiser launch? Or when people are traveling? That would be great!

We will be running full traffic tests and banners at a low level while the survey is set to run. I understand the percentage of people who will see the survey is very very low the overlap will be minimal. In general, I think it could be better not to run more notices/call to actions before a big campaign. In this case, it sounds like the traffic will be so low it should not have an impact on the fundraiser.

I would check in with Comms to give them a heads up in case they get questions about the survey and because they are aiming to run a banner next Tuesday. Thanks!

leila added a comment.Nov 19 2015, 7:27 PM

@MeganHernandez_WMF, @jhobs @atgo

To help us move forward: Research needs the survey to run until Monday, Nov. 23. We won't need more data to run it another extra two days.

Change 254322 had a related patch set uploaded (by Jhobs):
Enable new QuickSurveys survey on mobile enwiki

https://gerrit.wikimedia.org/r/254322

leila assigned this task to jhobs.Nov 19 2015, 11:39 PM

Change 254322 merged by jenkins-bot:
Enable new QuickSurveys survey on mobile enwiki

https://gerrit.wikimedia.org/r/254322

The survey is now live. I'll schedule a deployment for the 23rd to take it down. Would you prefer I leave it through the day on Monday and do an evening SWAT, or disable it during the monring?

jhobs updated the task description. (Show Details)Nov 20 2015, 1:44 AM

Let's start using the same config in our description that we'll use in the code to avoid hiccups like today's sampling rate issue. I've updated the description to reflect this.

leila added a comment.Nov 20 2015, 5:13 AM

@jhobs thanks for taking care of this today. Disabling the survey on Monday morning would work well. Thanks!

leila renamed this task from Deploy new survey for research to Second QuickSurvey for reader segmentation research - mobile.Nov 20 2015, 5:14 AM

Change 254831 had a related patch set uploaded (by Jhobs):
End first mobile QuickSurveys campaign

https://gerrit.wikimedia.org/r/254831

^ is scheduled for this morning's SWAT at 8am Pacific.

Change 254831 merged by jenkins-bot:
End first mobile QuickSurveys campaign

https://gerrit.wikimedia.org/r/254831

Survey has been disabled (config completely removed).

leila added a comment.Nov 23 2015, 4:44 PM

Thanks a lot, @jhobs! :-)

leila moved this task from Staged to Done (current quarter) on the Research board.
phuedx closed this task as Resolved.Nov 23 2015, 5:41 PM
phuedx added a subscriber: phuedx.

wgQuickSurveysEnabledSurveys was confirmed empty/falsy by @jhobs and I and there didn't appear to be anything in the logs – I checked fatalmonitor.