Page MenuHomePhabricator

Large scale QuickSurvey for reader segmentation research
Closed, ResolvedPublic1 Story Points

Description

We want to run a survey very similar to T116433 at a larger scale and with the updated schema and URL parameter we will be passing to the survey.

SurveyDurationStart Date and timeEnd Datearray
Test8 hours2016-02-16 at 16:00 PST2016-02-17 at 16:00 PST[1]
Test no DNT30 min2016-02-25 at 16:00 PST30 min after[2]
Main7 days2016-02-29 at 16:00 PST2016-03-08 08:00 PST[3]

[1]
array(

		"@description"=> "description of the survey",
		"description" => "Reader-segmentation-1-description",

		"@link"=> "external link to the survey (must be https)",
		"link"=> "Reader-segmentation-3prime-link",


		"@question"=> "survey question message key",
		"question"=> "Reader-segmentation-3-message",

		"@privacyPolicy"=> "message key for privacy policy. May contain links.",
		"privacyPolicy"=> "Reader-segmentation-3prime-privacy",

		"@coverage"=> "percentage of users that will see the survey",
		"coverage"=> "0.00005", // 5 out of 100K

		"@platforms"=> "for each platform (desktop, mobile), which version of it is targeted (stable, beta, alpha)",
		"platforms"=> array(
			"desktop"=> ["stable"],
			"mobile"=> ["stable"]
		),

                "instanceTokenParameterName" => "entry.901222064"

)

[2]
array(

		"@description"=> "description of the survey",
		"description" => "Reader-segmentation-1-description",

		"@link"=> "external link to the survey (must be https)",
		"link"=> "Reader-segmentation-3prime-link",


		"@question"=> "survey question message key",
		"question"=> "Reader-segmentation-3-message",

		"@privacyPolicy"=> "message key for privacy policy. May contain links.",
		"privacyPolicy"=> "Reader-segmentation-3prime-privacy",

		"@coverage"=> "percentage of users that will see the survey",
		"coverage"=> "0.002", // 1 out of 500

		"@platforms"=> "for each platform (desktop, mobile), which version of it is targeted (stable, beta, alpha)",
		"platforms"=> array(
			"desktop"=> ["stable"],
			"mobile"=> ["stable"]
		),

                "instanceTokenParameterName" => "entry.901222064"

)

[3] We looked at the past surveys to estimate the correct sampling rate for collecting 50K responses in a week. Based on this, we are recommending a 0.02 coverage value.
array(

		"@description"=> "description of the survey",
		"description" => "Reader-segmentation-1-description",

		"@link"=> "external link to the survey (must be https)",
		"link"=> "Reader-segmentation-3prime-link",


		"@question"=> "survey question message key",
		"question"=> "Reader-segmentation-3-message",

		"@privacyPolicy"=> "message key for privacy policy. May contain links.",
		"privacyPolicy"=> "Reader-segmentation-3prime-privacy",

		"@coverage"=> "percentage of users that will see the survey",
		"coverage"=> "0.02", // 1 out of 50

		"@platforms"=> "for each platform (desktop, mobile), which version of it is targeted (stable, beta, alpha)",
		"platforms"=> array(
			"desktop"=> ["stable"],
			"mobile"=> ["stable"]
		),

                "instanceTokenParameterName" => "entry.901222064"

)

Details

Related Gerrit Patches:
operations/mediawiki-config : masterRemove reader segmentation survey
operations/mediawiki-config : masterChange rate for reader segmentation survey
operations/mediawiki-config : masterRun the survey at lowered rate
operations/mediawiki-config : masterRun the survey at normal rate to test DNT
operations/mediawiki-config : masterRun the survey at normal rate to test DNT
operations/mediawiki-config : masterEnable survey at reduced sample rate
operations/mediawiki-config : masterEnable survey at reduced sample rate

Event Timeline

There are a very large number of changes, so older changes are hidden. Show Older Changes
leila updated the task description. (Show Details)Feb 15 2016, 7:25 PM

Change 270792 had a related patch set uploaded (by Bmansurov):
Run the survey at normal rate

https://gerrit.wikimedia.org/r/270792

Change 270344 merged by jenkins-bot:
Enable survey at reduced sample rate

https://gerrit.wikimedia.org/r/270344

@bmansurov, @jhobs, @Jdlrobson do we have regular coverage to schedule and watch the SWATs at 1600 PST in the Description's schedule? If so, who?

I'm usually available during that time. I think Jon too.

Change 270985 had a related patch set uploaded (by Bmansurov):
Enable survey at reduced sample rate

https://gerrit.wikimedia.org/r/270985

@leila, sorry we missed the train. The patch will be deployed today in the evening window.

leila added a comment.Feb 16 2016, 7:15 PM

thanks for the update, @bmansurov. We have created a QA task for this T127101. In case there is a need to stop the survey fast, we will contact you on this card. If there is no major issue, we will let it run until tomorrow morning. I'm updating the task description on this task. If there is someone specific we can contact for stopping the test in case of an emergency, please let us know.

leila updated the task description. (Show Details)Feb 16 2016, 7:16 PM

@leila, I'm not sure if anyone can stop the survey until the next window. Do you think we should reduce the sampling rate then? So the initial estimate was for 8 hours, and it looks like this will run 16 hours.

leila added a comment.Feb 16 2016, 7:21 PM

@bmansurov the sampling rate won't have that much impact, it's very low at the moment. I guess if there is something seriously wrong, we will know it soon after you deploy the code, in which case we can let you know.

Change 270985 merged by jenkins-bot:
Enable survey at reduced sample rate

https://gerrit.wikimedia.org/r/270985

leila updated the task description. (Show Details)Feb 17 2016, 6:47 AM
leila added a comment.Feb 17 2016, 6:49 AM

@bmansurov we did some QA on the test data, the data looks generally healthy, but we have a missing data point that I need to discuss with Adam about before proceeding to the main survey with high sampling rate. As a result, I changed the description of this task and requested that we allow the test survey (currently on) to continue running until afternoon PST, and also that we do not start the main survey until I get a chance to discuss it in the morning with Adam. Thank you! :-)

OK, I'll move the currently scheduled SWAT deploy to the evening window for now.

bmansurov moved this task from To Do to Doing on the Reading-Web-Sprint-66-Harry is Tired board.

@leila, should I still deploy the change or should I call it off?

@bmansurov, we need few more hours to finalize everything on our end. If we have a time slot for tomorrow at 16:00 PST, let's do it then. If not, tomorrow at 8:00 PST would work, too. Thanks for helping and sorry if it wasn't very smooth.

OK, no problem. I've moved it to tomorrow 16:00 PST.

perfect. thank you. updating the task description.

leila updated the task description. (Show Details)Feb 17 2016, 11:46 PM
leila updated the task description. (Show Details)Feb 18 2016, 10:42 PM

@bmansurov we have run into a problem in QA. I have made the QA task a blocking task for the deployment, updated the deployment schedule in Description, and am requesting that we don't deploy the survey with 1:500 at 16:00 PST today. We will look into the issue and hopefully can solve it soon.

@leila, OK. Please let me know the exact time when you want the change deployed. You don't have to tell me know, but whenever things become clearer.

sounds good. thanks, @bmansurov.

leila added a comment.Feb 24 2016, 9:37 PM

We should plan to increase the sampling rate by 2 folds in this task and run the test for 30-min after T127980 is deployed. We can then roll back the sampling rate to the current rate. This way, we can quickly make a conclusion about the data loss issue. I update the description of this task.

leila updated the task description. (Show Details)Feb 24 2016, 9:38 PM

@leila, how does SWATting tomorrow morning sound? Should I just tell the SWATer to deploy the decrease patch 30mins after the initial patch has been merged?

leila added a comment.Feb 25 2016, 1:02 AM

that would be great, as long as we stop it after 30 min. (I'm not sure stopping should happen in the SWAT deploy time or not).

Change 273169 had a related patch set uploaded (by Bmansurov):
Run the survey at lowered rate

https://gerrit.wikimedia.org/r/273169

bmansurov added a comment.EditedFeb 25 2016, 1:44 AM

Scheduled: https://wikitech.wikimedia.org/w/index.php?title=Deployments&type=revision&diff=327221&oldid=326775

Sorry, the dependency (T127980) is not merged yet. I'll SWAT deploy the change once it's in production.

Change 270792 merged by jenkins-bot:
Run the survey at normal rate to test DNT

https://gerrit.wikimedia.org/r/270792

Change 273261 had a related patch set uploaded (by Bmansurov):
Run the survey at normal rate to test DNT

https://gerrit.wikimedia.org/r/273261

leila added a comment.Feb 25 2016, 5:31 PM

@bmansurov will this be deployed in the 16:00-17:00 PST SWAT deploy time interval today? If so, it would be great if we can do it before 16:30 so we can stop it after 30 min, and inside the SWAT deploy schedule.

@leila, yes, it will be done exactly as you described.

leila moved this task from In Progress to Blocked on the Research board.Feb 25 2016, 5:46 PM

Change 273261 merged by jenkins-bot:
Run the survey at normal rate to test DNT

https://gerrit.wikimedia.org/r/273261

Change 273169 merged by jenkins-bot:
Run the survey at lowered rate

https://gerrit.wikimedia.org/r/273169

leila added a comment.Feb 26 2016, 1:08 AM

@bmansurov You can turn the survey back to the slow rate described in [1] in Description. We will be looking at the data and we will report.

The 48 minute test has been carried out between about 7:20pm and 8:08pm EST. Sorry for the mismatch.

leila added a comment.Feb 26 2016, 1:52 AM

@bmansurov: thank you so much! no worries, longer length could only help! :-)

leila updated the task description. (Show Details)Feb 26 2016, 9:59 PM
leila updated the task description. (Show Details)Feb 29 2016, 7:01 AM

@bmansurov we have finished the QA and we are ready for starting the survey on Monday afternoon at a higher rate. I've updated the task description with more details. Could you help us with this?

Jdlrobson closed this task as Resolved.Feb 29 2016, 5:24 PM
Jdlrobson removed bmansurov as the assignee of this task.

@bd808 just to make sure: the very last part of this task is left. After Baha deploys it, it's done done. :-)

jhobs added a comment.EditedFeb 29 2016, 6:03 PM

@leila We have a LOT of people out today and I'll be gone during the SWAT deploy window, so I'm not sure we'll be able to get to this today. Is this something we could walk @schana through or would you rather wait until tomorrow's window?

Edit: I'll probably be quicker to respond on IRC, but my connection has been poor so far this morning so sorry if there's a delay in communication at all.

leila added a comment.Feb 29 2016, 6:26 PM

@jhobs got you. If @schana has bandwidth to look into this with you today, that'd be great. This can also help us for future deployments.

@jhobs got you. If @schana has bandwidth to look into this with you today, that'd be great. This can also help us for future deployments.

For the record, I believe he's already met with @bmansurov, so he may already know everything he needs to. I'll be around if that's not the case, though. :)

I have met with @bmansurov. I think I know most of what needs to happen aside from two things:

  1. What permissions do I need to be able to clone the repo from gerrit? (I tried git clone ssh://nschaaf@gerrit.wikimedia.org:29418/operations/mediawiki-config, which fails)
  2. What git workflow is followed for making changes?
jhobs added a comment.EditedFeb 29 2016, 7:04 PM

I have met with @bmansurov. I think I know most of what needs to happen aside from two things:

  1. What permissions do I need to be able to clone the repo from gerrit? (I tried git clone ssh://nschaaf@gerrit.wikimedia.org:29418/operations/mediawiki-config, which fails)
  2. What git workflow is followed for making changes?
  1. Hmm... that should work and you shouldn't need any special permissions AFAIK. Maybe try HTTPS?
  2. I generally do the following:
    1. git pull
    2. git checkout -b Txxxx (phab task)
    3. make changes, append/squash to one commit, including task in commit message as "Bug: Txxxxx"
    4. update master and git rebase master (just in case)
    5. git review

Basically, just the standard workflow for any gerrit repo, with a bit more emphasis on rebasing.

  1. Hmm... that should work and you shouldn't need any special permissions AFAIK. Maybe try HTTPS?

Well, I can't seem to pull from previously cloned repos either. Using HTTPS doesn't allow me to push anything.

leila updated the task description. (Show Details)Feb 29 2016, 10:56 PM
greg added a subscriber: greg.Feb 29 2016, 11:00 PM

You can setup an HTTPS password in Gerrit https://gerrit.wikimedia.org/r/#/settings/http-password to let you push over HTTPS.

Change 274005 had a related patch set uploaded (by Nschaaf):
Change rate for reader segmentation survey

https://gerrit.wikimedia.org/r/274005

leila updated the task description. (Show Details)Feb 29 2016, 11:11 PM

Change 274005 merged by jenkins-bot:
Change rate for reader segmentation survey

https://gerrit.wikimedia.org/r/274005

leila reopened this task as Open.Mar 4 2016, 5:08 PM
leila updated the task description. (Show Details)

@schana can you help us with stopping the survey on Monday? I added the stop time and date in Description.

@leila I won't be available at that time for the swat deploy, but I could do the work earlier in the day

Jdlrobson removed a project: QuickSurveys.

Removed QuickSurveys tag as this is a config change and nothing to do with the extension. Removed Patch-For-Review as there seems to be no open patchset on this card.

leila added a comment.Mar 7 2016, 5:42 PM

@bmansurov @jhobs any chance one of you can disable the survey in the 16:00-17:00 PST SWAT deploy time today?

@leila I'm in the same situation as @schana and I believe @bmansurov is out today. If something changes and I'm able to do it, I'll ping you here & on IRC, but right now it's unlikely.

Change 275669 had a related patch set uploaded (by Nschaaf):
Remove reader segmentation survey

https://gerrit.wikimedia.org/r/275669

I've set up for the survey to be taken down in the morning SWAT deploy on the 8th.

Change 275669 merged by jenkins-bot:
Remove reader segmentation survey

https://gerrit.wikimedia.org/r/275669

leila added a comment.Mar 8 2016, 4:32 PM

I'm Resolving this task as we've just stopped the survey and no more action is needed on Research's end. @dr0ptp4kt please resurrect it if you need.

Thanks everyone for the team work for making this happen. :-)

leila closed this task as Resolved.Mar 8 2016, 4:33 PM
leila updated the task description. (Show Details)