Large scale QuickSurvey for reader segmentation research
Closed, ResolvedPublic1 Estimated Story Points
Actions

Assigned To

None

Authored By

	leila
	Feb 5 2016, 7:35 AM

Description

We want to run a survey very similar to T116433 at a larger scale and with the updated schema and URL parameter we will be passing to the survey.

Survey	Duration	Start Date and time	End Date	array
Test	8 hours	2016-02-16 at 16:00 PST	2016-02-17 at 16:00 PST	[1]
Test no DNT	30 min	2016-02-25 at 16:00 PST	30 min after	[2]
Main	7 days	2016-02-29 at 16:00 PST	2016-03-08 08:00 PST	[3]

[1]
array(

		"@description"=> "description of the survey",
		"description" => "Reader-segmentation-1-description",

		"@link"=> "external link to the survey (must be https)",
		"link"=> "Reader-segmentation-3prime-link",


		"@question"=> "survey question message key",
		"question"=> "Reader-segmentation-3-message",

		"@privacyPolicy"=> "message key for privacy policy. May contain links.",
		"privacyPolicy"=> "Reader-segmentation-3prime-privacy",

		"@coverage"=> "percentage of users that will see the survey",
		"coverage"=> "0.00005", // 5 out of 100K

		"@platforms"=> "for each platform (desktop, mobile), which version of it is targeted (stable, beta, alpha)",
		"platforms"=> array(
			"desktop"=> ["stable"],
			"mobile"=> ["stable"]
		),

                "instanceTokenParameterName" => "entry.901222064"

)

[2]
array(

		"@description"=> "description of the survey",
		"description" => "Reader-segmentation-1-description",

		"@link"=> "external link to the survey (must be https)",
		"link"=> "Reader-segmentation-3prime-link",


		"@question"=> "survey question message key",
		"question"=> "Reader-segmentation-3-message",

		"@privacyPolicy"=> "message key for privacy policy. May contain links.",
		"privacyPolicy"=> "Reader-segmentation-3prime-privacy",

		"@coverage"=> "percentage of users that will see the survey",
		"coverage"=> "0.002", // 1 out of 500

		"@platforms"=> "for each platform (desktop, mobile), which version of it is targeted (stable, beta, alpha)",
		"platforms"=> array(
			"desktop"=> ["stable"],
			"mobile"=> ["stable"]
		),

                "instanceTokenParameterName" => "entry.901222064"

)

[3] We looked at the past surveys to estimate the correct sampling rate for collecting 50K responses in a week. Based on this, we are recommending a 0.02 coverage value.
array(

		"@description"=> "description of the survey",
		"description" => "Reader-segmentation-1-description",

		"@link"=> "external link to the survey (must be https)",
		"link"=> "Reader-segmentation-3prime-link",


		"@question"=> "survey question message key",
		"question"=> "Reader-segmentation-3-message",

		"@privacyPolicy"=> "message key for privacy policy. May contain links.",
		"privacyPolicy"=> "Reader-segmentation-3prime-privacy",

		"@coverage"=> "percentage of users that will see the survey",
		"coverage"=> "0.02", // 1 out of 50

		"@platforms"=> "for each platform (desktop, mobile), which version of it is targeted (stable, beta, alpha)",
		"platforms"=> array(
			"desktop"=> ["stable"],
			"mobile"=> ["stable"]
		),

                "instanceTokenParameterName" => "entry.901222064"

)

Details

Project	Branch	Lines +/-	Subject
operations/mediawiki-config	master	+0 -18	Remove reader segmentation survey
operations/mediawiki-config	master	+1 -1	Change rate for reader segmentation survey
operations/mediawiki-config	master	+1 -1	Run the survey at lowered rate
operations/mediawiki-config	master	+1 -1	Run the survey at normal rate to test DNT
operations/mediawiki-config	master	+1 -1	Run the survey at normal rate to test DNT
operations/mediawiki-config	master	+18 -0	Enable survey at reduced sample rate
operations/mediawiki-config	master	+13 -0	Enable survey at reduced sample rate

Customize query in gerrit

Related Objects
Search...

Status	Assigned	Task
Resolved	• madhuvishy	T128408 Clean up Client IP and hashing related code on Eventlogging {oryx} {mole}
Resolved	• madhuvishy	T128407 Remove Client IP from Eventlogging capsule {mole}
Resolved	leila	T121727 [Epic] Reader segmentation research (behavioral analysis)
Resolved	None	T125946 Large scale QuickSurvey for reader segmentation research
Resolved	• schana	T124511 Add URL Parameter to Survey
Resolved	• ellery	T127101 QA Eventlogging to Google Forms set up

Event Timeline

There are a very large number of changes, so older changes are hidden. Show Older Changes

A SWAT deploy has been scheduled: https://wikitech.wikimedia.org/w/index.php?title=Deployments&type=revision&diff=307974&oldid=307973

leila updated the task description. (Show Details)Feb 15 2016, 7:25 PM

Change 270792 had a related patch set uploaded (by Bmansurov):
Run the survey at normal rate

https://gerrit.wikimedia.org/r/270792

A SWAT deploy to change the sampling rate has been scheduled: https://wikitech.wikimedia.org/w/index.php?title=Deployments&type=revision&diff=307997&oldid=307974

bmansurov moved this task from -1 (Needs More Work) to Code Review on the Reading-Web-Sprint-65-Game of Phones board.Feb 15 2016, 7:46 PM

Change 270344 merged by jenkins-bot:
Enable survey at reduced sample rate

https://gerrit.wikimedia.org/r/270344

@bmansurov, @jhobs, @Jdlrobson do we have regular coverage to schedule and watch the SWATs at 1600 PST in the Description's schedule? If so, who?

I'm usually available during that time. I think Jon too.

Change 270985 had a related patch set uploaded (by Bmansurov):
Enable survey at reduced sample rate

https://gerrit.wikimedia.org/r/270985

Jdlrobson removed a project: Reading-Web-Sprint-65-Game of Phones.Feb 16 2016, 5:07 PM

leila mentioned this in T127101: QA Eventlogging to Google Forms set up.Feb 16 2016, 6:54 PM

@leila, sorry we missed the train. The patch will be deployed today in the evening window.

thanks for the update, @bmansurov. We have created a QA task for this T127101. In case there is a need to stop the survey fast, we will contact you on this card. If there is no major issue, we will let it run until tomorrow morning. I'm updating the task description on this task. If there is someone specific we can contact for stopping the test in case of an emergency, please let us know.

leila updated the task description. (Show Details)Feb 16 2016, 7:16 PM

@leila, I'm not sure if anyone can stop the survey until the next window. Do you think we should reduce the sampling rate then? So the initial estimate was for 8 hours, and it looks like this will run 16 hours.

@bmansurov the sampling rate won't have that much impact, it's very low at the moment. I guess if there is something seriously wrong, we will know it soon after you deploy the code, in which case we can let you know.

Change 270985 merged by jenkins-bot:
Enable survey at reduced sample rate

https://gerrit.wikimedia.org/r/270985

Jdlrobson removed a project: Patch-For-Review.Feb 17 2016, 12:44 AM

leila updated the task description. (Show Details)Feb 17 2016, 6:47 AM

@bmansurov we did some QA on the test data, the data looks generally healthy, but we have a missing data point that I need to discuss with Adam about before proceeding to the main survey with high sampling rate. As a result, I changed the description of this task and requested that we allow the test survey (currently on) to continue running until afternoon PST, and also that we do not start the main survey until I get a chance to discuss it in the morning with Adam. Thank you! :-)

OK, I'll move the currently scheduled SWAT deploy to the evening window for now.

bmansurov claimed this task.Feb 17 2016, 2:28 PM

bmansurov moved this task from To Do to Doing on the Reading-Web-Sprint-66-Harry is Tired board.

@leila, should I still deploy the change or should I call it off?

@bmansurov, we need few more hours to finalize everything on our end. If we have a time slot for tomorrow at 16:00 PST, let's do it then. If not, tomorrow at 8:00 PST would work, too. Thanks for helping and sorry if it wasn't very smooth.

OK, no problem. I've moved it to tomorrow 16:00 PST.

perfect. thank you. updating the task description.

leila updated the task description. (Show Details)Feb 17 2016, 11:46 PM

• DarTar added a project: Reader-research.Feb 18 2016, 8:44 PM

leila added a subtask: T127101: QA Eventlogging to Google Forms set up.Feb 18 2016, 10:39 PM

@bmansurov we have run into a problem in QA. I have made the QA task a blocking task for the deployment, updated the deployment schedule in Description, and am requesting that we don't deploy the survey with 1:500 at 16:00 PST today. We will look into the issue and hopefully can solve it soon.

@leila, OK. Please let me know the exact time when you want the change deployed. You don't have to tell me know, but whenever things become clearer.

sounds good. thanks, @bmansurov.

leila mentioned this in T127980: Do not show Quick Surveys to Do Not Track User.Feb 24 2016, 7:28 PM

We should plan to increase the sampling rate by 2 folds in this task and run the test for 30-min after T127980 is deployed. We can then roll back the sampling rate to the current rate. This way, we can quickly make a conclusion about the data loss issue. I update the description of this task.

leila updated the task description. (Show Details)Feb 24 2016, 9:38 PM

@leila, how does SWATting tomorrow morning sound? Should I just tell the SWATer to deploy the decrease patch 30mins after the initial patch has been merged?

that would be great, as long as we stop it after 30 min. (I'm not sure stopping should happen in the SWAT deploy time or not).

Change 273169 had a related patch set uploaded (by Bmansurov):
Run the survey at lowered rate

https://gerrit.wikimedia.org/r/273169

gerritbot added a project: Patch-For-Review.Feb 25 2016, 1:36 AM

Scheduled: https://wikitech.wikimedia.org/w/index.php?title=Deployments&type=revision&diff=327221&oldid=326775

Sorry, the dependency (T127980) is not merged yet. I'll SWAT deploy the change once it's in production.

Change 270792 merged by jenkins-bot:
Run the survey at normal rate to test DNT

https://gerrit.wikimedia.org/r/270792

Change 273261 had a related patch set uploaded (by Bmansurov):
Run the survey at normal rate to test DNT

https://gerrit.wikimedia.org/r/273261

@bmansurov will this be deployed in the 16:00-17:00 PST SWAT deploy time interval today? If so, it would be great if we can do it before 16:30 so we can stop it after 30 min, and inside the SWAT deploy schedule.

@leila, yes, it will be done exactly as you described.

leila moved this task from In Progress to Blocked on the Research board.Feb 25 2016, 5:46 PM

Change 273261 merged by jenkins-bot:
Run the survey at normal rate to test DNT

https://gerrit.wikimedia.org/r/273261

Change 273169 merged by jenkins-bot:
Run the survey at lowered rate

https://gerrit.wikimedia.org/r/273169

@bmansurov You can turn the survey back to the slow rate described in [1] in Description. We will be looking at the data and we will report.

The 48 minute test has been carried out between about 7:20pm and 8:08pm EST. Sorry for the mismatch.

@bmansurov: thank you so much! no worries, longer length could only help! :-)

leila updated the task description. (Show Details)Feb 26 2016, 9:59 PM

Jdlrobson removed a project: Patch-For-Review.Feb 26 2016, 11:04 PM

@bmansurov we have finished the QA and we are ready for starting the survey on Monday afternoon at a higher rate. I've updated the task description with more details. Could you help us with this?

bd808 edited projects, added Reading-Web-Sprint-67-If, Then, Else...?; removed Reading-Web-Sprint-66-Harry is Tired.Feb 29 2016, 5:08 PM

Jdlrobson closed this task as Resolved.Feb 29 2016, 5:24 PM

Jdlrobson removed bmansurov as the assignee of this task.

Jdlrobson moved this task from Needs Analysis to Doing on the Reading-Web-Sprint-67-If, Then, Else...? board.

bd808 moved this task from Doing to Done on the Reading-Web-Sprint-67-If, Then, Else...? board.Feb 29 2016, 5:29 PM

@bd808 just to make sure: the very last part of this task is left. After Baha deploys it, it's done done. :-)

@leila We have a LOT of people out today and I'll be gone during the SWAT deploy window, so I'm not sure we'll be able to get to this today. Is this something we could walk @schana through or would you rather wait until tomorrow's window?

Edit: I'll probably be quicker to respond on IRC, but my connection has been poor so far this morning so sorry if there's a delay in communication at all.

@jhobs got you. If @schana has bandwidth to look into this with you today, that'd be great. This can also help us for future deployments.

In T125946#2072939, @leila wrote:

@jhobs got you. If @schana has bandwidth to look into this with you today, that'd be great. This can also help us for future deployments.

For the record, I believe he's already met with @bmansurov, so he may already know everything he needs to. I'll be around if that's not the case, though. :)

I have met with @bmansurov. I think I know most of what needs to happen aside from two things:

What permissions do I need to be able to clone the repo from gerrit? (I tried git clone ssh://nschaaf@gerrit.wikimedia.org:29418/operations/mediawiki-config, which fails)
What git workflow is followed for making changes?

In T125946#2072979, @schana wrote:

I have met with @bmansurov. I think I know most of what needs to happen aside from two things:

What permissions do I need to be able to clone the repo from gerrit? (I tried git clone ssh://nschaaf@gerrit.wikimedia.org:29418/operations/mediawiki-config, which fails)

What git workflow is followed for making changes?

Hmm... that should work and you shouldn't need any special permissions AFAIK. Maybe try HTTPS?
I generally do the following:
1. git pull
2. git checkout -b Txxxx (phab task)
3. make changes, append/squash to one commit, including task in commit message as "Bug: Txxxxx"
4. update master and git rebase master (just in case)
5. git review

Basically, just the standard workflow for any gerrit repo, with a bit more emphasis on rebasing.

In T125946#2073103, @jhobs wrote:

Hmm... that should work and you shouldn't need any special permissions AFAIK. Maybe try HTTPS?

Well, I can't seem to pull from previously cloned repos either. Using HTTPS doesn't allow me to push anything.

leila added a parent task: T128407: Remove Client IP from Eventlogging capsule {mole}.Feb 29 2016, 10:27 PM

leila mentioned this in T128407: Remove Client IP from Eventlogging capsule {mole}.

leila updated the task description. (Show Details)Feb 29 2016, 10:56 PM

You can setup an HTTPS password in Gerrit https://gerrit.wikimedia.org/r/#/settings/http-password to let you push over HTTPS.

Change 274005 had a related patch set uploaded (by Nschaaf):
Change rate for reader segmentation survey

https://gerrit.wikimedia.org/r/274005

gerritbot added a project: Patch-For-Review.Feb 29 2016, 11:08 PM

leila updated the task description. (Show Details)Feb 29 2016, 11:11 PM

Change 274005 merged by jenkins-bot:
Change rate for reader segmentation survey

https://gerrit.wikimedia.org/r/274005

@schana can you help us with stopping the survey on Monday? I added the stop time and date in Description.

@leila I won't be available at that time for the swat deploy, but I could do the work earlier in the day

Removed QuickSurveys tag as this is a config change and nothing to do with the extension. Removed Patch-For-Review as there seems to be no open patchset on this card.

@bmansurov @jhobs any chance one of you can disable the survey in the 16:00-17:00 PST SWAT deploy time today?

@leila I'm in the same situation as @schana and I believe @bmansurov is out today. If something changes and I'm able to do it, I'll ping you here & on IRC, but right now it's unlikely.

Change 275669 had a related patch set uploaded (by Nschaaf):
Remove reader segmentation survey

https://gerrit.wikimedia.org/r/275669

gerritbot added a project: Patch-For-Review.Mar 7 2016, 9:34 PM

I've set up for the survey to be taken down in the morning SWAT deploy on the 8th.

thanks, @schana.

Change 275669 merged by jenkins-bot:
Remove reader segmentation survey

https://gerrit.wikimedia.org/r/275669

I'm Resolving this task as we've just stopped the survey and no more action is needed on Research's end. @dr0ptp4kt please resurrect it if you need.

Thanks everyone for the team work for making this happen. :-)

leila closed this task as Resolved.Mar 8 2016, 4:33 PM

leila updated the task description. (Show Details)

leila mentioned this in T131949: Repeat the big English reader survey in one or two more languages .Apr 6 2016, 4:25 PM

• ggellerman closed subtask T127101: QA Eventlogging to Google Forms set up as Resolved.Apr 29 2016, 10:14 PM

	F3331980: Screen Shot 2016-02-11 at 04.16.51.png
	Feb 11 2016, 4:21 AM

Large scale QuickSurvey for reader segmentation researchClosed, ResolvedPublic1 Estimated Story PointsActions

Description

Details

Related ObjectsSearch...

Event Timeline

Large scale QuickSurvey for reader segmentation research
Closed, ResolvedPublic1 Estimated Story Points
Actions

Related Objects
Search...