Page MenuHomePhabricator

Run Reader Survey 1 in Persian (mobile + desktop combined)
Closed, ResolvedPublic1 Estimated Story Points

Description

  • Deploy QuickSurveys to Persian Wikipedia
  • Deploy config change to enable survey ----

Repeat T113443 in Persian or Japanese. Platform should be changed to Mobile + Desktop, instead of Desktop only as in the first survey.

array(
		"@description"=> "description of the survey",
		"description" => "Reader-segmentation-1-description",

		"@link"=> "external link to the survey (must be https)",
		"link"=> "Reader-segmentation-1-link",


		"@question"=> "survey question message key",
		"question"=> "Reader-segmentation-1-message",

		"@privacyPolicy"=> "message key for privacy policy. May contain links.",
		"privacyPolicy"=> "Reader-segmentation-1-privacy",

		"@coverage"=> "percentage of users that will see the survey",
		"coverage"=> "0.007519", // 1 out of 133

		"@platform"=> "for each platform (desktop, mobile), which version of it is targeted (stable, beta, alpha)",
		"platform"=> array(
			"desktop"=> ["stable"],
			"mobile"=> ["stable"]
		)
	)

Event Timeline

leila claimed this task.
leila raised the priority of this task from to Needs Triage.
leila updated the task description. (Show Details)
leila added subscribers: Jdlrobson, Cervisiarius, Aklapper and 4 others.
leila triaged this task as Medium priority.Jan 15 2016, 11:58 PM
leila set Security to None.

Does this survey need to be run separately on mobile and desktop? If so can you add T119367 as a blocking task (currently platform is still not working)

It doesn't need to run separately, @Jdlrobson. I'll update the description and title.

leila renamed this task from Repeat the first reader survey in Persian or Japanese to Repeat the first and second reader survey in Persian or Japanese, combined.Jan 20 2016, 7:21 PM
leila updated the task description. (Show Details)

@dr0ptp4kt is it possible to run this survey from 2016-01-28 to 2016-02-01? We have the translations ready, and we don't need extra Legal and Security approval as this is no different than what we did last quarter.

leila renamed this task from Repeat the first and second reader survey in Persian or Japanese, combined to Repeat the first and second reader survey in Persian, combined.Jan 22 2016, 6:45 PM
ellery renamed this task from Repeat the first and second reader survey in Persian, combined to Run Reader Survey 1 in Persian (mobile + desktop combined).Jan 22 2016, 6:47 PM
ellery moved this task from Backlog to Staged (ready to execute) on the Research board.

Just to make sure, funnel analysis is not important for this survey.

I asked to have this landed next in the backlog.

Jdlrobson updated the task description. (Show Details)

@dr0ptp4kt the keys should still be created in https://en.wikipedia.org/wiki/MediaWiki even though these surveys will run in eswiki and fawiki, right?

@leila - just to be sure, I'll redirect that question :) - @jhobs, would you please confirm where the keys should be created?

@dr0ptp4kt the keys should still be created in https://en.wikipedia.org/wiki/MediaWiki even though these surveys will run in eswiki and fawiki, right?

The keys should be created within the "Mediawiki" namespace of each individual wiki on which you want to run it. If possible, try to use the same key names across all wikis in order to reduce config changes.

E.g. key "Reader-research-survey-title" would require the following pages to run on enwiki, eswiki, and fawiki:
en.wikipedia.org/wiki/MediaWiki:Reader-research-survey-title
es.wikipedia.org/wiki/MediaWiki:Reader-research-survey-title
fa.wikipedia.org/wiki/MediaWiki:Reader-research-survey-title

With only these pages defined, it could not, for example, run on the french wiki.

@ellery when you are computing the coverage for eswiki, can you also compute it for fawiki and fill out the task Description here? Thanks!

This ticket is not finalized yet. Please do not start the survey yet. :-)

@dr0ptp4kt this task is ready to be claimed by your team.

@leila, just to double check. This card is ready to be worked on by us and SWAT deployed? I'm thinking of SWAT deploying it tomorrow? cc @jhobs

@bmansurov, yes! thanks for checking. SWAT deploy today or tomorrow is fine. :-)

@bmansurov just let me know when you want to do it, it may be a good idea to test it to make sure everything works if possible.

Change 266955 had a related patch set uploaded (by Bmansurov):
Enable QuickSurveys in fawiki

https://gerrit.wikimedia.org/r/266955

@bmansurov I did (2). Where does survey name go? This is new for me.

@leila, thanks. I'm not sure if the name appears in the front end. It's just one of the keys in the config variable and I didn't want the survey to fail because of the missing name.

@bmansurov I'm hoping the user doesn't see item (1). :D I created it for you though: Reader-segmentation-1-name

The patch will be SWAT deployed during the 08:00–09:00 PST window today.

Change 266955 merged by jenkins-bot:
Enable QuickSurveys on fawiki

https://gerrit.wikimedia.org/r/266955

Change 267040 had a related patch set uploaded (by Bmansurov):
Enable QuickSurveys on fawiki

https://gerrit.wikimedia.org/r/267040

Change 267040 merged by jenkins-bot:
Enable QuickSurveys on fawiki

https://gerrit.wikimedia.org/r/267040

@bmansurov: Why is the survey shown under the info box in fawiki?

I'm not sure we'll have to investigate. I'll create a ticket. Here it is: T125116

@bmansurov we are two orders of magnitude off in our sampling. We had this issue in the previous surveys. I /think/ what's happening is that the way EL is sampling is multiplying whatever number we provide by 0.01. Can you multiply the sampling rate by 100?

Change 267071 had a related patch set uploaded (by Bmansurov):
Bump up the QuickSurveys sampling rates for es and fa wikis

https://gerrit.wikimedia.org/r/267071

The changes to multiply the sampling rate by 100 will be SWAT deployed during this evening window.

@bmansurov If you get some extra time, would you like to try looking into why that keeps happening? Last time I looked into it all the code pointed to working as it should be, so I'm at a loss.

thanks @bmansurov.

@jhobs if @ellery and/or I can be of any help debugging it, please ping us as well.

Nothing seems to be wrong with sampling. I ran the following code multiple times and I got similar results each time (around 100 for bucket A):

var i, token, countA = 0, countControl = 0, bucket,
	buckets = {
		control: .99,
		A: .01
	};
for (i = 0; i < 10000; i++) {
	token = mw.user.generateRandomSessionId();
	bucket = mw.experiments.getBucket( {
		name: 'reader-segmentation',
		enabled: true,
		buckets: buckets
	}, token );
	if ( bucket === 'A' ) {
		countA += 1;
	} else if ( bucket === 'control' ) {
		countControl += 1;
	}
} 
console.log( 'count A = ' + countA + ', count Control = ' + countControl );

One theory is that users have set DoNotTrack setting to true so that we don't even know if a survey is even shown because EventLogging is not recording it.

Be careful comparing previous survey results on enwiki with this one.

It's worth noting readership is much smaller on this wiki, so the same sampling rate as enwiki will yield less results.
Also it's a completely different context... Maybe surveys don't perform so well in this culture/language... we haven't done any user tests around QuickSurveys yet.

@bmansurov can you ping me if you have some time to chat over Hangout? It may be easier to discuss it there.

@Jdlrobson we have corrected the sampling rates for eswiki and fawiki. For example, in fawiki, we requested for 1:133 requests sampling rate while in enwiki we have 5 out of 100K. culture/language may play a role though.

@bmansurov is 'coverage' in https://gerrit.wikimedia.org/r/#/c/267071/1/wmf-config/InitialiseSettings.php equal to A in your example? I can't find the part of the code that receives coverage as an input.

@bmansurov can you ping me if you have some time to chat over Hangout? It may be easier to discuss it there.

Which channel are you on? I'm on #wikimedia-mobile and online right now.

We discussed this in the channel. Based on
https://github.com/wikimedia/mediawiki-extensions-QuickSurveys/blob/master/resources/ext.quicksurveys.lib/lib.js#L223
and
https://github.com/wikimedia/mediawiki-extensions-QuickSurveys/blob/master/resources/ext.quicksurveys.lib/lib.js#L156
it seems that the code is handling coveragevariable correctly. Research's estimates are off and we need to figure out what the problem is. Until then, multiplying everything by 100 will take care of the issue for now.

Change 267071 merged by jenkins-bot:
Bump up the QuickSurveys sampling rates for es and fa wikis

https://gerrit.wikimedia.org/r/267071

@bmansurov, we have enough responses for this survey. Could you stop it? I'm sure you know this already but just in case I checked with @greg and he approves stopping the survey on Friday (not starting one :-)

Change 267282 had a related patch set uploaded (by Bmansurov):
Stop the first survey in fawiki and eswiki

https://gerrit.wikimedia.org/r/267282

This is OK for today, but in general, let's try to time these things to not happen on Fridays. I'm not always around in a "who wants to deploy something, ping me on IRC" capacity on Friday; I have other work to do as well. :)

@greg understood. The plan was to run it until Monday, but I see a lot of responses, and our predictions didn't expect that. I figured if it's easy we should exhaust fewer people. But point taken, we will be more careful in the future.

Change 267282 merged by jenkins-bot:
Stop the first survey in fawiki and eswiki

https://gerrit.wikimedia.org/r/267282