Does this survey need to be run separately on mobile and desktop? If so can you add T119367 as a blocking task (currently platform is still not working)

It doesn't need to run separately, @Jdlrobson. I'll update the description and title.

@dr0ptp4kt is it possible to run this survey from 2016-01-28 to 2016-02-01? We have the translations ready, and we don't need extra Legal and Security approval as this is no different than what we did last quarter.

Just to make sure, funnel analysis is not important for this survey.

I asked to have this landed next in the backlog.

@dr0ptp4kt the keys should still be created in https://en.wikipedia.org/wiki/MediaWiki even though these surveys will run in eswiki and fawiki, right?

@leila - just to be sure, I'll redirect that question :) - @jhobs, would you please confirm where the keys should be created?

In T123771#1962833, @leila wrote:

@dr0ptp4kt the keys should still be created in https://en.wikipedia.org/wiki/MediaWiki even though these surveys will run in eswiki and fawiki, right?

The keys should be created within the "Mediawiki" namespace of each individual wiki on which you want to run it. If possible, try to use the same key names across all wikis in order to reduce config changes.

E.g. key "Reader-research-survey-title" would require the following pages to run on enwiki, eswiki, and fawiki:
en.wikipedia.org/wiki/MediaWiki:Reader-research-survey-title
es.wikipedia.org/wiki/MediaWiki:Reader-research-survey-title
fa.wikipedia.org/wiki/MediaWiki:Reader-research-survey-title

With only these pages defined, it could not, for example, run on the french wiki.

Thanks, @jhobs and @dr0ptp4kt, will do.

@ellery when you are computing the coverage for eswiki, can you also compute it for fawiki and fill out the task Description here? Thanks!

This ticket is not finalized yet. Please do not start the survey yet. :-)

@dr0ptp4kt this task is ready to be claimed by your team.

@leila, just to double check. This card is ready to be worked on by us and SWAT deployed? I'm thinking of SWAT deploying it tomorrow? cc @jhobs

@bmansurov, yes! thanks for checking. SWAT deploy today or tomorrow is fine. :-)

@bmansurov just let me know when you want to do it, it may be a good idea to test it to make sure everything works if possible.

Change 266955 had a related patch set uploaded (by Bmansurov):
Enable QuickSurveys in fawiki

https://gerrit.wikimedia.org/r/266955

@leila, can you also create the following pages?

1. Survey name goes here: https://fa.wikipedia.org/wiki/%D9%85%D8%AF%DB%8C%D8%A7%D9%88%DB%8C%DA%A9%DB%8C:Reader-segmentation-1-name
2. Survey link (that's in the description) goes here: https://fa.wikipedia.org/wiki/%D9%85%D8%AF%DB%8C%D8%A7%D9%88%DB%8C%DA%A9%DB%8C:Reader-segmentation-1-link

@bmansurov I did (2). Where does survey name go? This is new for me.

@leila, thanks. I'm not sure if the name appears in the front end. It's just one of the keys in the config variable and I didn't want the survey to fail because of the missing name.

@bmansurov I'm hoping the user doesn't see item (1). :D I created it for you though: Reader-segmentation-1-name

Thanks, I don't think they will.

The patch will be SWAT deployed during the 08:00–09:00 PST window today.

Change 266955 merged by jenkins-bot:
Enable QuickSurveys on fawiki

https://gerrit.wikimedia.org/r/266955

Change 267040 had a related patch set uploaded (by Bmansurov):
Enable QuickSurveys on fawiki

https://gerrit.wikimedia.org/r/267040

Change 267040 merged by jenkins-bot:
Enable QuickSurveys on fawiki

https://gerrit.wikimedia.org/r/267040

@leila you can test the survey here: https://fa.wikipedia.org/wiki/%D8%B4%DB%8C%D9%86%D8%AF%D9%86%D8%AF?quicksurvey=true

@bmansurov: Why is the survey shown under the info box in fawiki?

I'm not sure we'll have to investigate. I'll create a ticket. Here it is: T125116

@bmansurov we are two orders of magnitude off in our sampling. We had this issue in the previous surveys. I /think/ what's happening is that the way EL is sampling is multiplying whatever number we provide by 0.01. Can you multiply the sampling rate by 100?

Change 267071 had a related patch set uploaded (by Bmansurov):
Bump up the QuickSurveys sampling rates for es and fa wikis

https://gerrit.wikimedia.org/r/267071

The changes to multiply the sampling rate by 100 will be SWAT deployed during this evening window.

@bmansurov If you get some extra time, would you like to try looking into why that keeps happening? Last time I looked into it all the code pointed to working as it should be, so I'm at a loss.

thanks @bmansurov.

@jhobs if @ellery and/or I can be of any help debugging it, please ping us as well.

Nothing seems to be wrong with sampling. I ran the following code multiple times and I got similar results each time (around 100 for bucket A):

var i, token, countA = 0, countControl = 0, bucket,
	buckets = {
		control: .99,
		A: .01
	};
for (i = 0; i < 10000; i++) {
	token = mw.user.generateRandomSessionId();
	bucket = mw.experiments.getBucket( {
		name: 'reader-segmentation',
		enabled: true,
		buckets: buckets
	}, token );
	if ( bucket === 'A' ) {
		countA += 1;
	} else if ( bucket === 'control' ) {
		countControl += 1;
	}
} 
console.log( 'count A = ' + countA + ', count Control = ' + countControl );

One theory is that users have set DoNotTrack setting to true so that we don't even know if a survey is even shown because EventLogging is not recording it.

Be careful comparing previous survey results on enwiki with this one.

It's worth noting readership is much smaller on this wiki, so the same sampling rate as enwiki will yield less results.
Also it's a completely different context... Maybe surveys don't perform so well in this culture/language... we haven't done any user tests around QuickSurveys yet.

@bmansurov can you ping me if you have some time to chat over Hangout? It may be easier to discuss it there.

@Jdlrobson we have corrected the sampling rates for eswiki and fawiki. For example, in fawiki, we requested for 1:133 requests sampling rate while in enwiki we have 5 out of 100K. culture/language may play a role though.

@bmansurov is 'coverage' in https://gerrit.wikimedia.org/r/#/c/267071/1/wmf-config/InitialiseSettings.php equal to A in your example? I can't find the part of the code that receives coverage as an input.

@leila, yes the coverage is A. Here is where it's being used: https://github.com/wikimedia/mediawiki-extensions-QuickSurveys/blob/master/resources/ext.quicksurveys.lib/lib.js#L209

@bmansurov can you ping me if you have some time to chat over Hangout? It may be easier to discuss it there.

Which channel are you on? I'm on #wikimedia-mobile and online right now.

We discussed this in the channel. Based on
https://github.com/wikimedia/mediawiki-extensions-QuickSurveys/blob/master/resources/ext.quicksurveys.lib/lib.js#L223
and
https://github.com/wikimedia/mediawiki-extensions-QuickSurveys/blob/master/resources/ext.quicksurveys.lib/lib.js#L156
it seems that the code is handling coveragevariable correctly. Research's estimates are off and we need to figure out what the problem is. Until then, multiplying everything by 100 will take care of the issue for now.

Change 267071 merged by jenkins-bot:
Bump up the QuickSurveys sampling rates for es and fa wikis

https://gerrit.wikimedia.org/r/267071

@bmansurov, we have enough responses for this survey. Could you stop it? I'm sure you know this already but just in case I checked with @greg and he approves stopping the survey on Friday (not starting one :-)

ok

Change 267282 had a related patch set uploaded (by Bmansurov):
Stop the first survey in fawiki and eswiki

https://gerrit.wikimedia.org/r/267282

This is OK for today, but in general, let's try to time these things to not happen on Fridays. I'm not always around in a "who wants to deploy something, ping me on IRC" capacity on Friday; I have other work to do as well. :)

@greg understood. The plan was to run it until Monday, but I see a lot of responses, and our predictions didn't expect that. I figured if it's easy we should exhaust fewer people. But point taken, we will be more careful in the future.

Change 267282 merged by jenkins-bot:
Stop the first survey in fawiki and eswiki

https://gerrit.wikimedia.org/r/267282