Quicksurvey for reader trust
Open, HighPublic

Description

SurveyDurationStart Date and timeEnd Date
reader-trust-en-v148 hoursnot schedulednot scheduled

[1] We looked at the past surveys to estimate the correct sampling rate for collecting 1000 responses in 48 hours. Based on this, we are recommending a 0.002 coverage value.

array(

                "name": "reader trust survey en v1",
                "type": "external", 
		"@description"=> "description of the survey",
		"description" => "Reader-trust-1-description",

		"@link"=> "external link to the survey (must be https)",
		"link"=> "Reader-trust-1-link",

		"@question"=> "survey question message key",
		"question"=> "Reader-trust-1-message",

		"@privacyPolicy"=> "message key for privacy policy. May contain links.",
		"privacyPolicy"=> "Reader-trust-1-privacy",

		"@coverage"=> "percentage of users that will see the survey",
		"coverage"=> "0.002", // 1 out of 500

	        "@instanceTokenParameterName": "parameter to add to link",
                "instanceTokenParameterName": "token",

		"@platforms"=> "for each platform (desktop, mobile), which version of it is targeted (stable, beta, alpha)",
		"platforms"=> array(
			"desktop"=> ["stable"],
			"mobile"=> ["stable"]
		),

)

Capt_Swing updated the task description. (Show Details)Mon, Nov 19, 9:14 PM
Capt_Swing updated the task description. (Show Details)Mon, Nov 19, 9:17 PM
leila added a subscriber: leila.Tue, Nov 20, 12:33 AM

@Capt_Swing following up on our etherpad discussion:

  • I did one pass over the description of this task and it looks good to me.
  • Re sampling rate estimation: it's a tough one and it's roughly inline with what I would estimate, though my current estimate is that you will get less than 1K in 48 hours. :) When we had a sampling rate of 1:40 in enwiki (mobile and desktop) for a week, we got ~28K responses. If you sample at a sampling rate which is an order of magnitude less, I expect you to get no more than ~3K responses in a week. and that may put you closer to 800 responses in 48 hours. If you can tolerate that margin of difference, the current sampling rate you have is good. If not, you may want to change it.
leila triaged this task as High priority.Tue, Nov 20, 12:33 AM

@leila Thanks! A smaller than 1k sample is probably better. The primary purpose of this round of survey is to gather enough free text responses to create a taxonomy of reasons people trust Wikipedia articles (and/or don't). Secondary purpose is to get a very rough distribution of responses. Second round of survey will use the reason taxonomy, and pull a larger sample.

Capt_Swing updated the task description. (Show Details)Tue, Nov 20, 10:57 PM
Capt_Swing updated the task description. (Show Details)Tue, Nov 20, 11:37 PM

Change 475105 had a related patch set uploaded (by Bmansurov; owner: Bmansurov):
[operations/mediawiki-config@master] Labs: enable the reader trust survey

https://gerrit.wikimedia.org/r/475105

@Capt_Swing in T209882#4765984 I've created a patch to load the survey on https://en.wikipedia.beta.wmflabs.org/. We'll need to load up the messages there now. Could you also ask a permission to beta labs while you're asking one for the production Wikipedia?

Capt_Swing updated the task description. (Show Details)Mon, Nov 26, 7:23 PM

Change 475105 merged by jenkins-bot:
[operations/mediawiki-config@master] Labs: enable the reader trust survey

https://gerrit.wikimedia.org/r/475105

Mentioned in SAL (#wikimedia-operations) [2018-11-27T00:05:06Z] <niharika29@deploy1001> Synchronized wmf-config/InitialiseSettings-labs.php: Enable trust survey on labs T209882 (duration: 00m 46s)

@bmansurov I was never able to get the QuickSurvey to load on this page or on any other I tried. Were you able to preview it?

Change 476027 had a related patch set uploaded (by Bmansurov; owner: Bmansurov):
[operations/mediawiki-config@master] Labs: display reader trust survey

https://gerrit.wikimedia.org/r/476027

@Capt_Swing yes, I'm looking into the issue. I'll deploy a fix today and will let you know.

Change 476027 merged by jenkins-bot:
[operations/mediawiki-config@master] Labs: display reader trust survey

https://gerrit.wikimedia.org/r/476027

Change 476368 had a related patch set uploaded (by Bmansurov; owner: Bmansurov):
[operations/mediawiki-config@master] Enable reader trust survey

https://gerrit.wikimedia.org/r/476368

Change 476370 had a related patch set uploaded (by Bmansurov; owner: Bmansurov):
[operations/mediawiki-config@master] Disable reader trust survey

https://gerrit.wikimedia.org/r/476370

Capt_Swing updated the task description. (Show Details)Wed, Nov 28, 7:42 PM

Updated the task description to reflect new deployment window.

Update: We tested it on beta labs on desktop and the survey seemed to work. Couldn't get the survey to show up on mobile and created T210646: Quick survey won't show up on mobile.

Capt_Swing updated the task description. (Show Details)Wed, Nov 28, 7:55 PM
Capt_Swing moved this task from Staged to In Progress on the Research board.Wed, Nov 28, 8:01 PM
Jdlrobson added a subscriber: Jdlrobson.

Hey all, can I please request a ping in phabricator whenever we enable QuickSurveys in future? They cause a large regression in performance on mobile which is scary to me and my team if we don't know the cause (and the fact its temporary).

@Jdlrobson I don't mind pinging you specifically, but is that the best option? Are there any other stakeholders who need to be alerted about mobile performance changes? Is there any better way to get that message out?

BTW we have decided to hold off on deploying QuickSurvey to production until mid-Jan.

Capt_Swing updated the task description. (Show Details)Mon, Dec 3, 5:02 PM

Holding off on deployment until mid-Jan. We decided that deploying during fundraising would be disruptive and counterproductive, given the competing CTAs and the severely constrained screen real estate.

bmansurov added a comment.EditedMon, Dec 3, 6:56 PM

@Jdlrobson I reported this in SoS last week: https://etherpad.wikimedia.org/p/Scrum-of-Scrums. I'm going to report again the next time we deploy it.

We are observing some interesting behavior on beta wikidata, both in desktop and mobile view.
A <div class="ext-qs-loader-bar mw-ajax-loader"></div> gets attached to what could be considered "the top of the page" in a traditional wiki - maybe the wikidata use case was not taken into consideration? What was the desired behavior in that regard?

I attached screenshots as the gray box does not to show consistently for everyone (maybe due to some A/B logic).

@Pablo-WMDE I think it's a case of messages not being created in the MediaWiki namespace. I probably should deploy QS only on enwiki on labs, if that's causing issues with your users. Please let me know.

@bmansurov Given the gray box appears right in the heart of a feature under active development, about to hit QA, I'd appreciate if it wasn't there.
I can't tell if solving this via configuration or by adding a check in the code (not to show on .wb-entitypage which generally don't seem to be compatible with it) is the better approach.
The latter would certainly act as a safety feature in case of accidental mal-configuration, too.

OK, Iʼll disable it on WD later today.

Change 478028 had a related patch set uploaded (by Bmansurov; owner: Bmansurov):
[operations/mediawiki-config@master] Labs: enable reader trust survey on enwiki only

https://gerrit.wikimedia.org/r/478028

I'm deploying the above patch in about two hours from now.

Change 478028 merged by jenkins-bot:
[operations/mediawiki-config@master] Labs: enable reader trust survey on enwiki only

https://gerrit.wikimedia.org/r/478028

@Pablo-WMDE, I've disabled the survey on Wikidata. Sorry for the trouble.

Capt_Swing moved this task from In Progress to Staged on the Research board.Mon, Dec 10, 6:04 PM