Page MenuHomePhabricator

Deploy/Undeploy Quicksurvey for reader trust
Closed, ResolvedPublic

Description

SurveyDurationStart Date and timeEnd Date
reader-trust-en-v1~48 hours2019/1/7 @ 1230 UTC (730 EST)2019/1/9 @ around the same time

[1] We looked at the past surveys to estimate the correct sampling rate for collecting 1000 responses in 48 hours. Based on this, we are recommending a 0.002 coverage value.

array(

                "name": "reader trust survey en v1",
                "type": "external", 
		"@description"=> "description of the survey",
		"description" => "Reader-trust-1-description",

		"@link"=> "external link to the survey (must be https)",
		"link"=> "Reader-trust-1-link",

		"@question"=> "survey question message key",
		"question"=> "Reader-trust-1-message",

		"@privacyPolicy"=> "message key for privacy policy. May contain links.",
		"privacyPolicy"=> "Reader-trust-1-privacy",

		"@coverage"=> "percentage of users that will see the survey",
		"coverage"=> "0.002", // 1 out of 500

	        "@instanceTokenParameterName": "parameter to add to link",
                "instanceTokenParameterName": "token",

		"@platforms"=> "for each platform (desktop, mobile), which version of it is targeted (stable, beta, alpha)",
		"platforms"=> array(
			"desktop"=> ["stable"],
			"mobile"=> ["stable"]
		),

)

Event Timeline

Capt_Swing updated the task description. (Show Details)Nov 19 2018, 9:14 PM
Capt_Swing updated the task description. (Show Details)Nov 19 2018, 9:17 PM
leila added a subscriber: leila.Nov 20 2018, 12:33 AM

@Capt_Swing following up on our etherpad discussion:

  • I did one pass over the description of this task and it looks good to me.
  • Re sampling rate estimation: it's a tough one and it's roughly inline with what I would estimate, though my current estimate is that you will get less than 1K in 48 hours. :) When we had a sampling rate of 1:40 in enwiki (mobile and desktop) for a week, we got ~28K responses. If you sample at a sampling rate which is an order of magnitude less, I expect you to get no more than ~3K responses in a week. and that may put you closer to 800 responses in 48 hours. If you can tolerate that margin of difference, the current sampling rate you have is good. If not, you may want to change it.
leila triaged this task as High priority.Nov 20 2018, 12:33 AM

@leila Thanks! A smaller than 1k sample is probably better. The primary purpose of this round of survey is to gather enough free text responses to create a taxonomy of reasons people trust Wikipedia articles (and/or don't). Secondary purpose is to get a very rough distribution of responses. Second round of survey will use the reason taxonomy, and pull a larger sample.

Capt_Swing updated the task description. (Show Details)Nov 20 2018, 10:57 PM
Capt_Swing updated the task description. (Show Details)Nov 20 2018, 11:37 PM

Change 475105 had a related patch set uploaded (by Bmansurov; owner: Bmansurov):
[operations/mediawiki-config@master] Labs: enable the reader trust survey

https://gerrit.wikimedia.org/r/475105

@Capt_Swing in T209882#4765984 I've created a patch to load the survey on https://en.wikipedia.beta.wmflabs.org/. We'll need to load up the messages there now. Could you also ask a permission to beta labs while you're asking one for the production Wikipedia?

Capt_Swing updated the task description. (Show Details)Nov 26 2018, 7:23 PM

Change 475105 merged by jenkins-bot:
[operations/mediawiki-config@master] Labs: enable the reader trust survey

https://gerrit.wikimedia.org/r/475105

Mentioned in SAL (#wikimedia-operations) [2018-11-27T00:05:06Z] <niharika29@deploy1001> Synchronized wmf-config/InitialiseSettings-labs.php: Enable trust survey on labs T209882 (duration: 00m 46s)

@bmansurov I was never able to get the QuickSurvey to load on this page or on any other I tried. Were you able to preview it?

Change 476027 had a related patch set uploaded (by Bmansurov; owner: Bmansurov):
[operations/mediawiki-config@master] Labs: display reader trust survey

https://gerrit.wikimedia.org/r/476027

@Capt_Swing yes, I'm looking into the issue. I'll deploy a fix today and will let you know.

Change 476027 merged by jenkins-bot:
[operations/mediawiki-config@master] Labs: display reader trust survey

https://gerrit.wikimedia.org/r/476027

Change 476368 had a related patch set uploaded (by Bmansurov; owner: Bmansurov):
[operations/mediawiki-config@master] Enable reader trust survey

https://gerrit.wikimedia.org/r/476368

Change 476370 had a related patch set uploaded (by Bmansurov; owner: Bmansurov):
[operations/mediawiki-config@master] Disable reader trust survey

https://gerrit.wikimedia.org/r/476370

Capt_Swing updated the task description. (Show Details)Nov 28 2018, 7:42 PM

Updated the task description to reflect new deployment window.

Update: We tested it on beta labs on desktop and the survey seemed to work. Couldn't get the survey to show up on mobile and created T210646: Quick surveys do not show up on pages without infoboxes that have table of contents.

Capt_Swing updated the task description. (Show Details)Nov 28 2018, 7:55 PM
Capt_Swing moved this task from Staged to In Progress on the Research board.Nov 28 2018, 8:01 PM
Jdlrobson added a subscriber: Jdlrobson.

Hey all, can I please request a ping in phabricator whenever we enable QuickSurveys in future? They cause a large regression in performance on mobile which is scary to me and my team if we don't know the cause (and the fact its temporary).

@Jdlrobson I don't mind pinging you specifically, but is that the best option? Are there any other stakeholders who need to be alerted about mobile performance changes? Is there any better way to get that message out?

BTW we have decided to hold off on deploying QuickSurvey to production until mid-Jan.

Capt_Swing updated the task description. (Show Details)Dec 3 2018, 5:02 PM

Holding off on deployment until mid-Jan. We decided that deploying during fundraising would be disruptive and counterproductive, given the competing CTAs and the severely constrained screen real estate.

bmansurov added a comment.EditedDec 3 2018, 6:56 PM

@Jdlrobson I reported this in SoS last week: https://etherpad.wikimedia.org/p/Scrum-of-Scrums. I'm going to report again the next time we deploy it.

We are observing some interesting behavior on beta wikidata, both in desktop and mobile view.
A <div class="ext-qs-loader-bar mw-ajax-loader"></div> gets attached to what could be considered "the top of the page" in a traditional wiki - maybe the wikidata use case was not taken into consideration? What was the desired behavior in that regard?

I attached screenshots as the gray box does not to show consistently for everyone (maybe due to some A/B logic).

@Pablo-WMDE I think it's a case of messages not being created in the MediaWiki namespace. I probably should deploy QS only on enwiki on labs, if that's causing issues with your users. Please let me know.

@bmansurov Given the gray box appears right in the heart of a feature under active development, about to hit QA, I'd appreciate if it wasn't there.
I can't tell if solving this via configuration or by adding a check in the code (not to show on .wb-entitypage which generally don't seem to be compatible with it) is the better approach.
The latter would certainly act as a safety feature in case of accidental mal-configuration, too.

OK, Iʼll disable it on WD later today.

Change 478028 had a related patch set uploaded (by Bmansurov; owner: Bmansurov):
[operations/mediawiki-config@master] Labs: enable reader trust survey on enwiki only

https://gerrit.wikimedia.org/r/478028

I'm deploying the above patch in about two hours from now.

Change 478028 merged by jenkins-bot:
[operations/mediawiki-config@master] Labs: enable reader trust survey on enwiki only

https://gerrit.wikimedia.org/r/478028

@Pablo-WMDE, I've disabled the survey on Wikidata. Sorry for the trouble.

Capt_Swing moved this task from In Progress to Staged on the Research board.Dec 10 2018, 6:04 PM

@Jdlrobson I reported this in SoS last week: https://etherpad.wikimedia.org/p/Scrum-of-Scrums. I'm going to report again the next time we deploy it.

SoS reporting sounds good. Make sure to directly ping reading web and performance to make sure we hear about this. I've updated https://www.mediawiki.org/wiki/Extension:QuickSurveys/Deploying_surveys as well! Good luck with your survey!

Capt_Swing updated the task description. (Show Details)Jan 4 2019, 9:15 PM
Capt_Swing added a comment.EditedJan 4 2019, 10:08 PM

Survey has been re-scheduled and will be deployed on Monday, 2019/1/7

Change 476368 merged by jenkins-bot:
[operations/mediawiki-config@master] Enable reader trust survey

https://gerrit.wikimedia.org/r/476368

Mentioned in SAL (#wikimedia-operations) [2019-01-07T12:29:55Z] <zfilipin@deploy1001> Synchronized wmf-config/InitialiseSettings.php: SWAT: [[gerrit:476368|Enable reader trust survey (T209882)]] (duration: 00m 45s)

bmansurov added a comment.EditedJan 7 2019, 12:39 PM

The survey has been deployed. To see it in action visit this page, for example (make sure your browser's DoNotTrack feature is turned off). Here's a screenshot too:

(Both links are working).

Change 476370 merged by jenkins-bot:
[operations/mediawiki-config@master] Disable reader trust survey

https://gerrit.wikimedia.org/r/476370

Mentioned in SAL (#wikimedia-operations) [2019-01-09T12:40:49Z] <zfilipin@deploy1001> Synchronized wmf-config/InitialiseSettings.php: SWAT: [[gerrit:476370|Disable reader trust survey (T209882)]] (duration: 01m 07s)

The survey has just been undeployed.

Capt_Swing moved this task from Staged to In Progress on the Research board.Jan 9 2019, 8:08 PM

Thank you, @bmansurov! We got about 430 responses. Which is an excellent number of responses :)

Im fairly confident that this may have caused a big spike in client side errors due to how cached HTML works. @bmansurov it seems the flag to disable the survey removed Extension:QuickSurveys from en.wikipedia.org, but didn't counter for the fact that calls to the following remained in the HTML:

mw.loader.load('ext.quicksurveys.init' )


Interestingly this throws a client side error which hits the statsv beacon. If nobody beats me to it i will restore the extension to enwiki, set enwiki to empty array and leave a large note for future survey deployers.

Change 483393 had a related patch set uploaded (by Phuedx; owner: Phuedx):
[operations/mediawiki-config@master] Re-enable QuickSurveys extension on enwiki

https://gerrit.wikimedia.org/r/483393

Change 483393 merged by jenkins-bot:
[operations/mediawiki-config@master] Re-enable QuickSurveys extension on enwiki

https://gerrit.wikimedia.org/r/483393

@Jdlrobson thanks for spotting this. I wonder what a proper fix would be. There's got to be a way of removing that config. Hopefully the fix reduces the errors.

Mentioned in SAL (#wikimedia-operations) [2019-01-10T12:28:05Z] <zfilipin@deploy1001> Synchronized wmf-config/InitialiseSettings.php: SWAT: [[gerrit:483393|Re-enable QuickSurveys extension on enwiki (T209882)]] (duration: 00m 52s)

phuedx added a subscriber: phuedx.Jan 10 2019, 12:35 PM

@Jdlrobson thanks for spotting this. I wonder what a proper fix would be. There's got to be a way of removing that config. Hopefully the fix reduces the errors.

Agreed.

Perhaps if there are no surveys defined then we shouldn't inject the ext.quicksurveys.init module (see https://github.com/wikimedia/mediawiki-extensions-QuickSurveys/blob/6a7a4e74bbacd3f5516e253cedef502c3ce0c90a/includes/Hooks.php#L82-L97). If this were the case, then the strategy would be to remove all defined surveys for the wiki, wait for a week (the maximum keep time for the edge caches AFAIK), then disable the extension on that wiki.

Nevertheless, I'll be watching https://grafana.wikimedia.org/d/000000566/reading-web-dashboard?orgId=1&panelId=15&fullscreen&from=now-24h&to=now to see whether the above change reduced the client-side error rate at all.

bmansurov renamed this task from Quicksurvey for reader trust to Deploy/Undeploy Quicksurvey for reader trust.Jan 15 2019, 7:37 PM
bmansurov closed this task as Resolved.
bmansurov claimed this task.
bmansurov moved this task from In Progress to Done (current quarter) on the Research board.