@Capt_Swing following up on our etherpad discussion:

• I did one pass over the description of this task and it looks good to me.
• Re sampling rate estimation: it's a tough one and it's roughly inline with what I would estimate, though my current estimate is that you will get less than 1K in 48 hours. :) When we had a sampling rate of 1:40 in enwiki (mobile and desktop) for a week, we got ~28K responses. If you sample at a sampling rate which is an order of magnitude less, I expect you to get no more than ~3K responses in a week. and that may put you closer to 800 responses in 48 hours. If you can tolerate that margin of difference, the current sampling rate you have is good. If not, you may want to change it.

@leila Thanks! A smaller than 1k sample is probably better. The primary purpose of this round of survey is to gather enough free text responses to create a taxonomy of reasons people trust Wikipedia articles (and/or don't). Secondary purpose is to get a very rough distribution of responses. Second round of survey will use the reason taxonomy, and pull a larger sample.

@Capt_Swing All good then.

Change 475105 had a related patch set uploaded (by Bmansurov; owner: Bmansurov):
[operations/mediawiki-config@master] Labs: enable the reader trust survey

https://gerrit.wikimedia.org/r/475105

@Capt_Swing in T209882#4765984 I've created a patch to load the survey on https://en.wikipedia.beta.wmflabs.org/. We'll need to load up the messages there now. Could you also ask a permission to beta labs while you're asking one for the production Wikipedia?

Change 475105 merged by jenkins-bot:
[operations/mediawiki-config@master] Labs: enable the reader trust survey

https://gerrit.wikimedia.org/r/475105

Mentioned in SAL (#wikimedia-operations) [2018-11-27T00:05:06Z] <niharika29@deploy1001> Synchronized wmf-config/InitialiseSettings-labs.php: Enable trust survey on labs T209882 (duration: 00m 46s)

@bmansurov I was never able to get the QuickSurvey to load on this page or on any other I tried. Were you able to preview it?

Change 476027 had a related patch set uploaded (by Bmansurov; owner: Bmansurov):
[operations/mediawiki-config@master] Labs: display reader trust survey

https://gerrit.wikimedia.org/r/476027

@Capt_Swing yes, I'm looking into the issue. I'll deploy a fix today and will let you know.

Change 476027 merged by jenkins-bot:
[operations/mediawiki-config@master] Labs: display reader trust survey

https://gerrit.wikimedia.org/r/476027

Change 476368 had a related patch set uploaded (by Bmansurov; owner: Bmansurov):
[operations/mediawiki-config@master] Enable reader trust survey

https://gerrit.wikimedia.org/r/476368

Change 476370 had a related patch set uploaded (by Bmansurov; owner: Bmansurov):
[operations/mediawiki-config@master] Disable reader trust survey

https://gerrit.wikimedia.org/r/476370

Updated the task description to reflect new deployment window.

Update: We tested it on beta labs on desktop and the survey seemed to work. Couldn't get the survey to show up on mobile and created T210646: Quick surveys do not show up on pages without infoboxes that have table of contents.

Hey all, can I please request a ping in phabricator whenever we enable QuickSurveys in future? They cause a large regression in performance on mobile which is scary to me and my team if we don't know the cause (and the fact its temporary).

@Jdlrobson I don't mind pinging you specifically, but is that the best option? Are there any other stakeholders who need to be alerted about mobile performance changes? Is there any better way to get that message out?

BTW we have decided to hold off on deploying QuickSurvey to production until mid-Jan.

Holding off on deployment until mid-Jan. We decided that deploying during fundraising would be disruptive and counterproductive, given the competing CTAs and the severely constrained screen real estate.

@Jdlrobson I reported this in SoS last week: https://etherpad.wikimedia.org/p/Scrum-of-Scrums. I'm going to report again the next time we deploy it.

We are observing some interesting behavior on beta wikidata, both in desktop and mobile view.
A <div class="ext-qs-loader-bar mw-ajax-loader"></div> gets attached to what could be considered "the top of the page" in a traditional wiki - maybe the wikidata use case was not taken into consideration? What was the desired behavior in that regard?

I attached screenshots as the gray box does not to show consistently for everyone (maybe due to some A/B logic).

@Pablo-WMDE I think it's a case of messages not being created in the MediaWiki namespace. I probably should deploy QS only on enwiki on labs, if that's causing issues with your users. Please let me know.

@bmansurov Given the gray box appears right in the heart of a feature under active development, about to hit QA, I'd appreciate if it wasn't there.
I can't tell if solving this via configuration or by adding a check in the code (not to show on .wb-entitypage which generally don't seem to be compatible with it) is the better approach.
The latter would certainly act as a safety feature in case of accidental mal-configuration, too.

OK, Iʼll disable it on WD later today.

Change 478028 had a related patch set uploaded (by Bmansurov; owner: Bmansurov):
[operations/mediawiki-config@master] Labs: enable reader trust survey on enwiki only

https://gerrit.wikimedia.org/r/478028

I'm deploying the above patch in about two hours from now.

Change 478028 merged by jenkins-bot:
[operations/mediawiki-config@master] Labs: enable reader trust survey on enwiki only

https://gerrit.wikimedia.org/r/478028

@Pablo-WMDE, I've disabled the survey on Wikidata. Sorry for the trouble.

Thank you, @bmansurov!

@Jdlrobson I reported this in SoS last week: https://etherpad.wikimedia.org/p/Scrum-of-Scrums. I'm going to report again the next time we deploy it.

SoS reporting sounds good. Make sure to directly ping reading web and performance to make sure we hear about this. I've updated https://www.mediawiki.org/wiki/Extension:QuickSurveys/Deploying_surveys as well! Good luck with your survey!

Survey has been re-scheduled and will be deployed on Monday, 2019/1/7

Change 476368 merged by jenkins-bot:
[operations/mediawiki-config@master] Enable reader trust survey

https://gerrit.wikimedia.org/r/476368

Mentioned in SAL (#wikimedia-operations) [2019-01-07T12:29:55Z] <zfilipin@deploy1001> Synchronized wmf-config/InitialiseSettings.php: SWAT: [[gerrit:476368|Enable reader trust survey (T209882)]] (duration: 00m 45s)

The survey has been deployed. To see it in action visit this page, for example (make sure your browser's DoNotTrack feature is turned off). Here's a screenshot too:

(Both links are working).

Change 476370 merged by jenkins-bot:
[operations/mediawiki-config@master] Disable reader trust survey

https://gerrit.wikimedia.org/r/476370

Mentioned in SAL (#wikimedia-operations) [2019-01-09T12:40:49Z] <zfilipin@deploy1001> Synchronized wmf-config/InitialiseSettings.php: SWAT: [[gerrit:476370|Disable reader trust survey (T209882)]] (duration: 01m 07s)

The survey has just been undeployed.

Thank you, @bmansurov! We got about 430 responses. Which is an excellent number of responses :)

Im fairly confident that this may have caused a big spike in client side errors due to how cached HTML works. @bmansurov it seems the flag to disable the survey removed Extension:QuickSurveys from en.wikipedia.org, but didn't counter for the fact that calls to the following remained in the HTML:

mw.loader.load('ext.quicksurveys.init' )

Change 483393 had a related patch set uploaded (by Phuedx; owner: Phuedx):
[operations/mediawiki-config@master] Re-enable QuickSurveys extension on enwiki

https://gerrit.wikimedia.org/r/483393

Change 483393 merged by jenkins-bot:
[operations/mediawiki-config@master] Re-enable QuickSurveys extension on enwiki

https://gerrit.wikimedia.org/r/483393

@Jdlrobson thanks for spotting this. I wonder what a proper fix would be. There's got to be a way of removing that config. Hopefully the fix reduces the errors.

Mentioned in SAL (#wikimedia-operations) [2019-01-10T12:28:05Z] <zfilipin@deploy1001> Synchronized wmf-config/InitialiseSettings.php: SWAT: [[gerrit:483393|Re-enable QuickSurveys extension on enwiki (T209882)]] (duration: 00m 52s)

In T209882#4869179, @bmansurov wrote:

@Jdlrobson thanks for spotting this. I wonder what a proper fix would be. There's got to be a way of removing that config. Hopefully the fix reduces the errors.

Agreed.

Perhaps if there are no surveys defined then we shouldn't inject the ext.quicksurveys.init module (see https://github.com/wikimedia/mediawiki-extensions-QuickSurveys/blob/6a7a4e74bbacd3f5516e253cedef502c3ce0c90a/includes/Hooks.php#L82-L97). If this were the case, then the strategy would be to remove all defined surveys for the wiki, wait for a week (the maximum keep time for the edge caches AFAIK), then disable the extension on that wiki.

Nevertheless, I'll be watching https://grafana.wikimedia.org/d/000000566/reading-web-dashboard?orgId=1&panelId=15&fullscreen&from=now-24h&to=now to see whether the above change reduced the client-side error rate at all.

The client-side error rate has since fallen to its previous level: https://grafana.wikimedia.org/d/000000566/reading-web-dashboard?orgId=1&panelId=15&fullscreen&from=1547035200000&to=now&refresh=5m

\o/, thanks @phuedx!