Page MenuHomePhabricator

Test survey on beta-enwiki
Closed, ResolvedPublic

Description

Test reader demographics QuickSurvey on beta enwiki.

  • Create interface pages
  • Deploy survey to beta
  • Test:
    • Make sure the browser's DNT feature is turned off.
    • Go to https://en.wikipedia.beta.wmflabs.org/wiki/Book (on desktop) and look for the survey on the top right of the article content.
    • If you see another survey, dismiss it, and reload the page. Multiple surveys maybe enabled at one time, so keep dismissing other surveys until you see yours.
    • Once you see your survey, take it and see if your survey token is being passed to the external URL.
    • Check other things as you find fit.
NOTE: be aware of T210646, which notes a bug about QuickSurveys not showing up on mobile that only affects beta, not production.

Event Timeline

Isaac created this task.Feb 26 2019, 6:43 PM

Please let me know the deploy date as early as possible as I need to reserve a slot in the deployment calendar, which may get filled.

Isaac added a comment.Feb 26 2019, 6:59 PM

Please let me know the deploy date as early as possible as I need to reserve a slot in the deployment calendar, which may get filled.

Thanks for reminder re: deployment -- still waiting on privacy statement but as soon as that is finalized and uploaded to foundation, I'll message you.

Also: interface pages now created on beta enwiki as detailed above so I think we're ready to test at least. Is there anything else that I can provide to help launch it on beta or specific things I should be checking for once it's launched?

Yes, let's add the configuration for the survey to the description of this task and the parent task (one for production — I imagine they'll be the same, they will be good documentation for the future).

Oh I see the parent task already has it: T217080.

Yeah, the parent-child relationships in phab are somewhat of a mess as I figure these things out but i linked this to T217080 so that it would be clearer

Isaac updated the task description. (Show Details)Feb 26 2019, 7:45 PM

Change 493103 had a related patch set uploaded (by Bmansurov; owner: Bmansurov):
[operations/mediawiki-config@master] Labs: Enable reader demographics survey

https://gerrit.wikimedia.org/r/493103

I'll enable the survey in the beta cluster in about 4 hours from now.

bmansurov updated the task description. (Show Details)Feb 26 2019, 8:16 PM
Isaac added a comment.Feb 26 2019, 8:23 PM

enabling this evening is great and thanks for updating the testing details. i'll try to do the testing then and also will add those details to the documentation

Change 493103 merged by jenkins-bot:
[operations/mediawiki-config@master] Labs: Enable reader demographics survey

https://gerrit.wikimedia.org/r/493103

The change's been deployed. It should be live in about 15-30 mins.

Scratch that, the config was not complete. I missed the 'enabled' => true, part. I'll redeploy tomorrow.

Isaac added a comment.Feb 27 2019, 1:49 AM

Not seeing the survey at https://en.wikipedia.beta.wmflabs.org/wiki/Book

Just the reader trust survey appears despite refreshes / dismissing it. On inspection, the only potential issue I can see is that it appears that the config is missing 'enabled' => true

Change 493221 had a related patch set uploaded (by Bmansurov; owner: Bmansurov):
[operations/mediawiki-config@master] Labs: actually enable the demographics survey

https://gerrit.wikimedia.org/r/493221

Change 493221 merged by jenkins-bot:
[operations/mediawiki-config@master] Labs: actually enable the demographics survey

https://gerrit.wikimedia.org/r/493221

@Isaac the survey should be live in about 15-30 mins.

Ignoring the issues with caching and the links (T217282), the pageviewToken is correctly being loaded into the Google Forms survey and matches the token being recorded in the QuickSurveyInitiation and QuickSurveysResponses schemas.

Interestingly, opening the survey in a new tab did not trigger the QuickSurveysResponses schema as far as I could tell from the client side (by watching the network tab in the web console), which is odd though not a blocker (the QuickSurveyInitiation schema gives not all but enough information in most cases)

Also, not sure if this is expected behavior or not, but I only see the survey on mobile when looking at Spain, but not Book:

See: https://en.m.wikipedia.beta.wmflabs.org/wiki/Spain
Don't see: https://en.m.wikipedia.beta.wmflabs.org/wiki/Book

That's true even when I append ?quicksurvey=external-survey-reader-demographics-en-pilot to the end of the URL

Interestingly, opening the survey in a new tab did not trigger the QuickSurveysResponses schema as far as I could tell from the client side (by watching the network tab in the web console), which is odd though not a blocker (the QuickSurveyInitiation schema gives not all but enough information in most cases)

It maybe worth creating a separate task for this.

The Book issue is known: T210646: Quick surveys do not show up on pages without infoboxes that have table of contents (The task you linked in the description).

Isaac added a comment.Feb 28 2019, 4:05 PM

Thanks for linking to that Book issue. After reading through, it looks like while it still is an issue on Beta, it's not supposed to affect production any more. I'll see what I can do to verify this when the pilot runs.

And I'll dig into the Responses EL issue to see the extent of it.

Isaac added a comment.Feb 28 2019, 7:13 PM

@bmansurov after doing a bit more digging, I think what is going on with the right-click + "open in new tab" is very similar to what was reported here re: hovercards: T147846

Not clear if this is just beta or production too. I see the behavior on Chrome and Firefox. Notably:

  • right-click + "open in new tab" = no events triggered (survey link remains instead of switching to "thank you" and no QuickSurveysResponses EL)
  • cmd+click (which also opens the survey in a new tab) = events triggered (survey box changes to "than you" and QuickSurveysResponses EL triggers)

@Isaac OK, thanks for looking into this. I think it's worth keeping in mind when doing analysis. If we want an event triggered on right-click + "open in new tab", then we should create a task for it.

Isaac added a comment.Mar 1 2019, 7:36 PM

Yeah, if I recall there weren't too many instances in the previous reader motivation survey where there was a completed survey but not a corresponding EL data point. I'll check this with the pilot and then see whether this seems to be an issue worth addressing (and quite possibly it doesn't even happen on production).

Isaac updated the task description. (Show Details)Mar 4 2019, 3:40 PM
Isaac closed this task as Resolved.Mar 4 2019, 3:44 PM

Once issues with caching of the interface text were worked out, beta test went well. Will pay attention to potential bug where QuickSurveysResponses EL does not trigger when a user right-clicks and opens the survey in a new tab (evidence of this would be QuickSurveyInitiation, a completed survey, but no QuickSurveysResponses EL).