Page MenuHomePhabricator

Set up quicksurveys for UI and non-UI experiments
Closed, ResolvedPublic3 Estimated Story Points

Description

NOTE: surveys will be enabled on September 9

Background

  • We would like to recruit participants to our UI and non-UI experiments using quicksurveys
  • We will run 2 surveys that will recruit participants for each of the experiments

User story

  • As a reader, I would like to be told when there's opportunity to do user testing for new features, so that I can help out Wikipedia

Requirements non-UI survey

Quicksurvey setup

Requirements UI survey

Requirement

Set up two QuickSurveys to recruit participants for UI and non-UI experiments on English and Spanish Wikipedias. The non-UI survey links to a Google form, while the UI survey links to the browser extension landing page. The surveys target logged-out users, with specific percentages for each wiki. The surveys should go live on September 9th.

BDD

Feature: Quicksurvey Setup for UI and Non-UI Experiments

  Scenario: Display non-UI survey for logged-out users
    Given the user is logged out
    And the user visits English or Spanish Wikipedia
    When the survey is displayed
    Then the user sees a single question linked to a Google form
    And the survey displays to 5% of users on English Wikipedia and 10% on Spanish Wikipedia

  Scenario: Display UI survey for logged-out users on Chrome
    Given the user is logged out and using Google Chrome
    And the user visits English or Spanish Wikipedia
    When the survey is displayed
    Then the user sees a single question linked to the browser extension landing page
    And the survey displays to 5% of users on English Wikipedia and 10% on Spanish Wikipedia

Test Steps

Test Case 1: Verify Non-UI Survey Display for Logged-Out Users

  1. Log out and visit English Wikipedia.
  2. AC1: Confirm that the survey displays for 5% of logged-out users with a link to a Google form.
  3. Repeat the steps for Spanish Wikipedia, ensuring the survey appears for 10% of logged-out users.

Test Case 2: Verify UI Survey Display for Logged-Out Chrome Users

  1. Log out and open English Wikipedia in Google Chrome.
  2. AC2: Confirm that the survey displays for 5% of logged-out Chrome users with a link to the browser extension landing page.
  3. Repeat the steps for Spanish Wikipedia, ensuring the survey appears for 10% of logged-out Chrome users.

Design

  • Add mockups and design requirements

Communication criteria - does this need an announcement or discussion?

  • Add communication criteria

Rollback plan

  • What is the rollback plan in production for this task if something goes wrong?

This task was created by Version 1.2.0 of the Web team task template using phabulous

QA Results - Prod

ACStatusDetails
1T373039#10139425
2T373039#10139425

Event Timeline

There are a very large number of changes, so older changes are hidden. Show Older Changes

@ovasileva the requirements of this task are a little unclear - are we hoping to run this survey this sprint, or prepare the survey for this sprint? If the latter we should update the acceptance criteria to reflect this. If the former, perhaps this needs to be moved to next sprint?

Currently blocked on having a URL for the browser extension in the Chrome store.

@ovasileva the requirements of this task are a little unclear - are we hoping to run this survey this sprint, or prepare the survey for this sprint? If the latter we should update the acceptance criteria to reflect this. If the former, perhaps this needs to be moved to next sprint?

Currently blocked on having a URL for the browser extension in the Chrome store.

Enabling the survey would be a separate step. I can make a different ticket for that

After talking about this with the team, we'll move this to next sprint when we have access to the browser extension URL. This will minimize the likelihood of mistakes.

Okay there are a few issues with task:

  • QuickSurveys don't go through translatewiki so we'll need the copy in Spanish for these surveys before we can work on this!
  • Looking through the survey proposed won't actually work - external surveys need a yes and no answer. Is that okay? Currently the requirements suggest a "Next" and only a "Yes" button.
  • It's not clear what the privacy policy text should be - this needs to be in the ticket!

Quick question that came up in separate conversation - do quick surveys target by browser? Would be nice if we can target only Chrome users, but not a hard requirement

@Jdlrobson - I've found our previous survey setup and translations. These can be used for the non-UI survey and already have Spanish translations. I've added the link to these in the task description. The UI survey will require separate copy. Will add that into the task description next. Saving the previous acceptance criteria here as well in case we need it later.

== Acceptance criteria

  • Obtain translations in Spanish
  • Create message on English Wikipedia at MediaWiki:ext-quicksurveys-T373039-no-ui-question with text:

Help us improve Wikipedia by answering a few questions. This will only take 5 minutes of your time.

  • Create message on English Wikipedia at MediaWiki:ext-quicksurveys-T373039-ui-question with text:

Would you like to help us improve Wikipedia by checking out a new feature?

  • Create message on English Wikipedia at MediaWiki:ext-quicksurveys-T373039-privacy-policy with text:

`<TODO
< English:: >
<Spanish: >`

  • Create the above 3 messages on https://es.wikipedia.org
  • Use ext-quicksurveys-survey-positive and ext-quicksurveys-survey-negative for "Yes" and "No" answers.
  • Setup survey using type external using these instructions: https://www.mediawiki.org/wiki/Extension:QuickSurveys#1.43
  • Survey should be set to "enabled": true, and "coverage" 0 to ensure the survey doesn't show until we are ready.
  • Make sure there is a privacy policy prior to enabling survey
ovasileva updated the task description. (Show Details)

Can we set up the survey on the beta cluster so that @JScherer-WMF can check it out before Monday?

Chatted with Isabel yesterday briefly. We're looking at Monday for translations of the Spanish non-UI survey. Depending on timing, that may push launch to Wednesday.

Change #1071240 had a related patch set uploaded (by Jdrewniak; author: Jdrewniak):

[mediawiki/extensions/WikimediaMessages@master] Messages for Web search experiment QuickSurvey

https://gerrit.wikimedia.org/r/1071240

Change #1071241 had a related patch set uploaded (by Jdrewniak; author: Jdrewniak):

[operations/mediawiki-config@master] Add Web search experiment quickSurvey on beta cluster

https://gerrit.wikimedia.org/r/1071241

Change #1071241 merged by jenkins-bot:

[operations/mediawiki-config@master] [BETA CLUSTER] Add Web search experiment quickSurvey

https://gerrit.wikimedia.org/r/1071241

Change #1071708 had a related patch set uploaded (by Jdlrobson; author: Jdrewniak):

[mediawiki/extensions/QuickSurveys@wmf/1.43.0-wmf.21] Support new heading layout

https://gerrit.wikimedia.org/r/1071708

Change #1071708 merged by jenkins-bot:

[mediawiki/extensions/QuickSurveys@wmf/1.43.0-wmf.21] Support new heading layout

https://gerrit.wikimedia.org/r/1071708

Mentioned in SAL (#wikimedia-operations) [2024-09-10T13:06:27Z] <logmsgbot> lucaswerkmeister-wmde@deploy1003 Started scap sync-world: Backport for [[gerrit:1071708|Support new heading layout (T373039 T374377)]]

Mentioned in SAL (#wikimedia-operations) [2024-09-10T13:10:10Z] <logmsgbot> lucaswerkmeister-wmde@deploy1003 lucaswerkmeister-wmde, jdlrobson: Backport for [[gerrit:1071708|Support new heading layout (T373039 T374377)]] synced to the testservers (https://wikitech.wikimedia.org/wiki/Mwdebug)

Mentioned in SAL (#wikimedia-operations) [2024-09-10T13:18:39Z] <logmsgbot> lucaswerkmeister-wmde@deploy1003 Finished scap sync-world: Backport for [[gerrit:1071708|Support new heading layout (T373039 T374377)]] (duration: 12m 11s)

Change #1071890 had a related patch set uploaded (by Jdrewniak; author: Jdrewniak):

[operations/mediawiki-config@master] Configure QuickSurvey for Web empty search state experiment

https://gerrit.wikimedia.org/r/1071890

FYI - Was able to confirm quicksurvey events are firing locally. No events are collected in beta so we would have to wait to validate events in prod

Screenshot 2024-09-10 at 12.36.43 PM.png (746×2 px, 188 KB)

We're going to prepare the non-UI survey but make sure it is disabled by default enabled=>false. If we do not have translations in time by the end of the week, we should create a new ticket for next sprint to enable it in production.

Change #1071890 merged by jenkins-bot:

[operations/mediawiki-config@master] Configure QuickSurvey for Web empty search state experiments

https://gerrit.wikimedia.org/r/1071890

Mentioned in SAL (#wikimedia-operations) [2024-09-10T20:13:34Z] <kindrobot@deploy1003> Started scap sync-world: Backport for [[gerrit:1071037|Enable native MathML by default on group0 (T373703)]], [[gerrit:1071890|Configure QuickSurvey for Web empty search state experiments (T373039)]]

Mentioned in SAL (#wikimedia-operations) [2024-09-10T20:15:46Z] <kindrobot@deploy1003> kindrobot, jdrewniak, physikerwelt: Backport for [[gerrit:1071037|Enable native MathML by default on group0 (T373703)]], [[gerrit:1071890|Configure QuickSurvey for Web empty search state experiments (T373039)]] synced to the testservers (https://wikitech.wikimedia.org/wiki/Mwdebug)

Mentioned in SAL (#wikimedia-operations) [2024-09-10T20:23:12Z] <kindrobot@deploy1003> Finished scap sync-world: Backport for [[gerrit:1071037|Enable native MathML by default on group0 (T373703)]], [[gerrit:1071890|Configure QuickSurvey for Web empty search state experiments (T373039)]] (duration: 09m 37s)

  • The browser extension survey is live (1 day late)
  • The non UI extension survey is live but disabled but we can't seem to get it showing via a debug URL. Jan is going to debug. Once we've worked out what's going on there, everything should be in place for enabling tomorrow.

Change #1072191 had a related patch set uploaded (by Jdrewniak; author: Jdrewniak):

[operations/mediawiki-config@master] Enable Web team search suggestions survey

https://gerrit.wikimedia.org/r/1072191

Change #1072191 merged by jenkins-bot:

[operations/mediawiki-config@master] Enable Web team search suggestions survey

https://gerrit.wikimedia.org/r/1072191

Test Result - Beta

Status: ❓Need More Info
Environment: Beta
OS: macOS
Browser: Chrome
Device: MS MBA
Emulated Device: NA

Test Artifact(s):

Test Steps

Test Case 1: Verify Non-UI Survey Display for Logged-Out Users

  1. Log out and visit English Wikipedia.
  2. AC1: Confirm that the survey displays for 5% of logged-out users with a link to a Google form. Repeat the steps for Spanish Wikipedia, ensuring the survey appears for 10% of logged-out users.

@Jdlrobson / @Jdrewniak I'm not sure if the non-UI survey is ready or not. If it is, is there a way to get the specific QuickSurvey, or is it random?

Test Case 2: Verify UI Survey Display for Logged-Out Chrome Users

  1. Log out and open English Wikipedia in Google Chrome.
  2. AC2: Confirm that the survey displays for 5% of logged-out Chrome users with a link to the browser extension landing page.

screenshot 1.mov.gif (906×1 px, 2 MB)

Test Result - Prod

Status: ❓Need More Info ✅ PASS
Environment: enwiki, eswiki
OS: macOS
Browser: Chrome
Device: MS MBA
Emulated Device: NA

Test Artifact(s):

Test Steps
NOTE: These are a pass per discussion during task sync.

Test Case 1: Verify Non-UI Survey Display for Logged-Out Users

  1. Log out and visit English Wikipedia.
  2. AC1: Confirm that the survey displays for 5% of logged-out users with a link to a Google form. Repeat the steps for Spanish Wikipedia, ensuring the survey appears for 10% of logged-out users.

@Jdlrobson / @Jdrewniak I'm not sure if the non-UI survey is ready or not. If it is, is there a way to get the specific QuickSurvey, or is it random?

Test Case 2: Verify UI Survey Display for Logged-Out Chrome Users

  1. Log out and open English Wikipedia in Google Chrome.
  2. AC2: Confirm that the survey displays for 5% of logged-out Chrome users with a link to the browser extension landing page. Repeat the steps for Spanish Wikipedia, ensuring the survey appears for 10% of logged-out Chrome users.

This works good, only question I have is if the expectation is that when a user on eswiki clicks on the privacy policy if it should navigate to the English or Spanish?

screenshot 3.mov.gif (906×1 px, 2 MB)

screenshot 2.mov.gif (906×1 px, 2 MB)

Change #1071240 abandoned by Jdrewniak:

[mediawiki/extensions/WikimediaMessages@master] Messages for Web search experiment QuickSurvey

https://gerrit.wikimedia.org/r/1071240

Jdlrobson claimed this task.
Jdlrobson updated the task description. (Show Details)

I've created T374743 to disable the surveys! Nice job all!