Page MenuHomePhabricator

Hovercards beta feature preference OFF not honored for logged in users when the experiment is enabled
Closed, ResolvedPublic

Description

Story
As a logged-in user, I want the ability to explicitly state whether I want hovercards or not without being part of the a/b test

Description
As a logged in user, my selection on beta features should always take preference over bucketing.

Failing test case:

  1. Visit as anon, get bucketed in group A (hovercards ON)
  2. Hovercards work
  3. Proceed to log in to your user account (which has beta feature hovercards OFF)
  4. After you log in, see that you are still seeing hovercards, even if you didn't select the beta feature ON.

Detailed steps (you need a user account with hovercards beta feature off in http://reading-web-staging.wmflabs.org/):

  1. Visit http://reading-web-staging.wmflabs.org/wiki/Main_Page in a new private/incognito window and wait for the page to fully load
  2. If you can see Enable previews in the footer link, then close the window and try again (goto 1)
  3. Once you don't see that link, verify by hovering over [[Test]] that you can see a hovercard. That means you've been bucketed as group A.
  4. Proceed to login into your no-hovercards-beta-feature user account
  5. Log in and go back to main page
ExpectedHovering over [[Test]] doesn't show a hovercard because on my user account i don't have the beta feature enabled
ActualHovering over [[Test]] shows a hovercard

Tech

For logged in users, return here https://github.com/wikimedia/mediawiki-extensions-Popups/blob/master/resources/ext.popups.experiment.js#L76-L79 instead of falling through to returning the bucketing token.

This will break unit tests and all the browser for settings that are based on logged in users.

Related abandoned patch https://gerrit.wikimedia.org/r/#/c/311397/

Event Timeline

Change 311397 had a related patch set uploaded (by Jhernandez):
Skip experiment condition for logged in users

https://gerrit.wikimedia.org/r/311397

Discussed possible use cases with @Jhernandez here and decided on option 2 (below). Adding showing/hiding cog/link at bottom to a/c:

Logged in users have hovercards enabled/disabled by the beta feature flag only
Pro:

  • Anon users get bucketed and can manage enable/disable via hovercards
  • Doesn't bucket logged in users or bother them
  • Logged in users use beta features for managing hovercards (preserves their choice as user)

Cons:

  • Enable previews link and cog don't work
  • Needs follow up task to hide link and cog for logged in users
  • Logged in users don't use hovercards link+cog for enable/disable
jhobs subscribed.

Reflecting who has actually been working on this.

Reviewed at Monday standup, but determined that the task is not ready yet, and there are comments happening in the parent task. Set to review again on Tuesday 09/20

We're going to QA the AC with the current master because it seems like that's mostly the behavior there is and that there was a bunch of confusion.

More tomorrow.

I'm doing different QA scenarios, here's one that's problematic (I believe):

  1. Visit as anon, get bucketed in group A (hovercards ON)
  2. Hovercards work
  3. Proceed to log in to your user account (which has beta feature hovercards OFF)
  4. After you log in, see that you are still seeing hovercards, even if you didn't select the beta feature ON.

Detailed steps (you need a user account with hovercards beta feature off in http://reading-web-staging.wmflabs.org/):

  1. Visit http://reading-web-staging.wmflabs.org/wiki/Main_Page in a new private/incognito window and wait for the page to fully load
  2. If you can see Enable previews in the footer link, then close the window and try again (goto 1)
  3. Once you don't see that link, verify by hovering over [[Test]] that you can see a hovercard. That means you've been bucketed as group A.
  4. Proceed to login into your no-hovercards-beta-feature user account
  5. Log in and go back to main page
ExpectedHovering over [[Test]] doesn't show a hovercard because on my user account i don't have the beta feature enabled
ActualHovering over [[Test]] shows a hovercard
Jhernandez renamed this task from Change test group for hovercards a/b test to exclude all logged-in users with hovercards off to Hovercards beta feature preference OFF not honored for logged in users when the experiment is enabled.Sep 20 2016, 10:35 AM
Jhernandez updated the task description. (Show Details)

Change 311397 abandoned by Jhernandez:
Skip experiment for logged in users

Reason:
Abandoning for now.

https://gerrit.wikimedia.org/r/311397

Note: for the above test, hovercards appear for the following use cases:

  1. User has hovercards off by default
  2. User has hovercards off by explicitly turning them on (and off)

Current behavior (2):

  1. Login. Set Beta feature for hovercards on.
  2. Hovercards are displayed
  3. Set beta feature for hovercards off
  4. Hovercares are not displayed
  5. Change browser session to be placed in test bucket
  6. Login.
  7. Hovercards are displayed
  8. Go to beta settings - hovercards checkbox is unchecked

The expected behavior we want is listed below. @Jhernandez, @dr0ptp4kt, let me know if this makes sense.

Run test for anons only

Users who have feature off via preferences (explicitly or not)

  1. If user is in a bucketed session prior to login, hovercards do not appear (turn hovercards off)
  2. If user is not in a bucketed session prior to login, hovercards do not appear (do nothing)

Users who has explicitly turned feature on via preferences

  1. If user is in a bucketed session prior to login, hovercards appear (keep hovercards ON)
  2. If user is not in a bucketed session prior to login, hovercards appear (turn hovercards ON)

Note: for all of the above, once a user is logged in, the settings cog and link at the bottom should be hidden and hovercards settings should be set from beta preferences only

Change 311727 had a related patch set uploaded (by Jhernandez):
Respect beta feature flag for logged in users

https://gerrit.wikimedia.org/r/311727

Change 311727 merged by jenkins-bot:
Respect beta feature flag for logged in users

https://gerrit.wikimedia.org/r/311727

Above cases were QA'd on staging by @Jhernandez, @dr0ptp4kt, @ovasileva
Behave as expected.