Page MenuHomePhabricator

Variant C/D: inconsistencies with welcome survey and discovery popups
Open, Needs TriagePublic

Description

I was testing the variants by creating new accounts in production using the query string parameter. I'm using Chrome in incognito mode. Here is what I noticed:

  • When creating accounts with Variants C and D, if I click "skip" from the welcome survey, I end up on whatever page from which I created the account (in this case it was Main Page), but I do not get the discovery popup that points to my username and says, "Go to your homepage to get started" (original specifications on T222852). This does not happen on Variant A. If I create an account with Variant A, and skip the welcome survey, I do get the popup.
  • On one of these Variant D accounts that I created, I happened to switch to mobile mode in my browser. Then I did see the mobile banner telling me to go to my homepage. Weird that I didn't get the desktop one but I did get the mobile one. That banner is pictured below.
  • I also created two accounts with Variant D in which I did not get the welcome survey at all.
  • These accounts were in Arabic, Portuguese, and Russian Wikipedia. I forgot which things happened to which accounts, but these are the usernames I was working with:
    • MMiller Test 15 ar
    • MMiller Test 16 ar
    • MMillerTest01pt
    • MMillerTest01ru
    • MMillerTest01ru
    • MMillerTest01ru
    • MMillerTest01ru

Event Timeline

I also noticed this, and was going to report it. https://people.wikimedia.org/~urbanecm/screencasts/T265754/variant-c-no-discovery.webm is me creating an account with variant C, https://people.wikimedia.org/~urbanecm/screencasts/T265754/default-variant-discovery.webm is the default version getting the discovery tours with no issue.

I looked into the specs T222852: Homepage: discovery of homepage after account creation (desktop) and re-tested skip the Welcome survey option (will test other paths too).

  • tested in cswiki, arwiki, ruwiki
  • forced creating user with C variant with Special:CreateAccount?geForceVariant=C
    • Never receives survey / Skips welcome survey (clicks the button on survey to skip)
    • From main page: redirect straight to homepage with GuidedTour to point at personal tools link, explaining how to return to >homepage Fig. B1.
    • From anywhere else: redirect straight to original context with GuidedTour to point at personal tools link to invite discovering the > homepage Fig. B2.
  • Doesn't click buttons on the survey confirmation screen / Abandons welcome survey (navigates away from survey with clicking >either button)
    • To whatever page: a GuidedTour points at personal tools link to invite discovering the homepage Fig. B4.
Fig. B1
Fig. B2
Fig. B4
NOTE: According to the above specs the workflow after skipping the survey depends on the initial page that a user was on:

comes from Main page -> redirect to the Homepage with Guided tour
comes from an article -> redirect to the article with Guided tour

The results of testing

user Starting pageWelcome survey received?Welcome survey actionuser Landing page after surveyGuided tour present there?Homepage enabled?Guided tour on Homepage is present?
Main pageYesskipsMain pageNoYesYes
Article pageYesskipsMain pageNoYesYes

from @MMiller_WMF

On one of these Variant D accounts that I created, I happened to switch to mobile mode in my browser. Then I did see the mobile banner telling me to go to my homepage. Weird that I didn't get the desktop one but I did get the mobile one.

Mobile Guided tour is independent from desktop - some additional testing on mobile also need to be done.

This is a priority for engineers to look into next week, either for a backport or with next week's train.

MMiller_WMF added a subscriber: Dyolf77_WMF.

Potentially related report by @Dyolf77_WMF via email: "if you ignore the Welcome survey after creating an account, you are asked to enter your password again!"

Potentially related report by @Dyolf77_WMF via email: "if you ignore the Welcome survey after creating an account, you are asked to enter your password again!"

I've observed this a few times in my local environment but hadn't seen it in production; I don't think this is particular to declining the welcome survey. If you take some action quickly after creating your account (navigating to another page) you're sometimes prompted to login again. That is probably worth its own task for investigating, I assume it is relating to some settings in MediaWiki core.

This issue is a consequence of logic we implemented in T229044: Homepage: users should only receive one discovery GuidedTour, where we said we didn't want to show the homepage welcome tour to users who had seen the homepage discovery tour, as well as some code in implementing T258009: Variant C/D: welcome popup on desktop where we excluded variant C and D users from the discovery tour.

So to rephrase:

  • All variants should see the "discover your homepage" tour if they don't go to the homepage after the welcome survey
  • Variants C/D should see the "welcome to your homepage" tour, even if they have already seen the "discover your homepage" tour

Is that correct?

Tgr added a comment.Oct 19 2020, 8:04 PM

Potentially related report by @Dyolf77_WMF via email: "if you ignore the Welcome survey after creating an account, you are asked to enter your password again!"

That sounds concerning, but it is definitely not related. https://www.mediawiki.org/wiki/Manual:How_to_debug/Login_problems has info on debugging login issues (which can be notoriously hard).

This issue is a consequence of logic we implemented in T229044: Homepage: users should only receive one discovery GuidedTour, where we said we didn't want to show the homepage welcome tour to users who had seen the homepage discovery tour, as well as some code in implementing T258009: Variant C/D: welcome popup on desktop where we excluded variant C and D users from the discovery tour.

So to rephrase:

  • All variants should see the "discover your homepage" tour if they don't go to the homepage after the welcome survey
  • Variants C/D should see the "welcome to your homepage" tour, even if they have already seen the "discover your homepage" tour

Is that correct?

@kostajh -- here's how I think it is (applicable to desktop only):

  • When a user is on the welcome survey, if they skip or abandon the survey or submit the survey but then don't click the button on the survey confirmation page to go to their homepage, then they get the popup that says, like, "Click here to visit your homepage" -- not matter what page they are on at that moment.
    • If you follow that popup, then you should still get the Variant C/D popup upon arrival to the homepage. That's because the Variant C/D popups contain an important call-to-action (especially in Variant C).
  • When a user is on the welcome survey, if they submit the survey and click the button to go to their homepage, then they should get the Variant C/D popups.

So basically, everyone gets Variant C/D popups upon arrival to homepage.

Does that all make sense?

Tgr claimed this task.Oct 19 2020, 9:49 PM

So:

  • show the non-homepage popup if the user is not on the homepage and hasn't seen any popups yet
  • show the homepage popup if the user is on the homepage and hasn't seen the homepage popup yet
  • this applies to all variants (A has a different spec but shouldn't get new users anymore so we don't care)
  • don't try to fix this for users who have registered before the fix is deployed, we accept that they don't see the popup

does that sound correct @MMiller_WMF?

Tgr added a comment.Oct 19 2020, 11:47 PM
  • don't try to fix this for users who have registered before the fix is deployed, we accept that they don't see the popup

Actually, the code that sets the "user has not seen the welcome popup" flag on registration is still in place for C/D, only the code that shows the popup and unsets the flag was removed. So the way it will work out if we don't put any extra effort into it is that those people will get the popup right when the fix is deployed.

Change 635115 had a related patch set uploaded (by Gergő Tisza; owner: Gergő Tisza):
[mediawiki/extensions/GrowthExperiments@master] Show homepage discovery popup in variant C/D

https://gerrit.wikimedia.org/r/635115

Change 635115 merged by jenkins-bot:
[mediawiki/extensions/GrowthExperiments@master] Show homepage discovery popup in variant C/D

https://gerrit.wikimedia.org/r/635115

@Tgr -- I think what you listed is correct. But about the last point: we don't want legacy users to all get new popups. We only want new users going forward to get the popups like they're supposed to.

Tgr added a comment.Oct 20 2020, 5:39 AM

By legacy users, I only mean the users who were selected into C/D and thus didn't get a welcome popup (so basically users registering between Monday and Thursday). I assume special-casing them is not worth the effort; if it is, we'll have to run a maintenance script the change the relevant user option for them.

Change 635331 had a related patch set uploaded (by Catrope; owner: Gergő Tisza):
[mediawiki/extensions/GrowthExperiments@wmf/1.36.0-wmf.13] Show homepage discovery popup in variant C/D

https://gerrit.wikimedia.org/r/635331

@Tgr -- that's right, special-casing this week's users is not important to do.

Change 635331 merged by jenkins-bot:
[mediawiki/extensions/GrowthExperiments@wmf/1.36.0-wmf.13] Show homepage discovery popup in variant C/D

https://gerrit.wikimedia.org/r/635331

Mentioned in SAL (#wikimedia-operations) [2020-10-21T23:14:55Z] <catrope@deploy1001> Synchronized php-1.36.0-wmf.13/extensions/GrowthExperiments/: T265751 T265754 (duration: 01m 08s)

For @MMiller_WMF review. I think that all that is in your comment - https://phabricator.wikimedia.org/T265754#6562268 - is in place
Please let me know if some more testing is needed

Tested on betalabs and production (wmf.14).

Summary of the test results:

  • all of newly created accounts have Welcome survey
  • all users get Homepage welcome tour when they land on Special:Homepage for the first time
  • after some actions (including "skip" or abandon) on Welcome survey I there are four outcomes
Special:WelcomeSurvey pageReturning to Main page or an articleDirecting to the HomepageLogging out
one or two buttons - "go to homepage" "return to Main (article) page- Homepage discovery tour will be present when a user navigates from Special:WelcomeSurvey pageHomepage discovery tour is presentHomepage welcome tour is presentobserved quite often on arwiki as a result of 'skip'/'abandon' action on Welcome survey