Page MenuHomePhabricator

QuickSurveys will not show on mobile if the beta optin experiment is not set
Open, NormalPublic

Description

Problem: The code to display a QuickSurvey in mobile is error prone. We have potential for dropping a small but unknown percentage of survey views due to this problem.

Background

QuickSurveys subscribes to a 'mobile.betaoptin' event to decide whether to show a survey in the mobile Minerva skin.

This event only fires inside MobileFrontend if displayBetaOptIn is called inside the mobile.init module which only happens if mw.config.get( 'wgMFExperiments' ).betaoptin is . (luckily this is true on production) and if the user hasn't dismissed a request to opt-into beta (luckily given the experiment is disabled this won't happen in production.

Proposed solution

  • QuickSurveys when running in mobile (wgMFMode is defined) should use mw.loader.using( 'mobile.init' ) before trying to run a survey. This will cause beta opt in panel to load if enabled
  • QuickSurveys should check the presence of a .panel element inside .mw-parser-output to decide whether it shows or not, rather than using the existing mobile.betaoptin track event.
  • mobile.betaoptin tracking event should be removed from MobileFrontend.

QA steps

  • When the beta optin panel is enabled and displayed, no QuickSurveys show
  • When I say "no thanks" to the beta opt in, the QuickSurveys shows
  • Ask a developer to disable the beta opt in panel on mobile and check the QuickSurvey shows for all page views.

Developer notes

With MobileFrotend and QuickSurveys installed, a survey will never show (in the current setup, despite the fact it should) with the following config:

$wgExperiments = []

Event Timeline

ovasileva triaged this task as High priority.Mar 13 2019, 6:48 PM
ovasileva moved this task from To Triage to Needs Analysis on the Readers-Web-Backlog board.
Aklapper edited projects, added QuickSurveys, Mobile; removed Surveys.Mar 14 2019, 9:53 AM
phuedx moved this task from Needs triage to Triaged on the Mobile board.Mar 14 2019, 5:18 PM
Isaac added a comment.Mar 14 2019, 7:38 PM

@Jdlrobson Revising a bit from what I said on IRC: for our recent survey (March 4/5) on English Wikipedia, I looked at the distribution of webhost (en vs. en.m) and browser family (Chrome, Chrome Mobile etc.) for those who received the survey (QuickSurveyInitiation) vs all of the webrequests to en.wikipedia for that same time period. It seems less about mobile vs. desktop and more about specific browsers or OSes. See below (and I'm happy to talk more):

Oversampled:

  • Chrome (desktop) by 25%
  • Chrome Mobile (mobile web) by 28%
  • Safari (desktop) by 4%
  • Samsung Internet (mobile web) by 35%
  • Edge (desktop) by 29%
  • Mobile Safari UI/WKWebView (mobile web) by 30%

Undersampled:

  • Mobile Safari (mobile web) by 4%
  • IE (desktop) by 70%
  • Firefox (desktop) by 33%
  • Chrome Mobile iOS (mobile web) by 75%
  • Other (desktop) by 99%
  • Opera Mini (mobile web) by 99%
  • Android (mobile web) by 85%

All webrequests query:

SELECT user_agent_map['browser_family'], access_method, count(1) AS num_views
FROM wmf.webrequest
WHERE year = 2019 AND month = 3 AND ((day = 4 AND hour > 18) or (day = 5))
      AND is_pageview AND agent_type = 'user' AND normalized_host.project_class = 'wikipedia' AND normalized_host.project = 'en'
GROUP BY user_agent_map['browser_family'], access_method
ORDER BY num_views LIMIT 1000;

QuickSurveyInitiation query (note that distinct surveysessiontoken largely results in the same trends):

SELECT useragent.browser_family, webhost, COUNT(DISTINCT(event.surveysessiontoken)) AS num_views
FROM event.quicksurveyinitiation
WHERE event.surveyCodeName = 'reader-demographics-en-pilot' AND year = 2019 AND month = 3 AND ((day = 4 and hour > 18) OR (day = 5))
GROUP BY useragent.browser_family, webhost
ORDER BY num_views
LIMIT 1000;
phuedx removed a subscriber: phuedx.Mar 18 2019, 9:21 AM

We estimated this as a Large/Medium. Right now quick surveys checks if the mobile beta opt in panel is displayed. If it is, the survey doesn't show.

An alternative solution I discussed today (which I'd hope would be more on the small estimation size) is that we could remove all this code and unconditionally show any surveys/beta optin panels that are showing. Instead of relying on code (which might be problematic if a quick survey campaign is not being shown during a mobile beta opt in campaign) we should rely on social queues e.g. No quick surveys on mobile during the month of May while we run this opt into beta campaign.

Jdlrobson moved this task from Upcoming to Product Owner Backlog on the Readers-Web-Backlog board.

Olga to think about this

The task that led to the introduction of the mobile.betaoptin event: T111188

If the code for QuickSurveys loads AFTER MobileFrontend, the hook will be registered too late and the survey will never show.

Per mw.trackSubscribes DocBlock:

Handlers will be called once for each tracked event, including any events that fired before the handler was registered…

It seems like QuickSurveys subscribing late is already dealt with.

Jdlrobson renamed this task from QuickSurveys will sometimes not show on mobile to QuickSurveys will not show on mobile if the beta optin experiment is not set.Apr 1 2019, 6:45 PM
Jdlrobson updated the task description. (Show Details)
Jdlrobson lowered the priority of this task from High to Normal.

Have revised the description/title based on Sam's quite right feedback. TO be clear

mw.trackSubscribe( 'mobile.betaoptin', function ( topic, data ) { console.log( 'Called', topic ); } );
mw.track( 'mobile.betaoptin' );

console.log will be run.

mw.track( 'mobile.betaoptin' );
mw.trackSubscribe( 'mobile.betaoptin', function ( topic, data ) { console.log( 'Called', topic ); } );

console.log will be run.

Isaac added a comment.EditedApr 4 2019, 10:23 PM

Not sure if this is the same issue as in this thread or should be separated into a new task, but...

Building on my comment above (T218243#5025475), I ran the same analyses for a new survey that we deployed for reader trust. We see very consistent over/undersampling where readers are certain platforms are much more/less likely to see a survey than expected. The vast majority of page views come from Chrome (desktop), Mobile Safari (mobile web), and Chrome Mobile (mobile web) and then IE (desktop) and Firefox (desktop).

#s refer to how much the survey over/undersampled that platform compared to general webrequests from that same time period. Platforms in order of how often their frequency in the webrequests query.

PlatformTrust SurveyDemographic Survey
Chrome (desktop)+23%+25%
Mobile Safari (mobile web)-1%-4%
Chrome Mobile (mobile web)+32%+28%
IE (desktop)-76%-70%
Firefox (desktop)-30%-33%
Safari (desktop)+18%+4%
Samsung Internet (mobile web)+23%+35%
Edge (desktop)+1%+29%
Chrome Mobile iOS (mobile web)-76%-75%
Other (desktop)-99%-99%

A few others of with large differences:

Opera Mini (mobile web)-99%-99%
Android (mobile web)-83%-85%
Mobile Safari UI/WKWebView (mobile web)+134%+30%

Reader trust query for all webrequests:

SELECT user_agent_map['browser_family'] AS browser_family, access_method, count(1) AS num_views
  FROM wmf.webrequest
  WHERE year = 2019 AND month = 3 AND day >= 18 AND day < 23 AND is_pageview AND agent_type = 'user'
        AND normalized_host.project_class = 'wikipedia' AND normalized_host.project = 'en'
 GROUP BY user_agent_map['browser_family'], access_method
 ORDER BY num_views DESC
 LIMIT 800;

Reader trust query for QuickSurveyInitiation:

SELECT useragent.browser_family AS browser_family, webhost as webhost, COUNT(DISTINCT(event.surveysessiontoken)) AS num_views
  FROM event.quicksurveyinitiation
 WHERE event.surveyCodeName = 'reader trust survey en v2' AND year = 2019 AND month = 3 AND day >= 18 AND day < 23
 GROUP BY useragent.browser_family, webhost
 ORDER BY num_views DESC
 LIMIT 800;
Nuria added a subscriber: Nuria.Apr 15 2019, 9:11 PM

FYI that the queries in webrequest table on this ticket do not take into account our browser support, clients like IE <11 do not receive EL at all, thus they will be present on "overall" traffic (webrequest) but cannot be present on quicksurveys. Ditto of some older versions of Opera, see browser support: https://www.mediawiki.org/wiki/Compatibility#Modern_(Grade_A)