@Jdlrobson Revising a bit from what I said on IRC: for our recent survey (March 4/5) on English Wikipedia, I looked at the distribution of webhost (en vs. en.m) and browser family (Chrome, Chrome Mobile etc.) for those who received the survey (QuickSurveyInitiation) vs all of the webrequests to en.wikipedia for that same time period. It seems less about mobile vs. desktop and more about specific browsers or OSes. See below (and I'm happy to talk more):

Oversampled:

• Chrome (desktop) by 25%
• Chrome Mobile (mobile web) by 28%
• Safari (desktop) by 4%
• Samsung Internet (mobile web) by 35%
• Edge (desktop) by 29%
• Mobile Safari UI/WKWebView (mobile web) by 30%

Undersampled:

• Mobile Safari (mobile web) by 4%
• IE (desktop) by 70%
• Firefox (desktop) by 33%
• Chrome Mobile iOS (mobile web) by 75%
• Other (desktop) by 99%
• Opera Mini (mobile web) by 99%
• Android (mobile web) by 85%

All webrequests query:

SELECT user_agent_map['browser_family'], access_method, count(1) AS num_views
FROM wmf.webrequest
WHERE year = 2019 AND month = 3 AND ((day = 4 AND hour > 18) or (day = 5))
      AND is_pageview AND agent_type = 'user' AND normalized_host.project_class = 'wikipedia' AND normalized_host.project = 'en'
GROUP BY user_agent_map['browser_family'], access_method
ORDER BY num_views LIMIT 1000;

QuickSurveyInitiation query (note that distinct surveysessiontoken largely results in the same trends):

SELECT useragent.browser_family, webhost, COUNT(DISTINCT(event.surveysessiontoken)) AS num_views
FROM event.quicksurveyinitiation
WHERE event.surveyCodeName = 'reader-demographics-en-pilot' AND year = 2019 AND month = 3 AND ((day = 4 and hour > 18) OR (day = 5))
GROUP BY useragent.browser_family, webhost
ORDER BY num_views
LIMIT 1000;

We estimated this as a Large/Medium. Right now quick surveys checks if the mobile beta opt in panel is displayed. If it is, the survey doesn't show.

An alternative solution I discussed today (which I'd hope would be more on the small estimation size) is that we could remove all this code and unconditionally show any surveys/beta optin panels that are showing. Instead of relying on code (which might be problematic if a quick survey campaign is not being shown during a mobile beta opt in campaign) we should rely on social queues e.g. No quick surveys on mobile during the month of May while we run this opt into beta campaign.

The task that led to the introduction of the mobile.betaoptin event: T111188

If the code for QuickSurveys loads AFTER MobileFrontend, the hook will be registered too late and the survey will never show.

Per [[ https://github.com/wikimedia/mediawiki/blob/6679632b770d0c883c26a78f483f340e59950e2c/resources/src/mediawiki.base/mediawiki.base.js#L345-L374 | mw.trackSubscribes DocBlock ]]:

Handlers will be called once for each tracked event, including any events that fired before the handler was registered…

It seems like QuickSurveys subscribing late is already dealt with.

Not sure if this is the same issue as in this thread or should be separated into a new task, but...

Building on my comment above (T218243#5025475), I ran the same analyses for a new survey that we deployed for reader trust. We see very consistent over/undersampling where readers are certain platforms are much more/less likely to see a survey than expected. The vast majority of page views come from Chrome (desktop), Mobile Safari (mobile web), and Chrome Mobile (mobile web) and then IE (desktop) and Firefox (desktop).

#s refer to how much the survey over/undersampled that platform compared to general webrequests from that same time period. Platforms in order of how often their frequency in the webrequests query.

A few others of with large differences:

Reader trust query for all webrequests:

SELECT user_agent_map['browser_family'] AS browser_family, access_method, count(1) AS num_views
  FROM wmf.webrequest
  WHERE year = 2019 AND month = 3 AND day >= 18 AND day < 23 AND is_pageview AND agent_type = 'user'
        AND normalized_host.project_class = 'wikipedia' AND normalized_host.project = 'en'
 GROUP BY user_agent_map['browser_family'], access_method
 ORDER BY num_views DESC
 LIMIT 800;

Reader trust query for QuickSurveyInitiation:

SELECT useragent.browser_family AS browser_family, webhost as webhost, COUNT(DISTINCT(event.surveysessiontoken)) AS num_views
  FROM event.quicksurveyinitiation
 WHERE event.surveyCodeName = 'reader trust survey en v2' AND year = 2019 AND month = 3 AND day >= 18 AND day < 23
 GROUP BY useragent.browser_family, webhost
 ORDER BY num_views DESC
 LIMIT 800;

FYI that the queries in webrequest table on this ticket do not take into account our browser support, clients like IE <11 do not receive EL at all, thus they will be present on "overall" traffic (webrequest) but cannot be present on quicksurveys. Ditto of some older versions of Opera, see browser support: https://www.mediawiki.org/wiki/Compatibility#Modern_(Grade_A)

Change 545368 had a related patch set uploaded (by Phuedx; owner: Phuedx):
[mediawiki/extensions/QuickSurveys@master] [init] Don't respond to mobile.betaoptin event

https://gerrit.wikimedia.org/r/545368

Change 545368 merged by jenkins-bot:
[mediawiki/extensions/QuickSurveys@master] [init] Don't respond to mobile.betaoptin event

https://gerrit.wikimedia.org/r/545368