Page MenuHomePhabricator

Instrument mobile web language switching user workflow
Closed, ResolvedPublic5 Story Points

Description

As a product owner, I want to understand the baseline language switching behavior on the mobile website and the impact of potential changes so that the change alternatives with the greatest evident impact are shipped and so that I can monitor the continued effectiveness of the changes for a period sufficient to guarantee effectiveness of the changes for users who use the feature.

Staging it

  • Confirmed on localhost
  • Confirmed on beta labs
  • Confirmed on mobile web beta channel
  • Confirmed on mobile web stable channel

Sampling

  • Beta channel: 10% funnel eligibility when no beta enrollment CTA in force; when beta enrollment CTA in force, adjust lower (probably 1%) to balance coverage across multiple languages without growing table too fast
  • Stable channel: 0.01% funnel eligibility

Note that sampling eligibility should only be considered for pages on which there is a language switcher button. This may create too sparse of data, but we'll adjust if needed.

Questions to be answered

  • What's the likelihood of a loaded page actually having an alternative language?
  • What's the impression rate for the language switcher button?
  • What's the clickthrough rate for the language switcher button when shown?
  • Given the user tapped the language switcher, is the user a repeat user of the switcher button (languageButtonTappedBucket 0, 1-4, 5-20, 20 or more; probably localStorage?)
  • Which user language is primary on the user's device? (Note that the langwiki will already be part of the event capsule, but it may or may not be aligned with the user's primary language.)
  • How many languages are available to users when the modal is first shown?
  • How often do users use the language switcher search?
  • How often do users tap a language from the language picker (irrespective of whether the language search was used)?
  • What's the position of a language in the list when tapped?
  • How many results were in the language list when the user took action to navigate away from the modal?
  • Was a language search result set being used at the time of tapthrough/dismiss?
  • How often do users dismiss the modal explicitly ('x' button)?
  • How long does the funnel interaction take, from language switcher button shown to language tapped?
  • How does behavior vary between beta channel and stable channel?
  • Which version of the button / chrome is used to get to the language switcher?
  • Which version of the language switcher modal is being used?

Important qualities of event logging

  • Either events must only fire for sendBeacon capable UAs, or the first event in the funnel must indicate whether the UA is sendBeacon capable (this helps with examining leaky funnels when the user taps a link at the end of the funnel).
  • To make the implicit explicit, there should be a unique funnel token for each eligible funnel. It does not need to be tied to the user's session ID. The funnel should end once the user leaves the modal (i.e., there's a relatively slim possibility that the user will have a new funnel - which would have a new unique value - on the same page).
  • Capture the mobileMode on the first event in the funnel.
  • Once a user is eligible for event logging on the page, all of the relevant events in the funnel should be sent as they occur, tied to the funnelToken.

Initial notes

Here were initial notes. They should be updated to reflect the criteria above.

Language switcher. What we need to know:
    
    Spike: https://phabricator.wikimedia.org/T123382
    
When language button is available, do users see the language button?
Do they tap it?
On modal:
Do they tap on the list?
Do they use the search input?
Do they get 0 results? (e.g. when typing "ja" for Japanese) 
Do they tap on results when they search?
Is the prioritized language list effective?
Is their language, as determined by Accept-Language (available in JS) available?
What is their language?
How long does it take to dismiss or tap on an option?
Is there repeat usage?
    
    BE SURE: only log with sendBeacon capable UAs
    
Characteristics:
    channel (beta, ...)
Events:
Page loaded
BOOL (or INT if inexpensive) alternative languages available
Language button impression
Language button tap
Version of button tapped (old vs new)
List loaded
Exit modal
Dismissed modal or tapped on result
Tapped 
on which language
relative position of tapped result at that instant
if search input has query
Length of list at time language tapped
User entered first char in language search input

Event Timeline

There are a very large number of changes, so older changes are hidden. Show Older Changes

Any reason for the different sampling rates for stable / beta?

It was mainly to account for the relatively small number of pageviews on beta (e.g., one recent weekday had only about 70K mobile web beta pageviews from the looks of it). Data may be too sparse for solid analysis, particularly for multiple langwikis (unlike prod where there's lots of access), without an active CTA to enroll in beta. Make sense?

The problem with this is it would skew analysis and lead to invalid conclusions being drawn if analyst doesn't know this difference in sampling.
It might make sense to use a different schema (a clone) for beta and stable. Something smells funny to me about using different sampling rates in different modes.

Certainly having 3 different sampling rates looks like a recipe for disaster and I'm not sure what the third one is about :)

@bmansurov Jenkins doesn't like your patch (npm and qunit issues)

The problem with this is it would skew analysis and lead to invalid conclusions being drawn if analyst doesn't know this difference in sampling.

@HaeB: We're increasing the sampling rate in MFβ because of it's comparatively tiny population. Please take this into account in your analysis. The sample rates are given in the description of this task.

Problem solved?

Seriously though, @bmansurov, @dr0ptp4kt: the MobileWebLanguageSwitcher schema doesn't include a mobileMode property, whereas most of our schemas do. Is this intentional? I for one, would be reassured to know that users don't change their behaviour when switching language based on $MFChannel.

@phuedx, I think @Jdlrobson was asking about why we have two different sampling rates for the beta users.
As for mobileMode, we could certainly add it. Are we trying to answer the question of whether users use the language switcher differently depending on which mode they are in?

@phuedx, I think @Jdlrobson was asking about why we have two different sampling rates for the beta users.
As for mobileMode, we could certainly add it. Are we trying to answer the question of whether users use the language switcher differently depending on which mode they are in?

Thanks for the clarification @bmansurov. I completely misunderstood @Jdlrobson's comment.

Beta channel: 10% funnel eligibility when no beta enrollment CTA in force; when beta enrollment CTA in force, adjust lower (probably 1%) to balance coverage across multiple languages without growing table too fast

@dr0ptp4kt: Are we to collect data indefinitely or for a limited amount of time? If it's the latter, then perhaps we could simply disable the beta opt-in while while we're doing so.

Let's add mobileMode.

@phuedx, I anticipate us running these at least until the design changes have been rolled to stable and sitting comfortably for a couple weeks.

@bmansurov, yeah, there's a nontrivial chance beta channel users will behave differently than stable channel users, so having the ability to change the SQL queries to differentiate between the two will be useful.

@Jdlrobson, I think for now we only need to sample differently between beta and stable channel. The point about the CTA is if we turn it back on we should probably lower the beta sampling rate config variable for this schema. But AFAIK we weren't necessarily planning to turn the CTA back on at this time.

dr0ptp4kt updated the task description. (Show Details)Jan 21 2016, 4:35 PM
dr0ptp4kt updated the task description. (Show Details)Jan 21 2016, 4:40 PM
Tbayer removed a subscriber: HaeB.Jan 21 2016, 5:49 PM

The problem with this is it would skew analysis and lead to invalid conclusions being drawn if analyst doesn't know this difference in sampling.

@HaeB: We're increasing the sampling rate in MFβ because of it's comparatively tiny population. Please take this into account in your analysis. The sample rates are given in the description of this task.
Problem solved?

Yes, using a higher sampling rate in beta makes sense to me. And as long as sampling rates are clearly documented here, I can work with that.
I think @Jdlrobson may have been talking about the general EventLogging problem that the different channels and their sampling rates are not documented in the central schema pages. (Someone even suggested recently to send the sampling rate with each event.) I have various unformed thoughts about that, but it's not something we need to solve right now.

Tbayer removed a subscriber: HaeB.Jan 21 2016, 6:38 PM
dr0ptp4kt updated the task description. (Show Details)Jan 22 2016, 12:16 AM
dr0ptp4kt updated the task description. (Show Details)Jan 22 2016, 12:48 AM
dr0ptp4kt updated the task description. (Show Details)Jan 22 2016, 12:58 AM
dr0ptp4kt updated the task description. (Show Details)Jan 22 2016, 1:03 AM

I'm on this, will end review this morning

phuedx claimed this task.Jan 27 2016, 5:47 PM

Change 264909 merged by jenkins-bot:
Allow logging events around language switcher

https://gerrit.wikimedia.org/r/264909

We should be able to test this on beta cluster. You'll need to disable sampling using user js:
http://en.m.wikipedia.beta.wmflabs.org/wiki/User:Jdlrobson/minerva.js

I'm not seeing an events though.
I suspect we need config changes for wgMFSchemaMobileWebLanguageSwitcherSampleRate on both beta labs and production before signing this off so moving back to ready for dev.

Jdlrobson removed phuedx as the assignee of this task.Jan 27 2016, 10:04 PM

There is already a patch for that: https://gerrit.wikimedia.org/r/#/c/265292/

It's also linked to the current task as you can see above.

We should be able to test this on beta cluster. You'll need to disable sampling using user js:
http://en.m.wikipedia.beta.wmflabs.org/wiki/User:Jdlrobson/minerva.js
I'm not seeing an events though.
I suspect we need config changes for wgMFSchemaMobileWebLanguageSwitcherSampleRate on both beta labs and production before signing this off so moving back to ready for dev.

I'm not sure that you'll see events with wgMFIgnoreEventLoggingBucketing set to true. @bmansurov has – correctly – used the upstreamed mw.eventLog.Schema class to implement the instrumentation. I'm not sure that the class cares, or should care, about wgMFIgnoreEventLoggingBucketing.

I've set both the beta and stable rates to 1 so that MobileWebLanguageSwitcher events should always be logged. However, when I did so I saw the following error being thrown:

Unknown dependency: schema.SchemaMobileWebLanguageSwitcher

Change 267018 had a related patch set uploaded (by Phuedx):
Fix "Unknown dependency" error in I43d1b69

https://gerrit.wikimedia.org/r/267018

I've deployed 267018 to the staging server.

phuedx removed phuedx as the assignee of this task.Jan 28 2016, 1:21 PM
phuedx updated the task description. (Show Details)

With both the beta and stable rates set to 1 and 267018 in place, I've confirmed that this works on the staging server.

Change 267020 had a related patch set uploaded (by Bmansurov):
SchemaMobileWebLanguageSwitcher: Ignore the sampling rate for testing purposes

https://gerrit.wikimedia.org/r/267020

^ should help us test the code on beta labs.

Change 267018 merged by jenkins-bot:
Fix "Unknown dependency" error in I43d1b69

https://gerrit.wikimedia.org/r/267018

Change 267025 had a related patch set uploaded (by Bmansurov):
Add sampling rates for mobile web language switcher in production

https://gerrit.wikimedia.org/r/267025

Change 267029 had a related patch set uploaded (by Phuedx):
SchemaMobileWebLanguageSwitcher: Ignore the sampling rate for testing purposes

https://gerrit.wikimedia.org/r/267029

Change 267020 abandoned by Bmansurov:
SchemaMobileWebLanguageSwitcher: Ignore the sampling rate for testing purposes

Reason:
https://gerrit.wikimedia.org/r/#/c/267029/

https://gerrit.wikimedia.org/r/267020

Change 267029 merged by jenkins-bot:
SchemaMobileWebLanguageSwitcher: Ignore the sampling rate for testing purposes

https://gerrit.wikimedia.org/r/267029

I've confirmed that this works on the staging server with just wgMFIgnoreEventLoggingBucketing set to true – I've also checked that nothing breaks with wgMFIgnoreEventLoggingBucketing set to false mind…

@Jdlrobson: Given the length and complexity of the history of this task, could you add a reason when you move the task?

There was a patch -1ed when I moved it. This doesn't seem to be the case any more and it looks like the only remaining patch is https://gerrit.wikimedia.org/r/#/c/267025/ which I've scheduled for SWAT as no harm in that going out early.

I'll get signing off.

Ah found it - @bmansurov https://gerrit.wikimedia.org/r/#/c/265292/ - is -1ed so I cannot test on beta labs. Moving back to -1

Do we need that patch? The sampling rates are so low that you won't be able to test anyway. Why wouldn't adding mw.config.set( 'wgMFIgnoreEventLoggingBucketing', true ); to the user script work?

True. Please abandon that patch with that decision to avoid confusion. Right now I'm unable to replicate on beta cluster... will see if I can work out why.

https://gerrit.wikimedia.org/r/#/c/265292/ needs review before we can sign off.
The user js method runs at the bottom of the page and only works with things using the Schema class.

@bmansurov, @Jdlrobson: I've reviewed 265292 and, after it's been tweaked, it LGTM.

Did you find someone to SWAT this @bmansurov ?
Given it's a labs change anyone should be able to do this outside the window.

@Jdlrobson, I talked to a couple of people and they were busy reverting some change. I was suggested to use the window for SWATs, which is what I'm doing.

Change 267025 merged by jenkins-bot:
Add sampling rates for mobile web language switcher in production

https://gerrit.wikimedia.org/r/267025

Change 265292 merged by jenkins-bot:
Add sampling rates for mobile web language switcher on beta labs

https://gerrit.wikimedia.org/r/265292

Looks good to me. I can see the events on beta labs.
I'll let Adam have the final say about moving it to done!

How do you see events on beta labs when mw.config.get( 'wgMFSchemaMobileWebLanguageSwitcherSampleRate', {} ) returns Object {beta: 0, stable: 0}?

Mm. I was seeing events. Not any more. Maybe a race condition to IgnoreSampling...?? What happened? Beta labs is playing up..

I also just got this issue with debug=true caused by the LanguageOverlay... we might want to look into that. So I guess work still needed :/

Uncaught TypeError: Cannot read property 'log' of undefinedOO.mfExtend.postRender @ load.php?debug=true&lang=en&modules=mobile.languages&only=scripts&skin=minerva&version=a2bc56d1f42b:62OO.mfExtend.render @ View.js:240OO.mfExtend._postInitialize @ View.js:201OO.mfExtend.initialize @ View.js:182View.extend.initialize @ Overlay.js:117View @ View.js:86Child @ View.js:349LanguageOverlay @ load.php?debug=true&lang=en&modules=mobile.languages&only=scripts&skin=minerva&version=a2bc56d1f42b:27(anonymous function) @ init.js:99fire @ load.php?debug=true&lang=en&modules=jquery%2Cmediawiki&only=scripts&skin=minerva&version=sWfOQgtB:3148self.fireWith @ load.php?debug=true&lang=en&modules=jquery%2Cmediawiki&only=scripts&skin=minerva&version=sWfOQgtB:3260deferred.(anonymous function) @ load.php?debug=true&lang=en&modules=jquery%2Cmediawiki&only=scripts&skin=minerva&version=sWfOQgtB:3350(anonymous function) @ PageGateway.js:282fire @ load.php?debug=true&lang=en&modules=jquery%2Cmediawiki&only=scripts&skin=minerva&version=sWfOQgtB:3148self.fireWith @ load.php?debug=true&lang=en&modules=jquery%2Cmediawiki&only=scripts&skin=minerva&version=sWfOQgtB:3260deferred.(anonymous function) @ load.php?debug=true&lang=en&modules=jquery%2Cmediawiki&only=scripts&skin=minerva&version=sWfOQgtB:3350(anonymous function) @ api.js:248fire @ load.php?debug=true&lang=en&modules=jquery%2Cmediawiki&only=scripts&skin=minerva&version=sWfOQgtB:3148self.fireWith @ load.php?debug=true&lang=en&modules=jquery%2Cmediawiki&only=scripts&skin=minerva&version=sWfOQgtB:3260done @ load.php?debug=true&lang=en&modules=jquery%2Cmediawiki&only=scripts&skin=minerva&version=sWfOQgtB:9314callback @ load.php?debug=true&lang=en&modules=jquery%2Cmediawiki&only=scripts&skin=minerva&version=sWfOQgtB:9718

dr0ptp4kt removed dr0ptp4kt as the assignee of this task.Jan 29 2016, 6:16 AM

My suspicion was incorrect. The beta-scap-eqiad job is now building successfully and I still can't see the effects of the config change on the Beta Cluster.

Mm. I was seeing events. Not any more. Maybe a race condition to IgnoreSampling...?? What happened? Beta labs is playing up..
I also just got this issue with debug=true caused by the LanguageOverlay... we might want to look into that. So I guess work still needed :/

Uncaught TypeError: Cannot read property 'log' of undefinedOO.mfExtend.postRender @ load.php?debug=true&lang=en&modules=mobile.languages&only=scripts&skin=minerva&version=a2bc56d1f42b:62OO.mfExtend.render @ View.js:240OO.mfExtend._postInitialize @ View.js:201OO.mfExtend.initialize @ View.js:182View.extend.initialize @ Overlay.js:117View @ View.js:86Child @ View.js:349LanguageOverlay @ load.php?debug=true&lang=en&modules=mobile.languages&only=scripts&skin=minerva&version=a2bc56d1f42b:27(anonymous function) @ init.js:99fire @ load.php?debug=true&lang=en&modules=jquery%2Cmediawiki&only=scripts&skin=minerva&version=sWfOQgtB:3148self.fireWith @ load.php?debug=true&lang=en&modules=jquery%2Cmediawiki&only=scripts&skin=minerva&version=sWfOQgtB:3260deferred.(anonymous function) @ load.php?debug=true&lang=en&modules=jquery%2Cmediawiki&only=scripts&skin=minerva&version=sWfOQgtB:3350(anonymous function) @ PageGateway.js:282fire @ load.php?debug=true&lang=en&modules=jquery%2Cmediawiki&only=scripts&skin=minerva&version=sWfOQgtB:3148self.fireWith @ load.php?debug=true&lang=en&modules=jquery%2Cmediawiki&only=scripts&skin=minerva&version=sWfOQgtB:3260deferred.(anonymous function) @ load.php?debug=true&lang=en&modules=jquery%2Cmediawiki&only=scripts&skin=minerva&version=sWfOQgtB:3350(anonymous function) @ api.js:248fire @ load.php?debug=true&lang=en&modules=jquery%2Cmediawiki&only=scripts&skin=minerva&version=sWfOQgtB:3148self.fireWith @ load.php?debug=true&lang=en&modules=jquery%2Cmediawiki&only=scripts&skin=minerva&version=sWfOQgtB:3260done @ load.php?debug=true&lang=en&modules=jquery%2Cmediawiki&only=scripts&skin=minerva&version=sWfOQgtB:9314callback @ load.php?debug=true&lang=en&modules=jquery%2Cmediawiki&only=scripts&skin=minerva&version=sWfOQgtB:9718

Verified. Given that the config change isn't taking effect, it feels something went wrong with the MobileFrontend deploy (!?)

bmansurov updated the task description. (Show Details)Jan 29 2016, 8:12 PM

Beta labs has the latest sampling rates.

over to you Adam!

dr0ptp4kt added a comment.EditedJan 30 2016, 12:35 AM

Signed off. Please reduce sampling rate on beta labs. Additionally: it's not a blocker but the languageOverlayVersion value is coming across as bottom-of-article for the languageListLoaded event. It should probably be something else like simpler-overlay, instead (this does not technically require a schema rev, but one is suggested because the description field for it isn't describing the overlay, but rather the button - they're two different things, though).

Using https://wikitech.wikimedia.org/wiki/Analytics/EventLogging/TestingOnBetaCluster, tail -f /srv/log/eventlogging/all-events.log | grep MobileWebLanguageSwitcher showed events tied to access of http://en.m.wikipedia.beta.wmflabs.org/wiki/San_Francisco.

MariaDB, as indicated below, also confirmed as much.

I do want to note seeing a CORS warning. Unsure if this is a known issue (and on beta labs), although it didn't seem to matter for the actual event transmission.

15:23:50.713 Cross-Origin Request Blocked: The Same Origin Policy disallows reading the remote resource at http://en.wikipedia.beta.wmflabs.org/beacon/event?%7B%22event%22%3A%7B%22funnelToken%22%3A%22315902a3e1b28faf1454109605752%22%2C%22mobileMode%22%3A%22stable%22%2C%22event%22%3A%22exitModal%22%2C%22exitModal%22%3A%22tapped-on-result%22%2C%22languageTapped%22%3A%22de%22%2C%22positionOfLanguageTapped%22%3A9%2C%22searchInputHasQuery%22%3Afalse%2C%22languageCount%22%3A119%7D%2C%22revision%22%3A15265240%2C%22schema%22%3A%22MobileWebLanguageSwitcher%22%2C%22webHost%22%3A%22en.m.wikipedia.beta.wmflabs.org%22%2C%22wiki%22%3A%22enwiki%22%7D;. (Reason: CORS header 'Access-Control-Allow-Origin' missing).1 <unknown>

DNT (and in Safari Mobile private mode) suppressed events as suspected on desktop UAs (Chrome and Firefox) and mobile (Safari Mobile). Without DNT events (had to leave private mode on Safari Mobile) came through as suspected.

The following queries might be copied and pasted out of chronological order, but no matter.

The funnel appeared to be working according to MariaDB:

MariaDB [log]> select event_event, count(*) from MobileWebLanguageSwitcher_15265240 where timestamp > '2016012923' group by event_event;
+--------------------------+----------+
| event_event              | count(*) |
+--------------------------+----------+
| exitModal                |        2 |
| languageButtonImpression |        3 |
| languageButtonTap        |        2 |
| languageListLoaded       |        2 |
| pageLoaded               |       36 |
| startLanguageSearch      |        1 |
+--------------------------+----------+

Looking at one of my several interactions, a full funnel worked.

MariaDB [log]> select * from MobileWebLanguageSwitcher_15265240 where event_funnelToken = '6a38ac19d0810b261454109295433' order by timestamp asc\G;
*************************** 1. row ***************************
                       timestamp: 20160129231455
                       userAgent: "Mozilla/5.0 (Macintosh; Intel Mac OS X 10.11; rv:44.0) Gecko/20100101 Firefox/44.0"
                         webHost: en.m.wikipedia.beta.wmflabs.org
                            wiki: enwiki
             event_beaconCapable: 1
                     event_event: pageLoaded
                 event_exitModal: NULL
               event_funnelToken: 6a38ac19d0810b261454109295433
event_languageButtonTappedBucket: NULL
     event_languageButtonVersion: NULL
             event_languageCount: NULL
    event_languageOverlayVersion: NULL
            event_languageTapped: NULL
                event_mobileMode: stable
  event_positionOfLanguageTapped: NULL
     event_primaryLanguageOfUser: NULL
       event_searchInputHasQuery: NULL
*************************** 2. row ***************************
                       timestamp: 20160129231504
                       userAgent: "Mozilla/5.0 (Macintosh; Intel Mac OS X 10.11; rv:44.0) Gecko/20100101 Firefox/44.0"
                         webHost: en.m.wikipedia.beta.wmflabs.org
                            wiki: enwiki
             event_beaconCapable: NULL
                     event_event: languageButtonImpression
                 event_exitModal: NULL
               event_funnelToken: 6a38ac19d0810b261454109295433
event_languageButtonTappedBucket: NULL
     event_languageButtonVersion: NULL
             event_languageCount: NULL
    event_languageOverlayVersion: NULL
            event_languageTapped: NULL
                event_mobileMode: stable
  event_positionOfLanguageTapped: NULL
     event_primaryLanguageOfUser: NULL
       event_searchInputHasQuery: NULL
*************************** 3. row ***************************
                       timestamp: 20160129231559
                       userAgent: "Mozilla/5.0 (Macintosh; Intel Mac OS X 10.11; rv:44.0) Gecko/20100101 Firefox/44.0"
                         webHost: en.m.wikipedia.beta.wmflabs.org
                            wiki: enwiki
             event_beaconCapable: NULL
                     event_event: languageButtonTap
                 event_exitModal: NULL
               event_funnelToken: 6a38ac19d0810b261454109295433
event_languageButtonTappedBucket: unknown
     event_languageButtonVersion: bottom-of-article
             event_languageCount: NULL
    event_languageOverlayVersion: NULL
            event_languageTapped: NULL
                event_mobileMode: stable
  event_positionOfLanguageTapped: NULL
     event_primaryLanguageOfUser: en-us
       event_searchInputHasQuery: NULL
*************************** 4. row ***************************
                       timestamp: 20160129231600
                       userAgent: "Mozilla/5.0 (Macintosh; Intel Mac OS X 10.11; rv:44.0) Gecko/20100101 Firefox/44.0"
                         webHost: en.m.wikipedia.beta.wmflabs.org
                            wiki: enwiki
             event_beaconCapable: NULL
                     event_event: languageListLoaded
                 event_exitModal: NULL
               event_funnelToken: 6a38ac19d0810b261454109295433
event_languageButtonTappedBucket: NULL
     event_languageButtonVersion: NULL
             event_languageCount: 119
    event_languageOverlayVersion: bottom-of-article
            event_languageTapped: NULL
                event_mobileMode: stable
  event_positionOfLanguageTapped: NULL
     event_primaryLanguageOfUser: NULL
       event_searchInputHasQuery: NULL
*************************** 5. row ***************************
                       timestamp: 20160129231645
                       userAgent: "Mozilla/5.0 (Macintosh; Intel Mac OS X 10.11; rv:44.0) Gecko/20100101 Firefox/44.0"
                         webHost: en.m.wikipedia.beta.wmflabs.org
                            wiki: enwiki
             event_beaconCapable: NULL
                     event_event: startLanguageSearch
                 event_exitModal: NULL
               event_funnelToken: 6a38ac19d0810b261454109295433
event_languageButtonTappedBucket: NULL
     event_languageButtonVersion: NULL
             event_languageCount: NULL
    event_languageOverlayVersion: NULL
            event_languageTapped: NULL
                event_mobileMode: stable
  event_positionOfLanguageTapped: NULL
     event_primaryLanguageOfUser: NULL
       event_searchInputHasQuery: NULL
*************************** 6. row ***************************
                       timestamp: 20160129231735
                       userAgent: "Mozilla/5.0 (Macintosh; Intel Mac OS X 10.11; rv:44.0) Gecko/20100101 Firefox/44.0"
                         webHost: en.m.wikipedia.beta.wmflabs.org
                            wiki: enwiki
             event_beaconCapable: NULL
                     event_event: exitModal
                 event_exitModal: dismissed
               event_funnelToken: 6a38ac19d0810b261454109295433
event_languageButtonTappedBucket: NULL
     event_languageButtonVersion: NULL
             event_languageCount: 1
    event_languageOverlayVersion: NULL
            event_languageTapped: NULL
                event_mobileMode: stable
  event_positionOfLanguageTapped: NULL
     event_primaryLanguageOfUser: NULL
       event_searchInputHasQuery: 1
6 rows in set (0.01 sec)

Looking at another interaction, repeat usage seemed to be tallied as well ("event_languageButtonTappedBucket: 1-4 taps").

MariaDB [log]> select * from MobileWebLanguageSwitcher_15265240 where event_funnelToken = '315902a3e1b28faf1454109605752' order by timestamp asc\G;
*************************** 1. row ***************************
                       timestamp: 20160129232006
                       userAgent: "Mozilla/5.0 (Macintosh; Intel Mac OS X 10.11; rv:44.0) Gecko/20100101 Firefox/44.0"
                         webHost: en.m.wikipedia.beta.wmflabs.org
                            wiki: enwiki
             event_beaconCapable: 1
                     event_event: pageLoaded
                 event_exitModal: NULL
               event_funnelToken: 315902a3e1b28faf1454109605752
event_languageButtonTappedBucket: NULL
     event_languageButtonVersion: NULL
             event_languageCount: NULL
    event_languageOverlayVersion: NULL
            event_languageTapped: NULL
                event_mobileMode: stable
  event_positionOfLanguageTapped: NULL
     event_primaryLanguageOfUser: NULL
       event_searchInputHasQuery: NULL
*************************** 2. row ***************************
                       timestamp: 20160129232117
                       userAgent: "Mozilla/5.0 (Macintosh; Intel Mac OS X 10.11; rv:44.0) Gecko/20100101 Firefox/44.0"
                         webHost: en.m.wikipedia.beta.wmflabs.org
                            wiki: enwiki
             event_beaconCapable: NULL
                     event_event: languageButtonImpression
                 event_exitModal: NULL
               event_funnelToken: 315902a3e1b28faf1454109605752
event_languageButtonTappedBucket: NULL
     event_languageButtonVersion: NULL
             event_languageCount: NULL
    event_languageOverlayVersion: NULL
            event_languageTapped: NULL
                event_mobileMode: stable
  event_positionOfLanguageTapped: NULL
     event_primaryLanguageOfUser: NULL
       event_searchInputHasQuery: NULL
*************************** 3. row ***************************
                       timestamp: 20160129232205
                       userAgent: "Mozilla/5.0 (Macintosh; Intel Mac OS X 10.11; rv:44.0) Gecko/20100101 Firefox/44.0"
                         webHost: en.m.wikipedia.beta.wmflabs.org
                            wiki: enwiki
             event_beaconCapable: NULL
                     event_event: languageListLoaded
                 event_exitModal: NULL
               event_funnelToken: 315902a3e1b28faf1454109605752
event_languageButtonTappedBucket: NULL
     event_languageButtonVersion: NULL
             event_languageCount: 119
    event_languageOverlayVersion: bottom-of-article
            event_languageTapped: NULL
                event_mobileMode: stable
  event_positionOfLanguageTapped: NULL
     event_primaryLanguageOfUser: NULL
       event_searchInputHasQuery: NULL
*************************** 4. row ***************************
                       timestamp: 20160129232205
                       userAgent: "Mozilla/5.0 (Macintosh; Intel Mac OS X 10.11; rv:44.0) Gecko/20100101 Firefox/44.0"
                         webHost: en.m.wikipedia.beta.wmflabs.org
                            wiki: enwiki
             event_beaconCapable: NULL
                     event_event: languageButtonTap
                 event_exitModal: NULL
               event_funnelToken: 315902a3e1b28faf1454109605752
event_languageButtonTappedBucket: 1-4 taps
     event_languageButtonVersion: bottom-of-article
             event_languageCount: NULL
    event_languageOverlayVersion: NULL
            event_languageTapped: NULL
                event_mobileMode: stable
  event_positionOfLanguageTapped: NULL
     event_primaryLanguageOfUser: en-us
       event_searchInputHasQuery: NULL
*************************** 5. row ***************************
                       timestamp: 20160129232350
                       userAgent: "Mozilla/5.0 (Macintosh; Intel Mac OS X 10.11; rv:44.0) Gecko/20100101 Firefox/44.0"
                         webHost: en.m.wikipedia.beta.wmflabs.org
                            wiki: enwiki
             event_beaconCapable: NULL
                     event_event: exitModal
                 event_exitModal: tapped-on-result
               event_funnelToken: 315902a3e1b28faf1454109605752
event_languageButtonTappedBucket: NULL
     event_languageButtonVersion: NULL
             event_languageCount: 119
    event_languageOverlayVersion: NULL
            event_languageTapped: de
                event_mobileMode: stable
  event_positionOfLanguageTapped: 9
     event_primaryLanguageOfUser: NULL
       event_searchInputHasQuery: 0
5 rows in set (0.00 sec)

Using Safari Mobile with DNT turned off and in non-private mode (both conditions must be met on this browser it seems), and doing repeated button taps showed the incrementing worked, based on a tail -f.

{"event": {"event": "languageButtonTap", ..., "languageButtonTappedBucket": "5-20 taps", ...}
dr0ptp4kt closed this task as Resolved.Jan 30 2016, 1:03 AM

Why are we worried about sampling rate on beta cluster? Surely testing it is more important and the traffic is tiny here as shown by the tiny amount of log entries since yesterday (?) despite 100% sampling.

I'd strongly advise we keep it at 100% while we test (note production is already setup at the much smaller sampling rate)