As a product owner, I want to understand the baseline language switching behavior on the mobile website and the impact of potential changes so that the change alternatives with the greatest evident impact are shipped and so that I can monitor the continued effectiveness of the changes for a period sufficient to guarantee effectiveness of the changes for users who use the feature.
- Confirmed on localhost
- Confirmed on beta labs
- Confirmed on mobile web beta channel
- Confirmed on mobile web stable channel
- Beta channel: 10% funnel eligibility when no beta enrollment CTA in force; when beta enrollment CTA in force, adjust lower (probably 1%) to balance coverage across multiple languages without growing table too fast
- Stable channel: 0.01% funnel eligibility
Note that sampling eligibility should only be considered for pages on which there is a language switcher button. This may create too sparse of data, but we'll adjust if needed.
Questions to be answered
- What's the likelihood of a loaded page actually having an alternative language?
- What's the impression rate for the language switcher button?
- What's the clickthrough rate for the language switcher button when shown?
- Given the user tapped the language switcher, is the user a repeat user of the switcher button (languageButtonTappedBucket 0, 1-4, 5-20, 20 or more; probably localStorage?)
- Which user language is primary on the user's device? (Note that the langwiki will already be part of the event capsule, but it may or may not be aligned with the user's primary language.)
- How many languages are available to users when the modal is first shown?
- How often do users use the language switcher search?
- How often do users tap a language from the language picker (irrespective of whether the language search was used)?
- What's the position of a language in the list when tapped?
- How many results were in the language list when the user took action to navigate away from the modal?
- Was a language search result set being used at the time of tapthrough/dismiss?
- How often do users dismiss the modal explicitly ('x' button)?
- How long does the funnel interaction take, from language switcher button shown to language tapped?
- How does behavior vary between beta channel and stable channel?
- Which version of the button / chrome is used to get to the language switcher?
- Which version of the language switcher modal is being used?
Important qualities of event logging
- Either events must only fire for sendBeacon capable UAs, or the first event in the funnel must indicate whether the UA is sendBeacon capable (this helps with examining leaky funnels when the user taps a link at the end of the funnel).
- To make the implicit explicit, there should be a unique funnel token for each eligible funnel. It does not need to be tied to the user's session ID. The funnel should end once the user leaves the modal (i.e., there's a relatively slim possibility that the user will have a new funnel - which would have a new unique value - on the same page).
- Capture the mobileMode on the first event in the funnel.
- Once a user is eligible for event logging on the page, all of the relevant events in the funnel should be sent as they occur, tied to the funnelToken.
Here were initial notes. They should be updated to reflect the criteria above. Language switcher. What we need to know: Spike: https://phabricator.wikimedia.org/T123382 When language button is available, do users see the language button? Do they tap it? On modal: Do they tap on the list? Do they use the search input? Do they get 0 results? (e.g. when typing "ja" for Japanese) Do they tap on results when they search? Is the prioritized language list effective? Is their language, as determined by Accept-Language (available in JS) available? What is their language? How long does it take to dismiss or tap on an option? Is there repeat usage? BE SURE: only log with sendBeacon capable UAs Characteristics: channel (beta, ...) Events: Page loaded BOOL (or INT if inexpensive) alternative languages available Language button impression Language button tap Version of button tapped (old vs new) List loaded Exit modal Dismissed modal or tapped on result Tapped on which language relative position of tapped result at that instant if search input has query Length of list at time language tapped User entered first char in language search input