Page MenuHomePhabricator

[Release] Confirm language overlay sampling on production working as expected
Closed, ResolvedPublic

Description

Changes to the language switcher roll out the week commencing 1st February.
Given the event logging will deploy to all wikis we should begin seeing data around 2nd/3rd February.

Let's confirm the sampling rate is as expected and that the data looks accurate.

Event Timeline

Jdlrobson raised the priority of this task from to Normal.
Jdlrobson updated the task description. (Show Details)
Jdlrobson added subscribers: bmansurov, phuedx, gerritbot and 9 others.
dr0ptp4kt renamed this task from Confirm sampling on production working as expected to [Release] Confirm language overlay sampling on production working as expected.Jan 30 2016, 2:52 PM
dr0ptp4kt set Security to None.

Change is live on production please run your tests.

Just a heads up, I started running some queries.

There definitely seem to be cases where there was an event for the impression of the button and there was an event for the language list having loaded, yet there was no intervening event for the button tap. This may be a reflection of a race condition, lack of sendBeacon support, or both.

I'm not yet willing to send this back for re-work, as the general ratios still may be sufficient for us to make some conclusions about general usage. But I'll provide an update next week. Here's the corresponding code in case anyone's interested to take a look at the button tap in the meantime.

https://github.com/wikimedia/mediawiki-extensions-MobileFrontend/blob/bb3c9a502b6cf5868e2737a0b37b98fe1a197b86/resources/skins.minerva.scripts/init.js#L103

dr0ptp4kt added a comment.EditedFeb 9 2016, 1:18 AM

Confining the query to sendBeacon capable UAs and setting the condition that a pageLoaded event must be received to initiate the funnel, for the stable channel with the simpler overlay, here's what's been observed thus far

select t1.event_event, t1.event_exitModal, t1.event_searchInputHasQuery, count(*)
from MobileWebLanguageSwitcher_15302503 t1
inner join
MobileWebLanguageSwitcher_15302503 t2 on t1.event_funnelToken = t2.event_FunnelToken
where t1.event_event in ('languageButtonImpression', 'languageListLoaded', 'startLanguageSearch', 'exitModal' ) and t2.event_event = 'pageLoaded' and t2.event_beaconCapable = 1 and t2.event_mobileMode = 'stable' group by t1.event_event, t1.event_exitModal, t1.event_searchInputHasQuery;

event_event	event_exitModal	event_searchInputHasQuery	count(*)
exitModal	dismissed	0	6
exitModal	tapped-on-result	0	38
languageButtonImpression	NULL	NULL	2888
languageListLoaded	NULL	NULL	61
startLanguageSearch	NULL	NULL	1

Stated differently, about 2.1% of users who saw the button made it to the language modal, with about a 62.3% tap through rate. From this group it would appear only 1 person (about 1.6% of 61 languageListLoaded) actually attempted to use the language search feature.

dr0ptp4kt added a comment.EditedFeb 11 2016, 2:57 PM

A few follow up comments.

  • I feel generally comfortable with the instrumentation at this point. I'll sign off.
  • It should be noted that in the case that the user navigates forward away from the modal, if the user navigates backward to the modal, the event funnel is still alive (as contrasted with explicitly dismissal of the modal). This isn't necessarily bad, it's just a thing to consider when doing analysis.
  • It seems that users didn't realize it was intended for speeding up the interaction, or perhaps didn't care about the language search feature in the modal. Either in the current work in T123384: Build language overlay design alternative or when T123384: Build language overlay design alternative is A/B tested (T123980: A/B test language switcher alternative) if it still shows low search language engagement we should consider making it more obvious somehow that the field is for searching or we should just get rid of the searching. People who are going to the modal are presumably at least somewhat interested in getting to a destination language, so it would seem logical that they might like the search capability in the list. But if it isn't the case, simplifying and getting back the screen real estate may be worth it.
dr0ptp4kt closed this task as Resolved.Feb 11 2016, 2:59 PM

It seems that users didn't realize it was intended for speeding up the interaction, or perhaps didn't care about the language search feature in the modal.

@dr0ptp4kt, I'm not sure if you already have considered this, but have you taken the number of languages into account? If the users saw about 5 languages per article then there is no need to search.

It seems that users didn't realize it was intended for speeding up the interaction, or perhaps didn't care about the language search feature in the modal.

@dr0ptp4kt, I'm not sure if you already have considered this, but have you taken the number of languages into account? If the users saw about 5 languages per article then there is no need to search.

Indeed. Is it worth showing the search bar, which takes up space, for small lists.

Good question.

select avg(event_positionOfLanguageTapped) from MobileWebLanguageSwitcher_15302503 where event_languageTapped is not null;
avg(event_positionOfLanguageTapped)

11.7025


select event_positionOfLanguageTapped, count(*) from MobileWebLanguageSwitcher_15302503 where event_languageTapped is not null group by event_positionOfLanguageTapped order by event_positionOfLanguageTapped asc;
event_positionOfLanguageTapped	count(*)
1	284
2	48
3	24
4	26
5	19
6	12
7	10
8	4
9	5
10	2
11	5
12	2
13	7
14	5
15	4
16	6
17	1
18	1
19	1
20	3
21	2
22	4
23	1
24	2
25	1
26	2
27	2
28	3
30	2
31	1
32	1
34	2
35	2
36	2
37	1
39	4
40	4
43	2
45	1
47	1
48	1
49	7
50	3
51	4
52	2
53	2
58	2
60	1
63	1
66	2
68	1
69	2
72	1
73	1
82	2
83	1
86	6
90	1
98	1
100	1
101	1
108	1
138	1
153	1
162	1
170	1
290	1

Certainly a lot of stuff was concentrated in the first several languages. A lot of it was also concentrated past the first several languages, too.

Adam do those values include preferred languages? It would be good to see those results without preferred languages and with. i.e. do preferred languages help users avoid having to search?

Adam do those values include preferred languages? It would be good to see those results without preferred languages and with. i.e. do preferred languages help users avoid having to search?

The schema doesn't expressly capture this, however it's able to be determined implicitly with a high degree of confidence.

Caveat: there appears to be a small bug in https://github.com/wikimedia/mediawiki-extensions-MobileFrontend/blob/b844b109e165e9e73d467b2575d489ec08df7f8f/resources/skins.minerva.scripts/init.js#L114 whereby a client that technically supports LocalStorage is sending a languageButtonTappedBucket value of unknown instead of 0 taps. I'll file a bug for that. The good thing is the actual LocalStorage committed value isn't wrong, it's just the value being reported to Event Logging. Anyway, it doesn't really matter for this analysis, because by definition we're interested in repeat users.

Looking at the distribution below, it's clear that repeat usage (i.e., taking into account the bug just mentioned, languageButtonTappedBucket not equal to unknown or 0 taps) is considerably more likely to result in tapping on the first element in the list, which is highly likely to be the previously tapped language. The very first language in the picklist is somewhere around 86% of taps for the repeat user versus about 40% for the unknown case.

This is a good sign. What will be interesting to see is whether we nudge up this engagement - and, for that matter, reduce unnecessary scrolling the first time people visit the modal - further with the structured overlay, which adds one more heuristic of device language plus the vertical separation and layout distinction.

Note: This query confines events to the stable channel, which at the moment is only using the simpler overlay. By observing the stable channel events, any potential bias from the beta channel is better avoided.

select t2.event_languageButtonTappedBucket, count(*) ct, t1.event_positionOfLanguageTapped
from MobileWebLanguageSwitcher_15302503 t1
inner join
MobileWebLanguageSwitcher_15302503 t2 on t1.event_funnelToken = t2.event_FunnelToken
where t2.event_event = 'languageButtonTap'and t1.event_languageTapped is not null
and t2.event_mobileMode = 'stable'
group by t2.event_languageButtonTappedBucket, t1.event_positionOfLanguageTapped order by t2.event_languageButtonTappedBucket, ct desc, t1.event_positionOfLanguageTapped;


event_languageButtonTappedBucket	ct	event_positionOfLanguageTapped

1-4 taps	59	1
1-4 taps	2	2
1-4 taps	2	4
1-4 taps	1	5
1-4 taps	1	13
1-4 taps	1	30
1-4 taps	1	37
1-4 taps	1	108
20+ taps	2	1
5-20 taps	19	1
5-20 taps	2	2
5-20 taps	1	36
unknown	52	1
unknown	16	2
unknown	13	3
unknown	9	5
unknown	6	6
unknown	4	4
unknown	3	7
unknown	3	20
unknown	2	14
unknown	2	43
unknown	1	8
unknown	1	10
unknown	1	13
unknown	1	19
unknown	1	22
unknown	1	24
unknown	1	28
unknown	1	35
unknown	1	39
unknown	1	48
unknown	1	52
unknown	1	60
unknown	1	68
unknown	1	69
unknown	1	72
unknown	1	73
unknown	1	82
unknown	1	83
unknown	1	90
unknown	1	100
unknown	1	138
unknown	1	153

Follow up note. On the stable channel, 1 out of 256 language tapthroughs (0.4%) had search in force.

select event_searchInputHasQuery, event_exitModal, count(*) from MobileWebLanguageSwitcher_15302503 where event_mobileMode = 'stable' and event_event = 'exitModal' group by event_searchInputHasQuery, event_exitModal;


event_searchInputHasQuery	event_exitModal	count(*)
0	dismissed	45
0	tapped-on-result	255
1	dismissed	6
1	tapped-on-result	1

I wonder how the results would change in the above query if we only considered overlays with, say, more than 10 languages (because scrolling through the 10 languages is arguably faster than typing on a mobile device). I hope the ratio will be something like 1 in 20?

As of now, it appears that's about 1 out of 130 (2 out of 260) on the stable channel.

select t1.event_exitModal, t1.event_searchInputHasQuery, count(*)
from MobileWebLanguageSwitcher_15302503 t1
inner join
MobileWebLanguageSwitcher_15302503 t2 on t1.event_funnelToken = t2.event_FunnelToken
where t1.event_event = 'exitModal' and t2.event_event = 'languageListLoaded' and t2.event_languageCount > 10 and t2.event_mobileMode = 'stable' group by t1.event_exitModal, t1.event_searchInputHasQuery;


event_exitModal	event_searchInputHasQuery	count(*)
dismissed	0	30
tapped-on-result	0	258
tapped-on-result	1	2