Implement Analytics/Event logging instrumentation to measure new multilingual features
Closed, ResolvedPublic

Description

NOTE: This is a generic placeholder for analytics work to measure the multilingual feature, and may need to be broken up into different parts

How do we measure the success of multilingual features?

Given the overall goal is to 'improve the multilingual experience', success can be seen as satisfying the following:

1. Feed engagement from Multilingual users (who have more content to read)
  • Existing engagement metrics:
    • Increased daily average users (being able to access more content they find interesting)
    • Longer average time spent in the Explore Feed
    • Increase in interactions with feed content
  • New:
    • Usage of customizing the feed by language
2. Article reading engagement
  • Do people spend more time reading or view more pages from different languages?

Ideally we would compare depth/session length from users with 1 vs those who have >1 language set in the app

3. Adoption and usage of multiple languages feature

Prior to the release of new multilingual features, ideally baseline usage of the app language switching should be measured, as well as general Android device multilingual usage. See T190092: Selected languages stats on Wikipedia Android app

  • New metrics:
    • Usage of adding/changing app languages from Search vs Settings
    • For users who have >1 language added, what are the top languages and language combinations
RHo created this task.Mar 28 2018, 12:22 PM
RHo removed RHo as the assignee of this task.
RHo removed subscribers: Aklapper, RHo.
RHo updated the task description. (Show Details)Mar 29 2018, 1:34 PM
RHo updated the task description. (Show Details)Apr 5 2018, 3:04 PM

@mpopov just pinging you on the ticket per our discussion... will wait on your notes about the schemas and how to go about these...

mpopov added a comment.EditedMay 25 2018, 12:22 AM

Having reviewed this, I have the following recommendations:

  • For multilanguage search analytics I suggest two changes to the existing MobileWikiAppSearch schema (SearchFunnel):
    • Include a "langswitch" as an action
    • For every event Include a language as a field that would contain "{language}={position}" strings where {position} is the 0-based priority of the language in the user's settings (1st place, 2nd place, 3rd place, …), so if the user's language list is: German, English, then we would see values like "de=0" and "en=1". If they switch the order, we would start seeing "en=0" and "de=1" instead.
    • In case of event_action == 'langswitch', language would be "{previous language}>{new language}". So tapping on "English" when viewing German search results, the value would be "de>en"
  • For feed customization analytics it's less simple. We could add a language (or languages) field to the MobileWikiAppFeedConfigure schema (and update FeedConfigureFunnel accordingly). Then there are two options:
    • (A) Send multiple events, one for each language. language would have 1 "{language}={position}" string (e.g. 1 event containing the configuration for "de=0" and 1 event containing the configuration for "en=1") -- I'm not a big fan of this but if this is the much easier of the two approaches to implement, that's okay.
    • (B) Or send just one event with multilingual data. Since enabledList and orderList can be arbitrary strings, they could be JSON representations with the lists of values for each language. For example, orderList could have the value of {"en":[0,1,2,3,4,5,6,7,8],"de":[8,7,6,5,4,3,2,1,0]}. Then languages would be the JSON array of languages in the order they're set – e.g. ["de","en"] (notice that is where we have to make sure to have the correct order, the array in the enabledList & orderList fields can have any order) -- I would prefer if the instrumentation implemented this approach.
  • For session stats and daily install stats, we should expand MobileWikiAppSessions and MobileWikiAppDailyStats schemas to include a languages field which would have an (ordered) JSON array of languages such as ["de","en"]
  • We would use these updated funnels for all users, not just those with multiple languages.

Change 436717 had a related patch set uploaded (by Sharvaniharan; owner: Sharvaniharan):
[apps/android/wikipedia@master] [WIP] Multilingual Analytics:

https://gerrit.wikimedia.org/r/436717

@mpopov some things we are not taking care of :

  1. Language changes from 'Settings'
  2. Article reading engagement
  3. Do we want to keep track of addition of languages in anyway? Things like, did the user open the screen, but not add any new languages? or did they? or any other scenario more useful than this?
  4. Top language combinations? I remember seeing a ticket for this, but not sure.

@RHo or @Charlotte any thoughts? or do you want to have a short meeting to discuss this?

RHo added a comment.Jun 1 2018, 11:42 AM

hi @Sharvaniharan and @mpopov - imho definitely want to ensure language changes in via app language settings is measured as well as a few other items. More comments below inline but perhaps we can discuss details in a short chat today after stand-up?

@mpopov some things we are not taking care of :

  1. Language changes from 'Settings'

+1. Can we measure both the actions taken on the app languages settings screen (when languages added and when removed, # languages per install) as well as the entry point to the page? I.e., whether users went to update settings via:
(i) the onboarding screen,
(ii) the search screen (by tapping on the language code icon or "MORE" button), or
(iii) via the Settings screen.

  1. Article reading engagement
  • Yes. Ideally we would compare depth/session length from users with 1 vs those who have >1 language set in the app.
  • It would be great to know whether users switch between languages when reading using the Search or the 'Change language' icon in the article itself.
  1. Do we want to keep track of addition of languages in anyway? Things like, did the user open the screen, but not add any new languages? or did they? or any other scenario more useful than this?
  • I guess? Less of a priority than ensuring capture of actions performed in the screen.
  1. Top language combinations? I remember seeing a ticket for this, but not sure.
  • Yes, this was mentioned by Mikhail in T190092#4201717 so is it just a matter of checking it is instrumented(?) in the updated app language management behavior?
mpopov added a comment.EditedJun 4 2018, 6:49 PM

Thanks for the excellent feedback @Sharvaniharan & @RHo! Additional recommendations per our discussion:

Language selection funnel will need to be re-instrumented to replace MobileWikiAppLangSelect with MobileWikiAppLanguageSettings.

That schema has a generic extras field that would include additional info depending on context (action):

  • In case of action = 'start', extras would have the source from which the language selection dialog was initiated:
    • "settings"
    • "search" (the "More" button when searching)
    • "onboarding"
  • In case of action = 'search':
    • extras is "<cancelled>" if user backs out without typing anything in the search bar
    • extras is the search string if user searches for a language but doesn't add and just backs out (e.g. "Spanglish")
    • extras is "<added> if user concludes interaction with searching for languages by adding a language they found
  • In case of action = 'add', extras depends on whether they actually added any languages
    • If user didn't add any languages (change of mind or they searched for a language and didn't find the one they were looking for), we just set extras to "<cancelled>"
    • If user did add a new language, we use the language code. For example, if user currently has English & French and they add Spanish, extras is "es"
    • This event may be triggered either by user finding the language in list OR if they search for a language by tapping the magnifying glass + type the name of language + tap the language
  • In case of action = 'arrange', extras would have a JSON array of the new order of languages
    • So if user has languages "English" "French" "Spanish" and they make Spanish first, the string would be ["es","en","fr"]
  • In case of action = 'remove', extras would have a JSON array of removed languages
    • So if user removed English and French and just left Spanish, extras would look like ["en","fr"]
  • In case of action = 'back', extras is a JSON array of languages the user ended up with.
    • So in the case of the user I've been using in examples above, extras would be ["es"]
  • For this funnel I think the sampling rate should be 100%

Thanks for the excellent feedback @Sharvaniharan & @RHo! Additional recommendations per our discussion:

Language selection funnel will need to be re-instrumented to replace MobileWikiAppLangSelect with MobileWikiAppLanguageSettings.

That schema has a generic extras field that would include additional info depending on context (action):

  • In case of action = 'start', extras would have the source from which the language selection dialog was initiated:
    • "settings"
    • "search" (the "More" button when searching)
    • "onboarding"
  • In case of action = 'search':
    • extras is "<cancelled>" if user backs out without typing anything in the search bar
    • extras is the search string if user searches for a language but doesn't add and just backs out (e.g. "Spanglish")
    • extras is "<added> if user concludes interaction with searching for languages by adding a language they found

Just to be clear here... we are only mentioning 'added', and not which language was added?

  • In case of action = 'add', extras depends on whether they actually added any languages
    • If user didn't add any languages (change of mind or they searched for a language and didn't find the one they were looking for), we just set extras to "<cancelled>"
    • If user did add a new language, we use the language code. For example, if user currently has English & French and they add Spanish, extras is "es"
    • This event may be triggered either by user finding the language in list OR if they search for a language by tapping the magnifying glass + type the name of language + tap the language
  • In case of action = 'arrange', extras would have a JSON array of the new order of languages
    • So if user has languages "English" "French" "Spanish" and they make Spanish first, the string would be ["es","en","fr"]

Do we not want the old order here? something like ["en',"fr","es"] => ["es","en","fr"]

  • In case of action = 'remove', extras would have a JSON array of removed languages
    • So if user removed English and French and just left Spanish, extras would look like ["en","fr"]
  • In case of action = 'back', extras is a JSON array of languages the user ended up with.
    • So in the case of the user I've been using in examples above, extras would be ["es"]

Just confirming... This is triggered on exit of the add language screen, in all 3 scenarios - search, onboarding and settings?

  • For this funnel I think the sampling rate should be 100%
  • In case of action = 'search':
    • extras is "<cancelled>" if user backs out without typing anything in the search bar
    • extras is the search string if user searches for a language but doesn't add and just backs out (e.g. "Spanglish")
    • extras is "<added> if user concludes interaction with searching for languages by adding a language they found

Just to be clear here... we are only mentioning 'added', and not which language was added?

Yep, because searching for a language and then tapping one should trigger a separate action = 'add' event and that's the one that would have the language code of the newly added language. The action = 'search', extras = '<added>' event marks a successful language search.

  • In case of action = 'arrange', extras would have a JSON array of the new order of languages
    • So if user has languages "English" "French" "Spanish" and they make Spanish first, the string would be ["es","en","fr"]

Do we not want the old order here? something like ["en',"fr","es"] => ["es","en","fr"]

Oooh. Hm… OH! OHHHH! What if extras for action = 'start' is something like:

{"source":"settings","langs":["en","fr"]}

Then it's 100% okay for us just to know the new order of languages after every action = 'arrange' event, no need to have "from => to" because we have that starting point in the beginning.

  • In case of action = 'remove', extras would have a JSON array of removed languages
    • So if user removed English and French and just left Spanish, extras would look like ["en","fr"]
  • In case of action = 'back', extras is a JSON array of languages the user ended up with.
    • So in the case of the user I've been using in examples above, extras would be ["es"]

Just confirming... This is triggered on exit of the add language screen, in all 3 scenarios - search, onboarding and settings?

Yep! The tracking logic is the same no matter how the user got to the languages screen. There's nothing functionally different, right? For example, there aren't any interactions on that screen that are only possible if the user arrived at the language screen from onboarding.

mpopov added a comment.Jun 6 2018, 4:27 PM

Dmitry brought up some good points on using a summary event approach instead so I'll redesign the schema. @Sharvaniharan: I'll ping you when it's ready. I'm sorry you'll have to reinstrument.

No problem! will look forward to your updates!

mpopov added a subscriber: Dbrant.Jun 7 2018, 7:14 PM

Okie dokie, here's where I ended up:

  • Redesigned MobileWikiAppLanguageSettings to be a summary (schema doc should be self-explanatory but please let me know if you have any questions or concerns). Thanks for the recommendation, @Dbrant! I like this approach way better and will remember it in the future.
  • Designed MobileWikiAppLanguageSearching which summarizes interactions with the search screen (also like Dmitry suggested).
  • The two are linked by a session_token which is generated at the beginning of user's interactions with the language settings screen and is then passed on to the language searching funnel. (There is a 1:many relationship between MobileWikiAppLanguageSettings summary events and MobileWikiAppLanguageSearching summary events.)

What do you think @Sharvaniharan (& @Dbrant)?

@Sharvaniharan: the schemas have been updated according to recommendations above. Please verify that the schemas and the instrumentation agree.

Here are the new revision IDs:

Also, I am super sorry for my brain hiccuping when writing this:

Having reviewed this, I have the following recommendations:

  • For multilanguage search analytics I suggest two changes to the existing MobileWikiAppSearch schema (SearchFunnel):
    • For every event Include a language as a field that would contain "{language}={position}" strings where {position} is the 0-based priority of the language in the user's settings (1st place, 2nd place, 3rd place, …), so if the user's language list is: German, English, then we would see values like "de=0" and "en=1". If they switch the order, we would start seeing "en=0" and "de=1" instead.

Haha, whoops. I think my brain was thinking about another funnel at the time and mistakenly wrote that because the order of languages doesn't matter here. What I actually should have written is: in the search funnel, all events that aren't action = 'langswitch' should just have the language code of the currently looked at set of results. So sorry!

Thanks for this Mikhail! Here are a few questions:

  1. Is one delete one interaction count or one count per language deleted
  2. What is the searched boolean in MobileWikiAppLanguageSettings keeping track of?
  3. Are we no longer tracking rearrange, search_cancelled, other than counting them as interactions? [They can be deduced, but they aren't getting logged as their own events within the parent event]
mpopov added a comment.Jun 8 2018, 4:20 PM
  1. Is one delete one interaction count or one count per language deleted

One interaction count. This also brings up the question of how to treat non-modifying interactions (going into remove languages mode or going into add language mode and backing out).

  • Option 1: ignore them and don't increment interaction counter unless user added/removed languages
  • Option 2: increment interaction counter either way

@Sharvaniharan @Dbrant and I will discuss this later today.

  1. What is the searched boolean in MobileWikiAppLanguageSettings keeping track of?

Just a flag if the user has searched for languages at least once during their time on the language settings screen.

  1. Are we no longer tracking rearrange, search_cancelled, other than counting them as interactions? [They can be deduced, but they aren't getting logged as their own events within the parent event]

Correct, they're not logged as their own events anymore.

@Sharvaniharan: I just remembered the analytics for the feed! I've updated the MobileWikiAppFeed schema (Revision 18115458) to include a language field, which should contain a language code for action = 'cardShown' and action = 'cardClicked' events. Also ,per T196783, the following field names have been renamed:

former namenew name
appInstallIDapp_install_id
tsclient_dt
sessionTokensession_token
timeSpenttime_spent
  1. Is one delete one interaction count or one count per language deleted

One interaction count. This also brings up the question of how to treat non-modifying interactions (going into remove languages mode or going into add language mode and backing out).

  • Option 1: ignore them and don't increment interaction counter unless user added/removed languages
  • Option 2: increment interaction counter either way

    @Sharvaniharan @Dbrant and I will discuss this later today.

Just to document on the ticket, we decided not to count the non-modifying interactions. In other words, going with Option 1.

Sharvaniharan added a comment.EditedJun 11 2018, 9:14 PM

@Sharvaniharan: I just remembered the analytics for the feed! I've updated the MobileWikiAppFeed schema (Revision 18115458) to include a language field, which should contain a language code for action = 'cardShown' and action = 'cardClicked' events. Also ,per T196783, the following field names have been renamed:

@mpopov For the cards that do not have different languages, like the FeaturedImageCard, should it be null?

@mpopov For the cards that do not have different languages, like the FeaturedImageCard, should it be null?

Yes, please.

  • (B) Or send just one event with multilingual data. Since enabledList and orderList can be arbitrary strings, they could be JSON representations with the lists of values for each language. For example, orderList could have the value of {"en":[0,1,2,3,4,5,6,7,8],"de":[8,7,6,5,4,3,2,1,0]}. Then languages would be the JSON array of languages in the order they're set – e.g. ["de","en"] (notice that is where we have to make sure to have the correct order, the array in the enabledList & orderList fields can have any order) -- I would prefer if the instrumentation implemented this approach.

Currently, the enabledList & orderList are just plain strings. Do you need them to be valid json? in which case, should it be a string:string hashmap looking like {"en":[0,1,2,3,4,5,6,7,8],"de":[8,7,6,5,4,3,2,1,0]} , or JSONArray of JSONArrays, looking like this {"en":["0", "1" , "2" , "3" , "4" , "5" , "6" , "7" , "8" ] , "de":[ "8" , "7" , "6" , "5" , "4" , "3" , "2" , "1" , "0"]}?

mpopov added a comment.EditedJun 12 2018, 10:23 PM
  • (B) Or send just one event with multilingual data. Since enabledList and orderList can be arbitrary strings, they could be JSON representations with the lists of values for each language. For example, orderList could have the value of {"en":[0,1,2,3,4,5,6,7,8],"de":[8,7,6,5,4,3,2,1,0]}. Then languages would be the JSON array of languages in the order they're set – e.g. ["de","en"] (notice that is where we have to make sure to have the correct order, the array in the enabledList & orderList fields can have any order) -- I would prefer if the instrumentation implemented this approach.

Currently, the enabledList & orderList are just plain strings. Do you need them to be valid json? in which case, should it be a string:string hashmap looking like {"en":[0,1,2,3,4,5,6,7,8],"de":[8,7,6,5,4,3,2,1,0]} , or JSONArray of JSONArrays, looking like this {"en":["0", "1" , "2" , "3" , "4" , "5" , "6" , "7" , "8" ] , "de":[ "8" , "7" , "6" , "5" , "4" , "3" , "2" , "1" , "0"]}?

Thanks for checking! [0,1,2,3,4,5,6,7,8] (for orderList and {"en":[0,1,…1,0],"de":[1,1,0,1,…,1]} for enabledList), please.

@mpopov you should be seeing this from patch set 18:

Change 436717 merged by jenkins-bot:
[apps/android/wikipedia@master] Multilingual Analytics:

https://gerrit.wikimedia.org/r/436717

Change 441249 had a related patch set uploaded (by Bearloga; owner: Bearloga):
[apps/android/wikipedia@master] Update Search funnel rev ID

https://gerrit.wikimedia.org/r/441249

Change 441249 merged by jenkins-bot:
[apps/android/wikipedia@master] Update Search funnel rev ID

https://gerrit.wikimedia.org/r/441249