Page MenuHomePhabricator

Collect baseline metrics around current navigation in the iOS app
Closed, ResolvedPublic

Description

In preparation for changes in the navigation of the iOS app we'd like to better understand the existing behaviors. Some initial questions this instrumentation could answer:

  • How often are each of the "main views" of the app used (the 5 tabs, the "Article view", and the Settings screen)?
  • What is the average/median time spent on each?
  • What are the most common "paths" through the tabs in sessions?
  • How often are each of the menu items in Settings accessed?

Event Timeline

JMinor triaged this task as Medium priority.Apr 20 2021, 9:25 PM
cmadeo updated the task description. (Show Details)

Initial coding is done, I will test and mark as ready for review tomorrow morning if all looks good.

In the meantime, @JMinor @cmadeo @SNowick_WMF I wanted to confirm some other navigation cases:

  1. Default selection - so far I am being very deliberate about listening for a UI tap event to log these calls. Should I also log these events if something is selected by default? For example, if the user launches the app and lands on the explore feed, "explore" will not get logged. But if they explicitly tap the Explore tab bar item, it will get logged. Similarly for "saved_all" - it will not be logged if it's pre-selected by default. It's only logged if the user taps "All articles" in the Saved tab.
  2. Programmatic navigation - I am not logging events if the user is automatically navigated somewhere, for example when deep linking into the app, or if something (like the variant onboarding modal) automatically takes them to a particular Settings screen. Is this okay or do we need to log in these cases too?

Thanks!

Per our meeting, lets keep this simple for this schema/iteration and not worry about opens/default screens or programmatic navigation. As a follow-up I will resuscitate https://phabricator.wikimedia.org/T198978 which defined potential tracking of opens and use of deep links, widgets and other ways of opening the app.

I have marked the PR as ready for review! I have just a couple of last FYI's for @SNowick_WMF :

  1. setting_usage_reports will only get sent if the toggle in Settings is turned on. Because, well, if they turned it off that means they don't want to send any analytics :) So that may affect counts.
  2. For setting_read_danger, that Settings cell actually only appears in our debug builds, so you shouldn't see any logs from TestFlight or AppStore builds with it. I still have it in there, it's not hurting anything, but I expect you'll see very low counts on it, basically from whenever the app is run directly through Xcode.

@SNowick_WMF if you haven't already started, please QA on TestFlight build 6.8.1 (1814), which is our (hopefully) last release candidate.

Verified that all events are showing counts in schema, no errors in Logstash.
Preliminary Superset Dashboard
Will re-check data before production release and post-release.

Looks good. I'll dive into the data with you and Carolyn once we've had some time to build up a good baseline.

Looking at the data in superset so far, there seem to be days on which the most popular tabs are showing "null" even when there is data for other areas (eg. settings_). For example on June 14, Search and other tabs show "null". Not sure if this is a superset thing or a data collection issue, but seems to be getting dropped somewhere.

https://superset.wikimedia.org/superset/dashboard/iosnavigationevents/

This issue was due to a low row limit setting in Superset. I increased the row limit to 50,000 in the Chart settings, data counts seem consistent now.
Did not see any data errors in Logstash

i-os-navigation-event-actions-2021-08-10T18-24-17.577Z.jpg (407×1 px, 140 KB)

Thanks for jumping on it!