Page MenuHomePhabricator

Get baseline data from iOS existing features for Watchlist
Closed, ResolvedPublic

Description

Background
For the watchlist feature that we are currently working on, we need metrics to inform our design decisions. This data will come from existing features in iOS. This data will be used to support our decision making.

The Task

  • How often are people clicking items in All Articles vs Reading List tab in iOS app? Bonus points if we have distribution of events in all articles vs reading list tab.
  • How often are users visiting diff pages
  • How often are users clicking affordances (buttons on the page) within the diff page
  • How often are users clicking the compare button
  • How often are users leaving the app to view contributions page
  • What is the monthly average edit rate in Northern and Western Europe and North America (US and Canada)
  • Distribution of installs to pageviews to edits in Northern and Western Europe and North America pre release vs post release

Requirements

  • Propose alternative questions to answer to have a base comparison for Watchlist
  • Update Instrumentation task as needed to collect information
  • Complete baseline metrics by the end of May

References
Link to Deepdive
Link to Editor Metrics

Event Timeline

@RWambua-WMF will update this task to put in what baseline metrics will be needed to acheive sldie 30 of our deep dive deck. Should be scaled down version of T329905.

RWambua-WMF renamed this task from Get Baseline metrics for watchlist to Get baseline data from iOS existing features for Watchlist .May 1 2023, 7:06 PM
RWambua-WMF raised the priority of this task from Medium to High.
RWambua-WMF updated the task description. (Show Details)

@RWambua-WMF and @JTannerWMF - see questions below:
Data availability notes:

For item:
How often are people clicking items in All Articles vs Reading List tab in iOS app? Bonus points if we have distribution of events in all articles vs reading list tab.

This is in progress, being instrumented currently.

For items:
How often are users visiting diff pages
How often are users clicking affordances (buttons on the page) within the diff page
How often are users clicking the compare button

Need to verify that these are the diffs being referenced in this ask, see schema deck: ios_edit_history_compare schema deck. This schema is operational on MEP as ios_edit_history_compare. The event counts are quite small - see current event counts daily

For buttons/engagement we can compare clicks on screens indicated in deck - show history, compare, compare after selecting edits, revision views and thank editor.

For item:
How often are users leaving the app to view contributions page

We need to clarify where this is in the app, not sure what contribution page we are referencing? We have counts for 'Users drills down to individual revision detail from full history page' and 'User drills down to individual rev detail from compare view' (see deck above). Is this what we want to know? If yes then we do have this data.

For item:
What is the monthly average edit rate in Northern and Western Europe and North America (US and Canada)?

Will have this data when iOS integration with EditAttemptStep schema is functional, currently in code review - see https://phabricator.wikimedia.org/T313561

For item:
Distribution of installs to pageviews to edits in Northern and Western Europe and North America pre release vs post release

We have this data from iOS App Store and pageviews_daily

Hey @SNowick_WMF thanks for doing this,

Need to verify that these are the diffs being referenced in this ask, see schema deck: ios_edit_history_compare schema deck. This schema is operational on MEP as ios_edit_history_compare. The event counts are quite small - see current event counts daily

Yes this is the diff that is being referred to, that you access through edit/revision history at the bottom of the article view. The screenshots in the presentation you have here are the same as the one in the iOS Watchlist shallow dive deck . Based on the spreadhseet you shared here it looks like we have our baseline, I am not too surprised the counts are small because the feature is really buried. With the introduction of watchlist, we believe these counts will increase because we will have a more exposed entry point that leads people to diffs. So this will be helpful in validating that theory and to verify things are working as expected.

For item:
How often are users leaving the app to view contributions page

We need to clarify where this is in the app, not sure what contribution page we are referencing? We have counts for 'Users drills down to individual revision detail from full history page' and 'User drills down to individual rev detail from compare view' (see deck above). Is this what we want to know? If yes then we do have this data.

When you are on a talk page and you click a user's name sometimes you go to their user page (it pushes you to mobile web) or sometimes it takes you to the contributions page (on mobile web). The new Diff will have a drop down like Android where users can choose to go to a user page, talk page, or view contributions. Initially in V1, user contributions will take you out of the app to mobile web. In V2. it will keep people in the app because we will build native contribution pages. You can test this out on your Android device on Watchlist by clicking a user's name. Below are screenshots of the interaction. So the question for now is did we instrument the button for contributions (a user name) when we built talk pages.

IMG_5705.PNG (2×1 px, 642 KB)

IMG_5706.PNG (2×1 px, 359 KB)

IMG_5707.PNG (2×1 px, 297 KB)

For the question of individual rev details, that info is going to be helpful but covered by the visits to diff pages because we are consolidating designs, there won't be "drill downs" per say anymore because there will be a continuous compare (Figma).

Thanks @JTannerWMF , this is helpful. Following up here with details on next steps based on your answers and my further research.

For item:
How often are users leaving the app to view contributions page

After discussion with Toni I verified that we are not currently tracking actual clicks on Article Talk or User Talk page links in the ios_talk_page_interactions schema, but we do track deeper interactions with Article Talk and User Talk pages ( open_topic, new_topic_click, reply_click, refresh, lang_change and submit) coming from the following sources ( talk_page, article, deep_link, notifications_center, account, search, in_app_web_view and unknown) See: T321841#8387540. We can extrapolate with the current data or we can add actual click tracking events to that schema - my opinion is we should add those click tracking events since the extrapolation may not answer design's baseline questions but if they do not deem it necessary we can defer. I can create a ticket for that based on your feedback.

We are going to move it to blocked until we are 3 weeks from Beta Release and need to get baseline data.

Lets start on this the week of June 7

We will begin pulling this data towards the end of July after the offsite

@RWambua-WMF @JTannerWMF
Want to very the syntax for this dataset:
"Distribution of installs to pageviews to edits in Northern and Western Europe and North America pre release vs post release" - are we meaning unique active devices by country or actual installs (which would be new and/or existing users as counted by the App Store having downloaded and installed the app in this time period)? I can get either user count but want to verify what we are talking about here. The Market Share analysis we have used in the past has Pageviews/Active Devices by Country (not Installs).

iOS Baseline data (using Active Devices):
Baseline Data

Note: Edit geodata not available, will add when issues with EditattemptStep are fixed. Also data for 'How often are users leaving the app to view contributions page' will not be available until most recent release.