Page MenuHomePhabricator

Analyse Android notifications release
Closed, ResolvedPublic

Description

The Android team released notifications at the same time as the newly redesigned navigation went out. We were targeting a:

  • 10% increase in Android contributor 7-day retention rate (people who contributed, got a notification, and contributed again within 7 days)
  • 10% increase in Android edit rate (edits/contributor)

for those who enabled and saw 1 or more notifications vs those who didn't. Did we achieve that? Did notifications have any other effect on editing or user retention?

Design deck and brief have been sent to @mpopov via email. @RHo is there anything else to add?

Event Timeline

thanks @Charlotte - nothing in terms of targeting, but it would be good to see if there are patterns in usage of Notifications itself (event logging from T201943). That is:

  • How many people turned on polling out of all those who saw Notifications?
  • Interactions with different notification types - how many Milestone, Welcome and Thank notifications were sent vs viewed?
  • Was there a particular notification type that shifted contribution/edit rate more than others?

Finally, tangentially it might be interesting if possible to see if notifications also lead to greater usage of the app in general (eg people opening the app to view notifications staying in-app afterwards).

Working on acquiring data for this and it turned out to be much, much harder and more involved than I anticipated. The analytics data we get from the app just has the notification ID, so I'm in the process of getting the Echo extension tables into Hive (that's the part that's causing problems & delays) so that I can get editing activity for users who got the notifications on Android.

I've come across a potential block and I would like some clarification. For context:

When double checking some sub-queries, I found something that's confusing me. The following Hive query (to look at the data on a small scale):

WITH android_notifications AS (
  -- This subquery refines Android app notification EL data
  SELECT
    event.app_install_id,
    CASE event.notification_type
         WHEN 'edit-thank' THEN 'thanks'
         WHEN 'thank-you-edit' THEN 'milestone'
         WHEN 'welcome' THEN 'welcome'
    END AS notification_type,
    event.notification_id,
    CASE event.action_rank
         WHEN 0 THEN 'mark as read and archive'
         WHEN 1 THEN 'primary'
         WHEN 2 THEN 'secondary (1)'
         WHEN 3 THEN 'secondary (2)'
    END AS notification_action,
    event.action_icon,
    event.client_dt
  FROM event.mobilewikiappnotificationinteraction
  WHERE revision = 18325732
    AND year = 2018 AND month = 11 AND day = 1
    AND event.notification_wiki = 'enwiki'
  LIMIT 10
)
SELECT
  client_dt AS date1, -- timestamp of interaction with notification in app/mobile OS
  notification_timestamp, -- timestamp of notification initially triggered by echo event
  event_id,
  notification_user AS user_id,
  app_install_id,
  notification_type,
  notification_action
FROM android_notifications
LEFT JOIN bearloga.echo ON (
  android_notifications.notification_id = echo.event_id
  AND echo.wiki_db = 'enwiki'
)
LIMIT 1000

yields the following results:

Screen Shot 2019-02-27 at 12.00.02 PM.png (429×1 px, 107 KB)

and the first thing you might notice is the difference in timestamps of the notification and the time the user interacted with it on their Android device. I'm not entirely sure who to ask about the Echo extension tables so I'm going with my best educated guess of @kaldari and @Catrope (please correct me if I'm wrong and you know who I should actually ask): am I correct in my understanding of the tables and their keys?

Dmitry told me that "[he knows] that the id is supposed to be the primary key of the notification in the actual database," so I just want to make sure that the reason for the differences in timestamps is because users are interacting with old notifications (e.g. from 2017) that they are just now seeing (e.g. in Nov 2018) on their devices (because maybe they don't edit on desktop/mobile web) and NOT because I've made a mistake with joining on the wrong keys.

Also if either of you maintain that extension (or know who is), can you please update (or ping someone to update) the descriptions of the fields in [[ https://www.mediawiki.org/wiki/Extension:Echo/echo_notification_table | echo_notification table ]] because currently none of them are described so I have to infer based on the column names in the schema.

Okay, Roan confirmed for me that's the correct interpretation.

Apologies for the lack of documentation in echo.sql. This patch adds documentation for all tables and fields.