Page MenuHomePhabricator

Notifications impact analysis
Closed, ResolvedPublic

Description

Lets grab some basic info about the use of notifications after release:

  • How many users have opted to receive native push notifications?
  • What projects do people get the most notifications from?
  • Which types of notifications have people been receiving?
  • What percent of our DAU have taken any on the notifications screen?
  • Comparison of before and after release, is there a change in the number of edits tagged as iOS edits in the article or talk namespace? And how do that compare to the number users receiving notifications.

Additional information by project/wiki is also helpful to note.

Event Timeline

In preparation for this I pulled 2022 editing data for iOS app users from mediawiki_history using revison_tag = ios app edit. I expected to see edits on commonswiki, particularly because data on ios_notification_interaction shows the most frequent source of notifications is commonswiki, with the majority of those notification types are edit-user-talk (see chart).

Need to investigate, it seems most likely that iOS edits made on Commons are not being properly attributed to iOS users in the revison_tag field. The data I'm looking at is from 2022-04 and going back to 2021 edit data and shows no iOS edits on Commonswiki at all.

We are planning to look at correlation between notification engagement and editing activity post-notification engagement, will post findings here w/r/t iOS edit tracking on Commons.

How many users have opted to receive native push notifications?
We track device_level_enabled for users who interact with notifications (66.6% enabled) and all iOS users overall (3335, 5.1%) (from schema mobilewikiappiosuserhistory). There is a high count of notDetermined which I will be looking into.

Device_level_enabledunique users% users
authorized33555.2%
denied4510.7%
notDetermined6126194.2%

Data

What projects do people get the most notifications from?
Superset chart
The top 5 projects notification users are receiving and engaging with are:

WikiCount% Total Notifications
Commons Wikipedia247234.8%
English Wikipedia142719.9%
Italian Wikipedia4316.1%
Arabic Wikipedia3264.6%
French Wikipedia2393.4%

Which types of notifications have people been receiving?
Superset chart
The top 5 notification types users are receiving and engaging with are:

Notification_typeCount% of Total
edit-user-talk426159.7%
welcome104314.5%
thank-you-edit7743.2%
reverted2292.7%
edit-thank1922.1%

What notification actions are users most frequently taking?
Superset chart

Top 5 actions:

ActionCount% of total
mark_read485967.8%
user_talk116316.2%
sender_page2854.0%
article2623.7%
diff2503.5%

What percent of our DAU have taken any action on the notifications screen?

The percent of notification users averages 0.4% of total iOS DAU.

Data

Will post here with results of editing/notification correlation data analysis when completed.

Post-notification implementation editing analysis:
Looking at editing activity post-notification shows a median +10.4% difference in edits made month over month, across 114 different language wikis. This smooths out with more data, using an average of edits by wiki for all of 2022 prior to Notification release, the median increase is +2.3% difference. We will need to look at next month's mediawiki_history editing data to see if this is an ongoing increase or a temporary lift.
Charts and Data

An additional analysis of notification impact on editing both on and off app as a result of notification activity would be in order as well when we have more data.

Added editing data for 2022-06 to analysis, the average monthly edits did decrease somewhat but we still see an overall increase in edits across 44.4% of wikis, with a median .74% positive difference (the average difference is 3.4%).

Updated Data and Charts