Page MenuHomePhabricator

Research: How Many Users Get a Lot of Notifications?
Closed, ResolvedPublic

Description

GOAL
Proposed enhancements to the Notifications Page include search and filtering tools designed to aid users in processing their notifications (see a mockup). It's assumed these tools will be most valuable to users who receive a lot of notifications.

Figures for the aggregate volume of notifications are available. But we don't have good information on the volume of notifications received by individual users. Or, to rephrase that slightly, we don't know what proportion of users receive a significant volume of notifications. This information would be helpful as we attempt to gauge the need for the proposed sorting and searching tools.

SPECIFICS

In thinking through the goals and methodology for this research, the following was discussed:

  • Since editors don't necessarily edit every day, a week seems like a good period to measure. I.e., how many notifications do people get in a week?
  • Since Active Users represent less than 0.1% of registered users, it's understood that the vast bulk of users get few notifications. So the analysis here should focus on those users who get the most messages. I.e., a graph showing that 99.9% of users get fewer than 5 messages/week is not the point. More helpful is to know how many users get more than 25 messages/week (the number that the flyout menus can display)? And among that group, what is the distribution--how many get a little more and how many get a lot more?
  • Another data point worth having would be to know how many messages go unread. I'm not sure what approach to this would be the most helpful. A few that come to mind include:
    • A simple count of the distribution of unread messages, presumably at some point in time during the measured week but without restricting when the messages were generated.
    • A count that looks only at the messages generated during the week. So, of messages generated during the week, how many remained unread at the end of the week. We'd still want a distribution – how many users have a few, how many users have a lot?
    • Some middle ground that would look measure the number of messages generated during a preceding month.
  • Neil says it would be easier to restrict the study to a few wikis. We discussed the following:
    • Enwiki and Commons
    • Some wiki or wikis that implement Flow most thoroughly. Neil will consult Nick and Benoit on the best choices there.

DELIVERY DATES

  • First draft of the report is due Feb 12.
  • Final draft due Feb. 17.

Find the meeting notes where this project was discussed here: https://etherpad.wikimedia.org/p/notif-metrics

Event Timeline

jmatazzoni raised the priority of this task from to Needs Triage.
jmatazzoni updated the task description. (Show Details)

I'm planning to pull data for the following wikis:

  • commons
  • enwiki
  • jawiki
  • frwiki [heavy Flow use]
  • zhwiki [heavy Flow use]
  • mediawiki

using the following queries:

SELECT 
	DATABASE() as "wiki",
	`notifications`,
	 COUNT(*) as "users"
FROM
(
	SELECT
		COUNT(*) as "total notifications",
	FROM echo_notification
	WHERE
		notification_timestamp > "20160111" AND
		notification_timestamp < "20160210"
	GROUP BY notification_user
) notifications_by_user
GROUP BY `total notifications`;
SELECT
	DATABASE() as "wiki",
	`unread notifications`,
	 COUNT(*) as "users"
FROM
(
	SELECT
		SUM( IF( notification_read_timestamp IS NOT NULL, 1, 0) ) as "unread notifications"
	FROM echo_notification
	WHERE
		notification_timestamp > "20160111" AND
		notification_timestamp < "20160210"
	GROUP BY notification_user
) notifications_by_user
GROUP BY `unread notifications`;

Okay, I've finished my report. You can check it out on Github along with the source data.

This looks done to me, but feel free to reopen if you see something important I've missed.

nshahquinn-wmf added a subscriber: SBisson.

@SBisson has noticed an error in my data collection, so I'm going to re-run the numbers.

Removing Contributors-Analysis since it seems like there's nothing more for us to do here :)