Page MenuHomePhabricator

Evaluate and decide if we need to capture any usage data for post-release analysis of Mute feature
Closed, ResolvedPublic

Description

We'll want to talk about what types of metrics might give us information about if this feature is successful or not (or worth spending more time developing.) Some of this might already be queryable, but we'll need to list the questions we want to answer first, then decide how to get the data.

Questions we'll want to answer with data

This is just a laundry list, we'll need to refine it.

  • Is this feature being used?
    • Number of users who have opted-in to use this feature
    • Average number of usernames added
  • Is this feature affecting how users participate on the wiki?
    • Retention for users who use this feature.
    • Edits before and after enabling this feature.
    • Number of users who set the blacklist and are subsequently blocked.
  • Is this feature helping alleviate harassment?
    • Retention for users who use this feature
    • Survey these users?

We'll need to stick to the Privacy by Design Policy and Ryan Kaldari may need to walk us through any privacy landmines.

Event Timeline

TBolliger renamed this task from Evaluate and decide if we need to tracking any usage data for post-release analysis to Evaluate and decide if we need to capture any usage data for post-release analysis.Jun 21 2017, 12:36 AM

Productivity would be useful to look at. Change in the number or type of edits.

Niharika renamed this task from Evaluate and decide if we need to capture any usage data for post-release analysis to Evaluate and decide if we need to capture any usage data for post-release analysis of Mute feature.Jul 24 2017, 9:34 PM

Seems like there could be 2 strategies to get this information:

  1. Identify users who have used this feature and contact them privately. If this is outside our privacy policy, we could have a (temporary) option in preferences to "privately survey me about why I use this feature."
  2. Query this data directly. This would require us to know the date when users first added a username so we could query against their contributions so we could detect changes in their contribution or retention.

I see value in either strategy. But overall I don't think getting this data is worth spending too many cycles on — we should pick a low-effort direction that's within our privacy policy.

Danny and I talked and we agreed that all we need is:

  1. Number of Wikimedia users who have 1+ name on their blacklist
  2. Average number of users blacklisted (or maybe a distribution?)

Both of these are already possible to pull from the db