Page MenuHomePhabricator

Metrics to distinguish different types of Beta Features users
Open, Needs TriagePublic

Description

Smart metrics would distinguish between:

  1. Active users who opted into a specific feature.
  2. Inactive users (haven't edited recently, maybe haven't even logged in recently) who opt-ed into a specific feature.
  3. Active users who opted into the general "automatic enable" and are thus using your feature (haven't specifically opted out of that individual feature)
  4. Inactive users who opted into the general "automatic enable" and are thus using your feature.

If they have automatic enable and are using the feature, that may just mean they haven't gone in and disabled the feature, not necessarily that they love it. However, it does indicate it's not problematic/disruptive enough for them to disable it, which is still positive.

The top-line number of users using the feature is:

$total_using_feature_x = $total_explicitly_opted_into_feature_x + $total_users_with_automatic_enable_date_before_feature_x_launch_date_without_explicit_opt_out

Both of those figures are calculable:

$total_explicitly_opted_into_feature_x - This covers people with a 1 for that specific feature in the database. This is already kept track of. We just need to not increment this figure when someone is using a feature by virtue of "automatically enable".

$total_users_with_automatic_enable_date_before_feature_x_launch_date_without_explicit_opt_out - This can be calculated, either dynamically on every page view or on the job queue (not sure the performance requirements, but I know it can be done in SQL). It's basically a date comparison (a date for each user compared to a static date), plus a check that they don't have =0 for the feature (you can have automatically enable and still explicitly opt out to an individual feature).