Page MenuHomePhabricator

Advanced tracking for beta features
Closed, ResolvedPublic5 Story Points

Description

Please create a dashboard with two graphs (ideally a joint dashboard with T140229):

  • Graph 1: Shows the activity of users visiting the beta feature page (mw-prefsection-betafeatures).
    • Show the total number of page visits + total number of enablements per feature per day + total number of people not enabling any feature per day (this does not account for people who already have a beta feature enabled and who will therefore not enable it again, it also does not account for people always enabling two features together, but I think it suffices for a start)
    • If possible, mark at the graph when a feature was newly introduced. If you cannot do that on the graph, please show that information elsewhere on the dashboard for all existing beta features. If a beta feature has been removed from beta feature status, please still include it with all relevant info in the graph, as long as it was active in the time frame observed
  • Graph 2: Same as graph 1, but considering the disablement of beta features

Event Timeline

Restricted Application added subscribers: Zppix, Aklapper. · View Herald TranscriptJul 13 2016, 10:46 AM
Addshore set the point value for this task to 5.Jul 13 2016, 10:46 AM
Addshore added a subscriber: Addshore.EditedJul 13 2016, 10:56 AM

Show the total number of page visits

Should this be per wiki or aggregated?
Beta features are enabled on the page "Special:Preferences#mw-prefsection-betafeatures".
I'm not sure how exactly the best way to go about tracking access to this anchor on the preferences page would be.

total number of enablements per feature per day

Does this mean, of the users that viewed the page, how many enabled a feature (each user would only be counted once)?

total number of people not enabling any feature per day (this does not account for people who already have a beta feature enabled and who will therefore not enable it again, it also does not account for people always enabling two features together, but I think it suffices for a start)

So this would be people viewing the page that have no beta features enabled and do not enable any features on their visit?

If possible, mark at the graph when a feature was newly introduced. If you cannot do that on the graph, please show that information elsewhere on the dashboard for all existing beta features. If a beta feature has been removed from beta feature status, please still include it with all relevant info in the graph, as long as it was active in the time frame observed

Showing on the graph is probably blocked by T119754. We can always have a block of text on the dashboard with the information in until that is solved.

Graph 2: Same as graph 1, but considering the disablement of beta features

My initial reaction to points 1 and 2 is that this is a fairly large amount of work.

Graph 3: For all beta features, track how many users in total have enabled that feature

This requires a script very similar to https://github.com/wikimedia/analytics-wmde-scripts/blob/master/src/catwatch/userprops.php but counting beta feature user props instead.
Also, see betafeatures_user_counts table

Lea_WMDE renamed this task from Enable basic tracking for beta features to Advanced tracking for beta features.Jul 13 2016, 11:15 AM
Lea_WMDE updated the task description. (Show Details)

Created a new task for basic

Show the total number of page visits

Should this be per wiki or aggregated?

Ideally you would be able to switch between wikis and aggregated view I guess

total number of enablements per feature per day

Does this mean, of the users that viewed the page, how many enabled a feature (each user would only be counted once)?

Yes, that would be great!

total number of people not enabling any feature per day (this does not account for people who already have a beta feature enabled and who will therefore not enable it again, it also does not account for people always enabling two features together, but I think it suffices for a start)

So this would be people viewing the page that have no beta features enabled and do not enable any features on their visit?

The brackets were meant for the whole bullet point as a reminder what is still not perfect methodologically. The total number of people not enabling any feature per day for now would just be unique people viewing the page and not enabling anything.

We can always have a block of text on the dashboard with the information in until that is solved.

Sounds good!

Addshore moved this task from Unsorted 💣 to Next on the User-Addshore board.

@Lea_WMDE For the removals / additions do we want this to be for global users accross all sites? As is, if a user has the feature enabled on dewiki and on enwiki, but then disabled it on enwiki, should this count as a removal or not? As the user is still using the feature on some other wiki?

Good question. Ideally at some point we'd be able to split by wiki. Why do you think would people disable a feature on just one wiki? Would this be their primary wiki or rather a secondary wiki (if that exists)?

I can split all of the numbers tracked by wiki (but of course that adds complexity)

As for why would people disable the feature on one wiki and not another, I guess there are lots of cases, but I'm not sure how valuable my guesses are.

Then I would say go for the option that you prefer for now and we try to to find out why people enable features in just one wiki in the meanwhile

Change 301002 had a related patch set uploaded (by Addshore):
Track global user enables * disables of beta features

https://gerrit.wikimedia.org/r/301002

Addshore moved this task from ToDo to Doing on the WMDE-Analytics-Engineering board.

Change 301018 had a related patch set uploaded (by Addshore):
Track global user enables & disables of beta features

https://gerrit.wikimedia.org/r/301018

Change 301002 merged by jenkins-bot:
Track global user enables & disables of beta features

https://gerrit.wikimedia.org/r/301002

Change 301018 merged by jenkins-bot:
Track global user enables & disables of beta features

https://gerrit.wikimedia.org/r/301018

Addshore triaged this task as Normal priority.Jul 26 2016, 9:11 AM