BetaFeatures: Track usage counts over time, for each feature
Open, LowPublic
Actions

Assigned To

None

Authored By

	Quiddity
	Jul 30 2015, 7:16 PM

Description

Then, as now, the "beta features" tab lists the number of users who have opted-in to a tool, but there is no comparative/objective explanation of what that actually means! For example, it tells me that 33,418 people have opted-in to "Hovercards", but is that good? How long did it take to reach that level? How many people have switched it off? What proportion of the active editorship is that? And most importantly - what relationship does this number have to whether Hovercards will 'graduate' or 'fall' the opt-in Beta process?

It would be useful for product managers (et al), to be able to track these stats over time. Ideally in the form of an area chart.

Event Timeline

Quiddity created this task.Jul 30 2015, 7:16 PM

Quiddity raised the priority of this task from to Needs Triage.

Quiddity updated the task description. (Show Details)

Quiddity added a project: BetaFeatures.

Quiddity added subscribers: Quiddity, Wittylama.

Restricted Application added a subscriber: Aklapper. · View Herald TranscriptJul 30 2015, 7:16 PM

Thanks for creating this @Quiddity!
Perhaps this can be broken down to sub-tasks. For opt-in numbers on Beta-features:

differentiate between those who chose this specific thing, and those who have opted to have all new features turned on automatically.
List the 'retention rate' of each feature. That is - what proportion of people who have tested this feature are still using it! This is an excellent quantitative measure of whether it is a net-benefit to users.
Indicate when the particular feature was made available in Beta-features, and the status of its development (e.g. 2-week turnaround for new feature iterations/no active development...)
Indicate what target is expected for this feature to either 'graduate' or 'fail' the beta process. NOTHING should be there permanently. By comparison, Vector skin targeted 80% retention. Think of it like the way crowdfunding websites give you a limited time period to raise money - the campaigns don't get to run indefinitely.
Indicate at the top of the page(?) how many active editors there are in total on the wiki, which gives the user a way to see if the number of users of a particular feature is actually a large number not. [Alternatively, indicate against each feature what proportion of the active editorship the usage number represents].

And finally, rollouts of new features/editor-facing changes should ALL [as far as practicable] go through some form of Beta process. It MUST be consistently used and with clear criteria for success/failure and methods of providing feedback. No one will be a Beta tester if they don't trust/understand the process.