Problem
our annual plan Key Results require measuring very specific outcomes (for example, constructive edits from editors with fewer than 100 edits), but it’s not currently easy to track those metrics. Even being able to see the total number of edits completed by users in the control and treatment groups during the experiment period would be a helpful indicator.
Previously we had considered having a hypothesis under WE 1.1 to the tune of
If we instrument constructive edits (produce "edit survived" events for edits which have been published and not reverted within 48h of their publication) in a way that allows us to count constructive edits made by users in experiments conducted through the Experiment Platform, we will be able to include this KR's metric in the automated analysis of controlled experiments we conduct.
But that might be over-engineering and we might not need it. This task is to investigate it for realsies.
Links/docs
- Experiment Platform Adoption Blockers (GSheet)
- Notes on experimentation with constructive edits/activation
Acceptance criteria
- Definitive answer to the question of can we offer it as a metric in the xLab automated analytics metrics catalog (and in the future the GrowthBook metrics catalog)
- EITHER documentation of why not and what we need to OR implementation and availability of it in the catalog