If we:
- Define an experimentation-compatible, instrument-able version/proxy of idealized Weekly engagement count
- Create an instrumentation specification (a blueprint which specifies all events that an experiment must produce for the metric to be calculable)
- Implement the specification in a reusable way where possible
Then we:
- Would be able to easily produce all required events when implementing a DE1.2 experiment
So that:
- When we ship winning interventions (tested with experiments), they are likely to positively impact Weekly engagement count.
Why do this?
The DE1.2 KR's metric Weekly engagement count is idealistically defined as:
The count of all edits to any namespace in the past week, whether reverted or not + the count of all support actions in the past week.
Where "all support actions" includes Non-editing actions to track for Connection metrics (WMF internal), sourced from a variety of data sources including MediaWiki History, MediaWiki tables such as logging, and Event Registration application data (stored in wikishared). These data sources are not compatible with use in experiments conducted through Test Kitchen. For a metric to be analyzed in an experiment, that experiment must produce the required events that are attributed to experiment subjects – which we call instrumentation.
Not all of the listed non-editing actions may be instrumentable, so we may only be able to work with a subset of the actions that are used in calculation of the idealized version of the metric referenced in the KR text. The resulting metric would therefore be an approximation of the idealized version – a kind of proxy metric.
Furthermore, the metric's framing must be compatible with experimentation, which usually means normalized in some way. For example, "total permanent accounts created" is not an experiment-compatible metric, but permanent account creation rate is.
Finally, since we are planning to test multiple potential interventions within DE1.2 to try to move the Weekly engagement count needle, we need to instrument the metric in such a way that allows easy re-use of instrumentation across multiple experiments. (Fortunately the Experiment Platform is working on a pattern that facilitates this.)
Questions and curiosities
- Which non-editing actions are instrumentable? Is that subset sufficient for a metric which is intended to be an approximation of the idealized version?