==Context
To build an effective experimentation platform (Experimentation Lab), we first need to understand how our teams currently measure success. This requires a thorough audit of all metrics, measurement tools, and data collection methods—both current and historical. By examining teams' existing behaviors and requirements, we'll identify where standardization is needed and what analytical capabilities are truly necessary.
Teams are unlikely to adopt new methods if they don't align with their current practices or if the tools don't support a unified approach to experimentation. Understanding these dynamics will help us prioritize which features to build into the Experimentation Lab.
Through this assessment, we'll discover the essential metrics and capabilities needed to foster a shared culture of experimentation, while ensuring we don't build more sophisticated analytical tools than teams can effectively use.
==Goal
In order to drive innovation with timely data-driven decision making we will advance our annual plan [[ https://meta.wikimedia.org/wiki/Wikimedia_Foundation_Annual_Plan/2024-2025/Product_%26_Technology_OKRs#SDS_KRs | SDS 2 Objective ]] we will conduct a thorough audit of existing metrics, instrumentation, and team practices to inform the development of a unified experimentation framework that teams will readily adopt. This work will Kick off our next KR: [[ https://docs.google.com/document/d/1gitkTWiIUAtagPa--IpU1JPre4lqpMS8-SC6hHPmuQo/edit?tab=t.0 | SDS 2.4 ]] which is focused on burning down key [[ https://docs.google.com/spreadsheets/d/1ZDzE-BQsS70lW2ur55PQjnSSJFN8FHzsTYYQOETOM7g/edit?gid=1829099304#gid=1829099304 | adoption risks ]].
====Tasks
(will need subtasks for each item)
[] Audit Current Metrics
[] Compile list of all metrics from Annual Plan Process (APP)
[] Document each metric's:
[] Definition and calculation method
[] Data sources and dependencies
[] Teams using the metric
[] Usage frequency and purpose (planning, KPIs, daily, weekly, monthly, etc.)
[] Code/query references
[] Industry standard vs. custom metrics
[] Current importance rating for the teams using it and for the business operations
[] Instrument & Data Source Analysis
[] Map all existing instrumentation in production, include:
[] Schema and repository location
[] Data volume and growth trends
[] Maintenance status (active/passive)
[] Known quality issues
[] Risk levels and SLOs
[] Access controls
[] Retention policies
[] Update frequency (Real-time data collection, Daily batch processing ,Weekly aggregations ,Monthly reporting cycles ,Event-driven updates (triggered by specific actions) ,Historical changes to update frequency, Impact on data freshness requirements)
[] Associated metadata
[] Who owns/maintains the collection
[] When it was implemented
[] Last modification date
[] Planned sunset date (if any)
[] Schema versions
[] Data validation rules
[] Processing pipeline details
[] Error handling procedures
[] Dependencies on other systems
[]
[]
[]