Change Details

==Description In order to drive innovation with timely data-driven decision making through [[ https://wikitech.wikimedia.org/wiki/Metrics_Platform | Metrics Platform ]]’s experimentation capabilities, we need to streamline the metrics collection and analysis process. This tasks captures the work of product team metrics requirements gathering that will define the next phase of work that will: - Establish standardized metrics definitions and calculations - Ensure compliance with privacy and other data retention policies - Reduce manual effort in the experimentation lifecycle ====Tasks (will need subtasks for each item) [] Create a list of all product team KR metrics included in annual plan [] Establish definitions and calculations for all additional product team metrics included in the annual plan [] Audit all pre-existing product team metrics definitions, calculations, and instruments used to gather those metrics [] Create a list of all pre-existing product team product health indicators (PHI) and success metrics based on instruments that are active on production and outlined in past measurement plans - Note instruments that are on production but are not actively used for PHI or data driven decision making. [] Document Standard metrics definitions and calculations [] Establish user segmentation definitions and criteria [] Audit current data sources and identify gaps [] Review data retention and deletion policies for blockers or required updates

==Context To build an effective experimentation platform (Experimentation Lab), we first need to understand how our teams currently measure success. This requires a thorough audit of all metrics, measurement tools, and data collection methods—both current and historical. By examining teams' existing behaviors and requirements, we'll identify where standardization is needed and what analytical capabilities are truly necessary. Teams are unlikely to adopt new methods if they don't align with their current practices or if the tools don't support a unified approach to experimentation. Understanding these dynamics will help us prioritize which features to build into the Experimentation Lab. Through this assessment, we'll discover the essential metrics and capabilities needed to foster a shared culture of experimentation, while ensuring we don't build more sophisticated analytical tools than teams can effectively use. ==Goal In order to drive innovation with timely data-driven decision making we will advance our annual plan [[ https://meta.wikimedia.org/wiki/Wikimedia_Foundation_Annual_Plan/2024-2025/Product_%26_Technology_OKRs#SDS_KRs | SDS 2 Objective ]] we will conduct a thorough audit of existing metrics, instrumentation, and team practices to inform the development of a unified experimentation framework that teams will readily adopt. This work will Kick off our next KR: [[ https://docs.google.com/document/d/1gitkTWiIUAtagPa--IpU1JPre4lqpMS8-SC6hHPmuQo/edit?tab=t.0 | SDS 2.4 ]] which is focused on burning down key [[ https://docs.google.com/spreadsheets/d/1ZDzE-BQsS70lW2ur55PQjnSSJFN8FHzsTYYQOETOM7g/edit?gid=1829099304#gid=1829099304 | adoption risks ]]. ====Tasks (will need subtasks for each item) [] Audit Current Metrics [] Compile list of all metrics from Annual Plan Process (APP) [] Document each metric's: [] Definition and calculation method [] Data sources and dependencies [] Teams using the metric [] Usage frequency and purpose (planning, KPIs, daily, weekly, monthly, etc.) [] Code/query references [] Industry standard vs. custom metrics [] Current importance rating for the teams using it and for the business operations [] Instrument & Data Source Analysis [] Map all existing instrumentation in production [] For each data collection point: [] Schema and repository location [] Data volume and growth trends [] Maintenance status (active/passive) [] Known quality issues [] Risk levels and SLOs [] Access controls [] Retention policies [] Update frequency (Real-time data collection, Daily batch processing ,Weekly aggregations ,Monthly reporting cycles ,Event-driven updates (triggered by specific actions) ,Historical changes to update frequency, Impact on data freshness requirements) [] Associated metadata [] Who owns/maintains the collection [] When it was implemented [] [] [] [] Document Standard metrics definitions and calculations [] Establish user segmentation definitions and criteria [] Audit current data sources and identify gaps [] Review data retention and deletion policies for blockers or required updates

==Description==Context In order to drive innovation with timely data-driven decision making through [[ https://wikitech.wikimedia.org/wiki/Metrics_Platform | Metrics Platform ]]’s experimentation capabilitieTo build an effective experimentation platform (Experimentation Lab), we first need to understand how our teams currently measure success. This requires a thorough audit of all metrics, we need to streamline the metrics collection and analysis process.measurement tools, This tasks captures the work of product team metrics requirements gathering that will define the next phase of work that will: - Establish standardized metrics definitions and calculationsand data collection methods—both current and historical. By examining teams' existing behaviors and requirements, we'll identify where standardization is needed and what analytical capabilities are truly necessary. - Ensure compliance with privacy and other data retention policiesTeams are unlikely to adopt new methods if they don't align with their current practices or if the tools don't support a unified approach to experimentation. Understanding these dynamics will help us prioritize which features to build into the Experimentation Lab. - Reduce manual effort in the experimentation lifecycleThrough this assessment, we'll discover the essential metrics and capabilities needed to foster a shared culture of experimentation, while ensuring we don't build more sophisticated analytical tools than teams can effectively use. ==Goal In order to drive innovation with timely data-driven decision making we will advance our annual plan [[ https://meta.wikimedia.org/wiki/Wikimedia_Foundation_Annual_Plan/2024-2025/Product_%26_Technology_OKRs#SDS_KRs | SDS 2 Objective ]] we will conduct a thorough audit of existing metrics, instrumentation, and team practices to inform the development of a unified experimentation framework that teams will readily adopt. This work will Kick off our next KR: [[ https://docs.google.com/document/d/1gitkTWiIUAtagPa--IpU1JPre4lqpMS8-SC6hHPmuQo/edit?tab=t.0 | SDS 2.4 ]] which is focused on burning down key [[ https://docs.google.com/spreadsheets/d/1ZDzE-BQsS70lW2ur55PQjnSSJFN8FHzsTYYQOETOM7g/edit?gid=1829099304#gid=1829099304 | adoption risks ]]. ====Tasks (will need subtasks for each item) [] Create a list of all product team KR mAudit Current Metrics included in annual plan [] Establish definitions and calculations for all additional product team metrics included in the annual plan [] Audit all pre-existing product team metrics definitions [] Compile list of all metrics from Annual Plan Process (APP) [] Document each metric's: [] Definition and calculation method [] Data sources and dependencies [] Teams using the metric [] Usage frequency and purpose (planning, KPIs, daily, weekly, monthly, calculations,etc.) [] Code/query references [] Industry standard vs. and instruments used to gather those metricscustom metrics [] Current importance rating for the teams using it and for the business operations [] Instrument & Data Source Analysis [] Map all existing instrumentation in production [] Create a list of all pre-existing product team product health indicators (PHI) and success metrics based on instruments that are active on production and outlined in past measurement plans [] For each data collection point: - Note instruments that are on production but are not actively used for PHI or data driven decision making. [] Schema and repository location [] Data volume and growth trends [] Maintenance status (active/passive) [] Known quality issues [] Risk levels and SLOs [] Access controls [] Retention policies [] Update frequency (Real-time data collection, Daily batch processing ,Weekly aggregations ,Monthly reporting cycles ,Event-driven updates (triggered by specific actions) ,Historical changes to update frequency, Impact on data freshness requirements) [] Associated metadata [] Who owns/maintains the collection [] When it was implemented [] [] [] [] Document Standard metrics definitions and calculations [] Establish user segmentation definitions and criteria [] Audit current data sources and identify gaps [] Review data retention and deletion policies for blockers or required updates