Background/Goal

As outlined in T354779: [Sprint 10 Goal][SDS 2.5] Implement tooling to set and monitor baselines for SDS 2.5's success indicator metrics, in support of the Product + Technology FY 22/23 Annual Plan, the Signals and Data Services (SDS) 2.5 Steering Committee has agreed upon a set of leading indicators that we will use as success metrics to monitor incremental progress towards the KR.

The three success indicators are:

• decrease time to data collection
• decrease number of custom data fields across all extant instruments
• decrease the number amount of engineering steps days required to make instruments

In order to set the targets for these goals, we need to first establish baselines:

KR/Hypothesis(Initiative)

SDS 2.5

Four feature teams use shared tools to evaluate and improve user experiences based on empirical data from user interactions.

If we Establish baselines for Metrics Platform & Experimentation success indicators
we can set realistic targets and measure our progress over time
in order to demonstrate whether or not the work is producing a return on investment (ROI)

User Stories

As the Data Products Product Manager
I need to explore data about the current baselines for product team Experiment Owners to get data from beginning to end of the instrumentation process
so I can set achievable goals, establish realistic expectations for our work, and have a starting point from which to tell the story of our team's journey.

As a Data Products Engineer,
I need to automate ways to empirically measure across extant instruments
so I can help leadership set realistic goals and explore data that demonstrates the impact of our engineering efforts over time

As a Senior Product + Tech Leader
I need to view visualized trustworthy empirical data about what the current problems are
so I can ensure we are prioritizing high impact efforts that can produce a reasonable return on investment (ROI)

Acceptance Criteria

• We are able to explore and visualize baseline data by January 26th, 2024
• solution is simple and fits within the time frame
• Solution has a maintenance plan
• Data Quality has been vetted
• Staff privacy is protected

In scope

• access to data based on entire history of all extant instruments -per instrument and -in aggregate
• We know how many custom fields existed across all extant instruments as of July 1st 2023 and can track that number across time
• all data is available on a monthly, weekly, and annual basis
  • Questions these metrics will answer:
    • How long does it currently take to make an instrument?
    • How many custom data fields currently exist across all extant instruments
    • How long does it currently take to get data once an instrumentation plan is place and the feature(s) are ready to instrument?
    • Is there a difference in how long it takes to make an instrument that produces data depending on how long the engineer has been with the foundation, if so what is that difference while protecting staff privacy?

Stretch Goal

• access to data that tells the number of years engineers have been with the foundation to surface potential discrepancies in terms of engineering level but also amount of time required to onboard

Out of Scope

• no custom visualization tools
• no overly complex software solution to get this information.
• the data can be made publicly available (future goal)