Background

As part of the effort to address WMF’s decision-making challenges (FY24-25 Signals & Data Services, Objective 2), the Data Products team decided to conceptualize and implement a custom tool that provides Product Managers (aka experiment owners) with the ability to autonomously configure experiments and instruments.

Throughout the development of our custom tool, engineers and data analysts collaborated to ensure it aligned with our data infrastructure. This has resulted in a robust solution from a technical standpoint. However, as we move forward, we recognize the importance of integrating feedback from the tool's primary users— Product Managers. To ensure our solution fully meets their needs and expectations, we are eager to involve them in this next phase of evaluation and refinement. Their input will be invaluable in optimizing the tool’s functionality and usability.

This effort corresponds to the FY 2024 hypothesis SDS 2.1.4:
"If we conduct usability testing on our prototype among pilot users of our experimentation process, we can identify and prioritize the primary pain points faced by product managers and other stakeholders in setting up and analyzing experiments independently. This understanding will lead to the refinement of our tools, enhancing their efficiency and impact."

Goals

Primary goal

[Ease of use] Identify and prioritize the primary pain points faced by product managers at the Wikimedia Foundation when setting up experiments using the Metrics Platform Instrument Configurator.

Secondary goals

• [Suitability] Understand the degree to which MPIC is ready to be integrated in its target users’ experimentation workflows. How well does the tool meet the needs, expectations and use cases of product managers.
• [Recommendations] Detect key opportunities for improvement to inform new iteration on the tool’s design

Methodology

Two part/mixed sessions consisting of:

1. Semi-structured interview
2. Task-based exploration following think-aloud protocol

Participants

We want to gather feedback from (at least) 5 Product Managers, who’ll consolidate the role of experiment owners at Wikimedia Foundation.

Open questions and considerations

• As part of the effort to establish a standard experimentation system that addresses WMF’s decision-making challenges, the Data Products team also conducted extensive research to identify a 3rd party solution that complies with most core product and technical requirements. Recently, stakeholders delivered a decision brief recommending the adoption of GrowthBook. If the tool passed the pending rounds of evaluation (e.g. community consultation) and was approved by senior leadership, then we'd abandon our efforts to implement a custom-built experimentation solution.

Resources

• Research plan
• Raw data: Interview transcripts, MPIC UI/Interaction feedback

Acceptance criteria

• We extract insights that will allow us to assess the suitability of our internal experimentation solution, and optimize it based on the needs and mental models of its intended end users