Wow, @apaskulin, this is really clarify, well done, and helpful. I deeply appreciate the flexibility for change and growth in the platform as part of how you have defined the information architecture.

Clarification:

MPIC will replace the docs as the entry point for users creating instruments

Not sure this is not exactly right. Engineers will still need to add instrumentation to their code, @phuedx are the APIs all developers need to add instrumentation?

Some thoughts:

• Looking at the emergent product analytics glossary, I'm thinking about "create an instrument" vs measuring product health metrics (PHI). When conducting experiments, the PHI are the secondary and guardrail metrics that are monitored for unintended impacts beyond the primary metric. Because we haven't yet really landed what we mean when we say "instrument" and each team has a bespoke way of measuring PHI, this might be an opportunity to create a standard approach... but maybe it is too soon? CC @Sarai-WMF @mpopov in case they have feedback / thoughts about this.
  • user flow #1 goal: measure product health metrics
    • use client libraries to add instrument code based on measurement plan > configure instrumentation to start collecting data via MPIC UI > monitor instrument landing page (via MPIC catalog)
  • user flow #2 goal: conduct a controlled experiment
    • use client libraries to add instrumentation based on experimentation scorecard > configure experiment instrumentation to start collecting data on select percentages of traffic on select wikis via MPIC UI > monitor experiment progress on its landing page (via MPIC catalog)
• In the templates section we say "measurement plan" but I think we need to disambiguate between a "product health metrics measurement plan" and an "experiment scorecard" T374981: Create a draft Experimentation Scorecard CC @mpopov and @nettrom_WMF for their thoughts.

schema pages and contextual attributes page will be separate so they can be linked to from different sections in the MPIC forms

Nice! The sandbox page is a great way to make the yaml files. much easier to parse. I also appreciate the at a glance review of the the expectations and understanding of the data types similar to https://schema.org/Dataset. I especially like that the design allows us the ability to link to the relevant parts from MPIC and stay synced with the code. <3 CC @phuedx
for review.

Thanks for this awesome feedback, @VirginiaPoundstone!

Not sure this is not exactly right. Engineers will still need to add instrumentation to their code, @phuedx are the APIs all developers need to add instrumentation?

That's correct, engineers will still need to instrument their code. By "entry point", I meant that an ideal "information experience" for MPIC will be one where users can complete as much of their task as possible without leaving the site. So instead of having the process for creating an experiment in a Google Doc or wiki page, the steps would be integrated into the MPIC flow. In the short term while things are in development, we may still rely on the docs to help people get started, but our eventual goal would be to replace the "guide to measuring product health" and the "guide to conducting an experiment" with a link to MPIC. Does this align with the way you've been thinking about the tool? Sarai and I discussed this briefly a few weeks ago, but @Sarai-WMF please correct me if this isn't aligned with your thinking.

user flow #1 goal: measure product health metrics
user flow #2 goal: conduct a controlled experiment

These two flows make a lot of sense to me! I've updated the task description to include these.

I'd like to confirm that we want users to merge their instrumentation code before they fill out an MPIC form. In the non-MPIC flow, users merge their instrument code after deploying their stream config to avoid errors from trying to publish events to a nonexistent stream. Maybe this is no longer an issue because of how MPIC works?

I think we need to disambiguate between a "product health metrics measurement plan" and an "experiment scorecard"

Nice work on the scorecard! I added it to the task description.

The "measurement plan" comes from the Record of Data Collection Activity form and has become a pattern with recent instrumentation projects. I understood the intention of the measurement plan to be to document enough information about the proposed data collection in order to start an L3SC review if necessary. I think the measurement plan template could work for both product health metrics and experiments with a few adjustments. Then we could link to it in the scorecard for "Product Manager finalizes experimentation plan". Alternatively, we could create two separate templates: a product health measurement plan template and an experimentation plan template. I'm open to both options.

The sandbox page is a great way to make the yaml files. much easier to parse.

I've updated the task description with an update on this: Per Dan's feedback in T372680, he thinks it would be easier to add rendering of human-friendly versions of the schemas to schema.wikimedia.org directly instead of using a bot. If this works out, this would be even better than a bot! Feedback and ideas are welcome on T376841: Render human-readable schemas on schema.wikimedia.org.

eventual goal would be to replace the "guide to measuring product health" and the "guide to conducting an experiment" with a link to MPIC. Does this align with the way you've been thinking about the tool?

Understood. Yes, ideally, if we continue to custom build, this will be the goal once mature.

I'd like to confirm that we want users to merge their instrumentation code before they fill out an MPIC form. In the non-MPIC flow, users merge their instrument code after deploying their stream config to avoid errors from trying to publish events to a nonexistent stream. Maybe this is no longer an issue because of how MPIC works?

@cjming and @Sfaci can you confirm?

Then we could link to it in the scorecard for "Product Manager finalizes experimentation plan".

Let's start with this streamlined path and see if needs adjustment later.

I'd like to confirm that we want users to merge their instrumentation code before they fill out an MPIC form. In the non-MPIC flow, users merge their instrument code after deploying their stream config to avoid errors from trying to publish events to a nonexistent stream. Maybe this is no longer an issue because of how MPIC works?

@VirginiaPoundstone
Precisely @apaskulin and I ended up talking about this yesterday, but, just in case I'm going to put it here as well:
I would say the flow should be same. In the end MPIC is just a tool to configure the instrument (the stream mainly) and save that information somewhere. Somehow we could consider MPIC as a replacement for the configuration manual step we are doing currently. Users will have to deploy their instrument code anyway. It wouldn't make sense to run an instrument without having deployed their stream configuration.

In T372690#10231860, @VirginiaPoundstone wrote:

I'd like to confirm that we want users to merge their instrumentation code before they fill out an MPIC form. In the non-MPIC flow, users merge their instrument code after deploying their stream config to avoid errors from trying to publish events to a nonexistent stream. Maybe this is no longer an issue because of how MPIC works?

@cjming and @Sfaci can you confirm?

Sorry to jump in but I don't think your question has been answered @apaskulin / @VirginiaPoundstone:

The Event Platform Client (and Metrics Platform Client, by extension) will silently drop events that are submitted to streams that don't exist. A stream can exist without an instrument and vice versa.

In T372690#10245624, @phuedx wrote:

The Event Platform Client (and Metrics Platform Client, by extension) will silently drop events that are submitted to streams that don't exist. A stream can exist without an instrument and vice versa.

This is good to know! @VirginiaPoundstone, it looks like we can have users write their instrument code either before or after using MPIC, depending on the product experience you prefer. From a docs/UI perspective, the difference will be whether we direct users to write their instrument code at the beginning or end of the MPIC form.

For now, they should use the docs and write their instrument code before going to MPIC.

In the future, if we expand MPIC to include the pre-test checklist steps, this could change.

Great! The workflow docs match that flow.