@apaskulin have you started documentation of the data contract yet?

Clare pointed me to:
materialized schema
https://github.com/wikimedia/schemas-event-secondary/tree/master/jsonschema/analytics/product_metrics
https://github.com/wikimedia/schemas-event-secondary/blob/master/jsonschema/analytics/product_metrics/web/base/1.2.0.yaml
https://github.com/wikimedia/schemas-event-secondary/blob/master/jsonschema/analytics/product_metrics/app/base/1.2.2.yaml
various wikitech pages
https://wikitech.wikimedia.org/wiki/Metrics_Platform/Event_Schema
https://wikitech.wikimedia.org/wiki/Metrics_Platform/Decision_Records/Deprioritize_Custom_Data
https://wikitech.wikimedia.org/wiki/Metrics_Platform/Analytics/Fragments

@phuedx What are best steps to advance this SPIKE?

@phuedx What are best steps to advance this SPIKE?

We know who, what, when, and where training will happen

Who: Data Products members
What: … ?
When: That depends on what
Where: On-wiki and synchronous meeting(s)

I'm uncertain what the "what" is here? Do we have any other onboarding docs that we can replicate? Could we replicate the structures from other projects or teams?

While we're thinking about that there are a couple of tightly-scoped things that we could do to move things forward here:

1. Collect all the tasks associated with developing the monoschema
2. Collect all the tasks associated with moving from monoschema to the data contract
3. Conduct a brief (30-50 minute) interview with the folks who did the move
4. Collect all the tasks associated with developing an instrument with the data contract and create case studies from them

In T370183#10106621, @VirginiaPoundstone wrote:

@apaskulin have you started documentation of the data contract yet?

I've been thinking about this as three separate docs:

1. Data contract docs for instrument creators using the base schemas
   • Page: Metrics_Platform/How_to/Create_First_Metrics_Platform_Instrument#Specifying_event_data
   • Status: Content completed and reviewed
2. Data contract docs for instrument creators using custom schemas yet
   • Page: Metrics_Platform/How_to/Create_a_Custom_Schema
   • Status: Updates not started, tracked in T372681
3. Data contract docs for maintainers of the Metrics Platform
   • Some of what maintainers need to know may be covered in the docs for instrument creators using custom schemas, but there will probably be additional information and context that's only relevant to maintainers.
   • Status: Out of scope for T329506 but I can help with where this should live relative to the other MP docs

Yeah... I guess it does fall more towards maintainers...

but... it would be really great to have a single table where people can go and see all the fields, script names, description, null Y/N, object types and objects in one place. This is often referred to as a "data dictionary". Having a JSON (or something) example to see it in practice would be an added bonus. An entity diagram would be the cherry on top to help people understand how all the parts fit together.

Having the data dictionary would be a really useful point of entry and reference.

In T370183#10131911, @VirginiaPoundstone wrote:

Having the data dictionary would be a really useful point of entry and reference.

This should be fairly easy to generate from the schemas themselves as every field has inline documentation…

This should be fairly easy to generate from the schemas themselves as every field has inline documentation…

Totally agree here. We should avoid manually duplicating the information in the schema to maintain the schema as the single source of truth. However, we should be able to either use a tool to visualize the schema in a more readable format or use the schema to generate a wikitable. I'm tracking this in T372680

1. Some of this overlaps with T376841: Render human-readable schemas on schema.wikimedia.org
2. @VirginiaPoundstone and @JEbe-WMF to meet to refine the scope of this task given the above. We'll estimate afterwards
3. Moved to In Progress to reflect the need to have a meeting