Page MenuHomePhabricator

Improve web instrument event data data lake management
Open, Needs TriagePublic

Description

Problem

With the volume of data flowing into product_metrics_web_base Hive table from Test Kitchen-based instruments and experiments, it has become impossible to build Presto-powered dashboards in Superset on top of this huge table.

Any existing dashboards that were developed when there was less data flowing into it now frequently time out (often leaving behind ghost queries, cf. T418542).

Even though dbt has made it easier to create pipelines to aggregate/calculate metrics and store their values, we still need a way of quickly standing up a Superset dashboard, especially if we don't have all the requirements (yet) for the shape of the aggregated data, including but not limited to:

  • the exact metric definition (easy to modify in presto query, harder to redefine and backfill)
  • the dimensions by which the metric should be sliceable (easy to modify in presto query, harder to expand table)

Proposed solution

Essentially, a generalized version of T429051: Personal Dashboard health metrics events Iceberg table where instead of siphoning data from product_metrics_web_base for just one instrument, we siphon all instrument-produced data into an Iceberg table that uses days(timestamp) and instrument_name partitioning scheme.

We can then decommission the pipeline created in T429051 and update the queries in the Superset dashboard to use wmf_product.web_instrument_event_v1 (and specify instrument_name = 'personal-dashboard-health-metrics') instead of wmf_product.personal_dashboard_event_v1.

Once the data is pre-processed, it might also be easier/faster to derive aggregated datasets from it. That is, querying the derived Iceberg table is less computationally intensive than querying the raw product_metrics_web_base table.

Maintenance

Maintenance should have a configuration like:

maintenance:
    schedule: "@monthly"
    expire_snapshots:
      older_than_days: 90
    data_delete:
      enabled: True
      where: "timestamp <= TIMESTAMP '{{ data_interval_end | subtract_days(90) | to_dt() }}'"

Acceptance criteria

  • An hourly-updated Iceberg table wmf_product.web_instrument_event_v1 with proposed partitioning scheme and 90 day data retention policy

Event Timeline

Funny enough, @JAllemandou and I both independently arrived at this proposal and it was a delight to find ourselves totally in-sync on this.

Maybe/possibly. Depends on how custom that custom partitioning can be.

I guess we could have an event_iceberg.product_metrics_web_base table with partitioning by:

  • days(timestamp) or hours(timestamp) (potentially)
  • instrument_name

Crucially, for experiment data (where instrument_name is null) which also flows into the table, would that data live in an instrument_name=null partition(?) which is functionally the same as the wmf_experiments.experiment_event_v1 table that we currently siphon experiment data into?

would that data live in an instrument_name=null partition(?)

Hm! I don't know!

I asked that question not to block this task at all, but just to see if there is another potential reason that we should have already done Refine to Iceberg. We keep kicking the can because prioritiez. :)

This looks good to me, thanks for proposing!

So overall the story for experiment owners becomes:

  1. Create your experiment.
  2. Use wmf_product.web_instrument_event_v1 as the basis for prototype dashboards, queries on a notebook etc.
  3. Optionally, move those notebooks to dbt models to make a finalized, performant dashboard.

Not in scope for this ticket, but over time we can also think of other "default" data we can offer to experiment owners. Like, maybe it would be helpful to have some default aggregation (daily count of events)? But for that we should consider the bigger picture of why/when people create dashboards vs Growthbook, and what are the commonalities across experiments.

I asked that question not to block this task at all, but just to see if there is another potential reason that we should have already done Refine to Iceberg

We were also considering that, and here's some longer thoughts (Slack link)! While Refine To Iceberg might have helped, I'm leaning on the somewhat-philosophical idea that product_metrics_web_base is not a great target anyway, as we shouldn't be telling people to query it directly from a dashboard.

product_metrics_web_base is not a great target anyway, as we shouldn't be telling people to query it directly from a dashboard.

Yeah, that makes sense. And could be true even if it was an Iceberg table? Okay, thank you!

So overall the story for experiment owners becomes:

  1. Create your experiment.

Instrument :) Experiment data is already going into wmf_experiments.experiment_event_v1 to power GrowthBook analysis.

  1. Use wmf_product.web_instrument_event_v1 as the basis for prototype dashboards, queries on a notebook etc.
  2. Optionally, move those notebooks to dbt models to make a finalized, performant dashboard.

Yep!

Not in scope for this ticket, but over time we can also think of other "default" data we can offer to experiment owners. Like, maybe it would be helpful to have some default aggregation (daily count of events)? But for that we should consider the bigger picture of why/when people create dashboards vs Growthbook, and what are the commonalities across experiments.

Actually we're planning to do something like that but with a Test Kitchen-aware EventGate. We could have per-instrument and per-experiment event counters in Prometheus/Grafana to make it possible to monitor event volume in real time for newly launched instruments and experiments.

@GGoncalves-WMF: By the way, @SNowick_WMF would benefit from this solution greatly because she's been doing analysis of the mobile app CentralNotice banner campaigns and that data is in product_metrics_web_base and it's been really tough for her to work with. Even though that campaign wrapped up the mobile apps teams are planning to run more campaigns in the next FY, so this would bring a lot of value.

mpopov renamed this task from Improve instrument event data data lake management to Improve web instrument event data data lake management.Thu, Jul 16, 1:37 PM