Page MenuHomePhabricator

Verify beta release data for Metrics Platform Java client library 2.0
Closed, ResolvedPublic

Description

Follow up to T351292

Description

Once the relevant stream config has been deployed to production, verify that data is coming into Hive from the 4 Android article instruments that are submitting events via Metrics Platform Java client library 2.0.

2 of the instruments validate against the app base schema:

  • article_link_preview_interaction
  • article_toolbar_interaction

2 of the instruments validate against custom schemas:

  • article_toc_interaction
  • find_in_page_interaction

Developer Notes

An engineer with analytics-privatedata-users credentials needs to query the event table in Hive to verify that the following 4 tables have been created:

  • android_product_metrics_article_link_preview_interaction
  • android_product_metrics_article_toc_interaction
  • android_product_metrics_article_toolbar_interaction
  • android_product_metrics_find_in_page_interaction

For each of the above tables, query for the contextual properties in the provide_values key of the producers config (see https://gerrit.wikimedia.org/r/c/operations/mediawiki-config/+/980963/) and make sure data is being populated for each property, as well as the relevant InteractionData properties.

For the custom data tables (article_toc_interaction and find_in_page_interaction), check that the custom data submitted by the relevant article instruments are being populated as expected.

An engineer with no access to the above may still check the event submission rate to those streams using the EventGate Grafana dashboard: https://grafana.wikimedia.org/d/ZB39Izmnz/eventgate?orgId=1&var-service=eventgate-analytics-external&var-stream=android.product_metrics.article_link_preview_interaction&var-stream=android.product_metrics.article_toc_interaction&var-stream=android.product_metrics.article_toolbar_interaction&var-stream=android.product_metrics.find_in_page_interaction&var-kafka_broker=All&var-kafka_producer_type=All&var-dc=thanos&var-site=All

Acceptance Criteria

Required

  • If data looks good and is coming in as expected, provide Grafana dashboard links to the Android team (sample queries with a sampling of results would be nice too - as a csv or spreadsheet) showing average number of events per sec
  • If unexpected data is showing up or any other errors are occurring, create bug tickets to remediate. << T355360 filed

Event Timeline

I started looking at the data coming in since we removed the old monoschema config.

Sadly there is missing data from the ClientData data object and there is a discrepancy between the current MEP instrument and the MP data contract instrument for the user_agent_map data.

I created a spreadsheet to see the different result rows (and corresponding queries) from the Hive tables generated by each version of the instrument:

  • MEP: modern event platform - current instrument in production
  • MPdc: metrics platform data contract - in beta
  • MPmono: metrics platform monoschema - in beta but removed about a week ago - data still exists from November/December tho

https://docs.google.com/spreadsheets/d/1-sDP6zKWrAc8jJbeYdyst7AqfgjkZtQaavtweZkfa34/edit#gid=1794012242.

Based on initial findings, we need to investigate why the following data is missing from MPdc:

  • agent: app_theme, app_flavor, app_version
  • page: page_title, page_content_language, page_id, page_namespace_id
  • performer: performer_is_logged_in, performer_session_id, performer_pageview_id, performer_language_groups, performer_language_primary, performer_groups
  • user_agent_map: browser_family, browser_major, device_family, os_family, os_major, os_minor, wmf_app_version

While I can confirm that some data is coming thru (i.e. some agent properties, InteractionData properties, etc), there are nulls coming for the above data objects, the majority of which are specified by stream config.

I have created a new ticket to remediate these issues T355360

Until we resolve the missing data (or have a legit explanation for why it's missing), we should pause on Product Analytics validation.

On the upside, custom data appears to be populating against the custom schemas. I need to dig into these more to determine accuracy.

cc @WDoranWMF @VirginiaPoundstone

cjming updated the task description. (Show Details)
cjming updated the task description. (Show Details)