Background

In WE5, Product Analytics needs to be able to analyze API request logs and ultimately produce the API Analytics Dashboard, allowing API user segmentation and compliance to the User-Agent policy to be tracked over time. Another possible byproduct of this data is a pipeline for lead generation for Wikimedia Enterprise.

@KCVelaga_WMF has a series of notebooks which prototype logic for:

• Identifying and extracting API requests from webrequest
• Labeling requests using IP provenance
• Segmenting requests based on User-Agent string
• Pre-aggregate and compute certain metrics for WE5

On the Data Engineering side, we're interested in establishing patterns for how this analysis can be simpler without having to resort to working directly with webrequest, while striking a pragmatic balance between (1) separating data ingestion from business logic, (2) facilitating common operations and logic changes, and (3) producing a reusable data product for multiple domains.

@KCVelaga_WMF, @JAllemandou and I met this week and agreed that a good solution would be to provide a table of API requests, filtered down from webrequest, and populated with a few fields of general interest; we hope this will enable KC to build downstream datasets, private to Product Analytics at first, to feed into the API Analytics dashboard, future ad-hoc analysis, and eventual standardization.

Essentially, this is pageview_actor, but for API requests.

Acceptance Criteria

This is mostly a paraphrase of Joseph's scoping in the above linked document (thanks!)

• webrequest is extended with is_api: boolean and api_endpoint: boolean attributes, which will allow a consistent definition of API requests for any downstream data products.
  • The fields are populated based on the logic in this notebook.
• webrequest also includes IP provenance information as an ip_provenance: string column.
  • For this MVP, we use the existing GetNetworkOrigin UDF to populate it with wikimedia, wikimedia_labs, and internet labels.
  • As a next step after this MVP, we will integrate the work in T418466 to offer richer, freshly updated labels.
• An apirequest table is created by filtering webrequest based on the new boolean fields, containing all top-level columns in webrequest, and unpacking four x_analytics fields for ease of discovrey and analysis:
  • x_analytics[‘trusted_req’]
  • x_analytics[‘x_is_browser’]
  • x_analytics[‘rl_class’]
  • x_analytics[‘auth_type’]