Background
Scope
- Create and start populating an initial version of user_agents_info table
- Note: The initial version (more like a beta version) will not have all the fields populated.
- We will have some placeholder dimensions that will be populated later in July 2026.
Acceptance criteria
- DDL for new table, wmf_traffic.user_agents_info is defined and reviewed (aligned with KAPOW and potential future information to be added)
- Table wmf_traffic.user_agents_info is created
- Addition DAG task or a new DAG populates the table (compute_user_agent_compliance_hourly_dag.py)
Schema
| column name | description |
|---|---|
| user_agent | User-Agent string |
| source | Source dataset of the User-Agent. For example, webrequest or Dumps |
| user_agent_struct | User-agent (map processed to struct) with browser_family, browser_major, device_family, os_family, os_major, os_minor and wmf_app_version keys and associated values |
| contact_info | Struct of contact information with like {"email": "foo@bar.com", "url": "https://foo.bar.com", ...} |
| ua_policy_compliance_level | compliant, partially_compliant, non_compliant, non_compliant_placeholder, or browser_like |
| bot_category | will be based on Cloudflare's bot taxonomy, T429280#12037609 - will NULL for the initial version |
| identified_agent | Internally identified agent types, wikipedia_app, instant_commons, mw_foreign_repo etc. |
| request_count | total rows in the source where the UA appeared during that hour |
| cached_requests_proportion | proportion of requests to cached content |
| response_size_total | total bandwidth served to the User-Agent during that hour |
| cached_response_size_total_proportion | proportion of response size to cached content |
| distinct_ip_count | distinct IPs that have sent this UA (for the source in that hour) |
| distinct_uri_host | distinct hosts that the UA has hit during that hour |
| hour | The hour during which the User-Agent was seen, at the respective source |