Page MenuHomePhabricator

Webrequest data collection
Open, In Progress, Needs TriagePublic

Description

As part of work to understand readers from different referrers (particularly non-search engine referrers), we'd like to collect data on on-platform reading behavior akin to the work being done under T424862 and am asking @fkaelin to assist here as it makes sense to collect the same metrics.

Because this data would not be collected just for QuickSurvey respondents, it may make sense to collect data on a random sample of reading sessions.

Event Timeline

Data extraction notebook to

  • Load QS responses + initiations for the reader-comparative experiment over the configured wikis/period: pulls event.quicksurveysresponses and event.quicksurveyinitiation, with bot filtering and dedup by pageview_token.
  • Session reconstruction: joins surveyed pages to wmf.pageview_actor on (page_id, ip, hour) to recover actor_signature_per_project_family, pulls all pageviews ±1 day, computes 1h-gap session boundaries, and keeps only the session containing the surveyed page (enriched with page_title from mediawiki_page).
  • Enrichment: flags is_image_click via a Media Viewer signature in wmf.webrequest (custom UDF for actor signature, joined back to sessions); matches initiations to pageview_actor via strong (with page_id) / weak (60s tolerance) paths to recover referer_class and derives response_status (yes/no/ignored/other); joins research.article_topics (prob ≥ 0.5).
  • Write & explore: writes the three Iceberg outputs, then a polars + altair section computes per-session summaries and charts (length distribution, inter-pageview gap, entry referer, country mix, cross-wiki browsing, image-click share, and response rates by referer_class).