Page MenuHomePhabricator

WE5.5.3 - Trending Indicators
Closed, ResolvedPublic

Description

Multiple prototypes already implement different definitions of "trending" (wme_pageviews batch moments comparison from enterprise, ), and multiple consumers want signals that mean subtly different things to each. This task produces a project document framing the problem, the algorithmic options, and an evaluation framework — giving PMs a shared decision aid for turning product requirements into engineering categories.

  • Taxonomy of streaming-friendly trending definitions, with three worked examples from existing prototypes (batch moments comparison, streaming top-k, streaming state machine)
  • Architectural implications (streaming vs batch vs hybrid) and operational trade-offs
  • Evaluation framework: latency, quality, and operational-complexity criteria
  • Offline evaluation harness (replay datasets + simple batch reimplementations of online algorithms) to make comparisons tractable in the absence of labelled data

Non-goals: picking a single production implementation; bot detection; complex offline algorithms (learned models, post-processing).

Event Timeline

Progress

  • Wrote project sketch: taxonomy, architectures, evaluation plan, product requirements matrix
  • Started implementation of offline comparison pipeline (builds on WE4.3.4 timeseries work)

Lifecycle stage

  • Implementation. Late start in the quarter; design sketch complete, pipeline implementation underway.

Blockers and risks

  • None

Progress this week

  • Implemented batch pipeline producing a 1-minute-granularity pageview timeseries dataset over a one-month window (builds on WE 4.3.4 traffic-patterns work)
  • Implemented the batch moments comparison algorithm (based on WME project) across several configurations (10min vs 60min, 1hour vs 4days, etc.)
  • Trend detection events stored in a shared Iceberg table with algorithm and configuration metadata, enabling cross-approach comparison
  • Started notebooks for visualizing results and computing evaluation metrics in an algorithm agnostic way

Progress this week

  • Implemented all planned trending algorithms in the replay harness:
    • Z-score. Flags a page as trending when its last hour of traffic is a statistical outlier against the page's own multi-day baseline, normalized by project-wide traffic to cancel daily cycles. Four configs: wme (Enterprise algorithm: hourly over a 97-hour baseline, work in progress for MWP by DE), wme_batch (identical to wme but announcing 90 minutes late to simulate batch ingestion delay), fast (10-min window over a 1-hour baseline, sub-hourly spikes), daily (1-hour over 1-day)
    • Top-k (streaming popularity). Maintains the k most-viewed pages per project over a sliding window and emits an event only when a page enters or leaves the list — detects popularity rather than unusualness. Three configs: k100_30m (fast-moving, spike-driven), k100_4h (a spike must sustain to displace incumbents), k100_1d (continuous analog of the daily top-articles list).
    • State machine (streaming relative trending). A page moves through a sequence of states: cold -> candidate -> trending, firing once it shows sustained, low-variance growth above a traffic floor and ending when traffic sags back toward pre-trend levels. Three configs named after their detection-latency target: 10min, 30min, 1hour.
  • Built an exploratory UI on CloudVPS for comparing how the approaches behave over the same month of history:
    • Overview — at-a-glance scorecard per approach (volume flagged, noisiness, reaction speed vs. baseline) plus pairwise agreement.
    • Explore over time — move through the month, zoom from a full day down to an hour, see which articles each approach flagged and who caught them first.
    • By page — any article's traffic alongside when each approach said it was trending; drag to focus on a time window.
    • Top behavior — ranks the noisiest, strongest, and most distinctive detections per approach.
    • Note: the CloudVPS UI uses pageviews re-aggregated to hourly buckets (same as AQS). An identical internal-only version (tunnel required) shows the 1-minute granularity the algorithms actually run on.
  • Built a spreadsheet exporter for bulk trend viewing. Given how convenient UI prototyping has become, polishing the spreadsheet isn't worth pursuing; kept the export code for future bulk evaluation.

Progress this week

  • Met with Halley to walk through the exploratory UI and collect feedback.
  • Incorporated the feedback by splitting the UI into two views:
    • Advanced: the previous UI in full. Surfaces a lot of information that can be overwhelming, geared toward working close to the algorithms (e.g. seeing how configuration changes affect results).
    • Simple: essential information only, focused on a curated set of examples that show stakeholders the differences between algorithms and the trade-offs each entails.
    • The curated examples are produced via an annotation mechanism in the advanced UI: users can annotate specific pages and events with lines, boxes, explanations, and labels.
  • Refined the algorithm set: the "compare z-scores with two moving windows" approach is now split into two configurations — dpe_mwp (streaming MWP that is work in progress, can fire within 1 minute) and the original wme (batch, snapped to the hour with 90-minute ingestion delay).

Final update:

  • Project sketch: framework, shared language, and taxonomy of streaming-friendly trending definitions.
  • Data pipelines: repeatable PySpark notebooks that replay a set of trending algorithms and configurations against the same window of history.
  • Web tool: workbench to visualize and compare trend detection results, with a curated gallery of cases and observations, and an annotation feature to share observations and collect labels. code.
  • Project report: connects the original sketch to the workbench, walks through the tool's functionality, and proposes next steps.