Page MenuHomePhabricator

WE4.3.4 Traffic pattern similarity search
Closed, ResolvedPublic

Description

To facilitate investigations of automated traffic that pose a risk to WMF infrastructure, we aim to to build a system that enables searching and clustering of similar traffic patterns.

  1. time series dataset: a pipeline that transforms the raw webrequest logs into a time series dataset that streamlines analysis, reduces computational cost, and provides consistent cost metrics. That datasets provides value for related projects (bot detection/pageview metric quality, combat scraping, content reuse, user segmentation)
  2. similarity-search service that identifies actors with matching traffic patterns, using compact frequency-domain embeddings generated from the time series dataset, to support investigations of automated behavior that causes issues for WMF infrastructure

Details

Related Changes in GitLab:
TitleReferenceAuthorSource BranchDest Branch
dding: traffic pattern streamingrepos/releng/gitlab-trusted-runner!152fabfab-main-patch-93629main
Customize query in GitLab

Event Timeline

Weekly update:

  • Backfilled 1 month of webrequest times-series for the analysis. the data pipeline is an hourly job generating an incremental iceberg table (partitioned by day). The backfill job is configured to run daily batches (<1hr to run, generating <200GB of data, compression to webrequest ~4x)
  • Started experimentation with configuration for fft analysis, length of window and timebucket. For the previous research poc I used 1 day of data in 5min buckets. There will be similarity search indices for different configurations. Initially similarity search will be an exploratory data tool; though we should work on evaluation methods (labelled data from the analysis Hamid&KC are doing for WE5.4.11 High-Confidence Signals for Automated DDoS Mitigation), or using spur data as label).

Weekly update

  • Merged code in research-datasets to build similarity-search indices on Ceph S3 and run large-scale evaluation with PySpark.
  • Merged a vector search service code exposing similarity-search endpoints over Ceph-backed indices; currently running on a stat machine.
  • Met with DPE SRE and confirmed a path to deploy the traffic pattern similarity search work. I’ll also write up Ceph S3 use cases & feedback to support upcoming security-model and Hadoop to Ceph/K8s migration design work.

Weekly update

Progress

  • Built UI for traffic pattern similarity search: takes a grouping column (e.g. IP) and similarity table (e.g. ESD) as inputs, returns nearest neighbors with raw timeseries, FFT, and ESD charts. The ui also shows metadata about the ip from ipoid service.
  • Implemented Spark pipeline for signal processing (FFT/ESD), tuned to handle 6-hour windows at 1-minute granularity.
  • Extended similarity tables and UI to support additional groupings (starting with ja3n).

Working through Blubber/GitLab CI config for Docker image builds and Helm chart for DSE cluster deployment.

Next

  • Continue deployment chart work: AWS secrets on instance, and CAS authentication (stretch).
  • Pick a time window of interest (Feb 20th incident is the current candidate) and build a labeled dataset of suspected bad actors.

Progress

  • Added Wikimedia deployment pipeline and Blubber integration for the vector-search repo; production images are now published to the WMF Docker registry.
  • Manually deployed vector-search service on DSE Kubernetes cluster, including the traffic pattern similarity search UI.
  • For the 5-hour window around the Feb 20th incident, built similarity search tables for request count, response size, and time to first byte. The UI is available on port 8012 on deployment.eqiad.wmnet (requires SSH tunnel, ui: http://localhost:8012/traffic-patterns/neighbours).
  • Investigated CAS authentication for the service — this is an even bigger stretch goal than anticipated; the authentication layer with the IdP would need to be handled by the service itself, on top of a security review and config changes from SRE.

Progress

  • Deployment chart review in progress; addressing reviewer comments.
  • Met with @CDanis ; he was able to use the tool and explore the data, and gave valuable feedback. Also aligned on remaining deliverables for this hypothesis and discussed potential next steps in Q4
  • Scoped wrap-up deliverables:
    • change UI to make nearest neighbors results links to perform a new search showing the nn of the clicked result.
    • user guide documentation for the similarity search UI, supported similarity types, and what each is looking for in the data.
    • interactive python notebook for SRE to run the pipeline end-to-end, including spinning up a local vector-search instance.
    • Generate similarity indices for specific configurations for the Feb 20th DDoS incident for further manual evaluation by SRE, deployed on the production instance. (groupings: ip and ja3n, time granularity: 1min and 5min, time window, 1 hour, 3 hours, 1 day)

Next

  • Wrap up open merge requests.
  • Enable timeseries dag on research instance and share with DPE team
  • Prepare documentation, notebooks, and evaluation indices.

Final update:

  • The hypothesis is supported. An SRE investigating suspicious traffic can now surface actionable signals in seconds via the similarity search UI, work that previously required hours of ad hoc analysis.
  • What was accomplished:
    • Hypothesis report doc (review requested from Leila and Chris before we close the hypothesis on Asana).
    • Similarity search UI for SREs to explore similar IP or JA3N fingerprints based on traffic pattern analysis (user guide)
    • Signal processing pipeline over webrequest data, storing similarity search indices on Ceph S3, scheduled hourly via Airflow (technical documentation)
    • Webrequest timeseries dataset published on DataHub, reusable as input for related OKR work (datahub)
    • Similarity indices available for IP and JA3N for last 5 hours (1 min window size) and last 24 hours (5 min window size), configurable via Airflow variable. The vector-search instance is deployed on DSE cluster.
  • Key lessons: There are valuable signals in our traffic data that we can leverage to protect our infrastructure — and there is more value still to be unlocked. By building a configurable signal processing pipeline, we open multiple pathways to contribute to the OKR: the search UI delivered this quarter, the timeseries dataset as a reusable input for other OKR work, and a foundation for detection (not just search — active discovery of bad actors such as IPs in a botnet).