In support of 4.3.11 APP FY24-25, Research is requesting a review of the suggested scraping detection approach and consulting on production pipeline implementation.
The proposed scraping analytics will allow an hourly detection of scraping efforts and traffic anomalies.
Expected output:
Separate metrics table that can be queried and dashboarded
Use cases
- Ban scraper IPs
- Identify and institutional scrapers
- Online rate limiting (might require streaming based approach)
Implementation approach:
https://gitlab.wikimedia.org/repos/research/research-datasets/-/tree/traffic_patterns/src/research_datasets/traffic_patterns
Example application:
https://gitlab.wikimedia.org/repos/research/research-datasets/-/blob/traffic_patterns/notebooks/traffic_patterns/metrics.ipynb
DE to help with
- Review analytics approach and estimate pipeline implementation complexity
- Estimate effort to make this production ready
- Estimate runtime complexity / acceptable latency for both 1) hourly run and 2) streaming