If we apply advanced signal processing and clustering techniques to media requests data, we will be able to recommend a set of new signals that can be used by SRE to detect media scrapers automatically.

Context: Over the past years, we have seen a substantial increase in automated traffic to Wikipedia. Specifically, media scrapers are the most costly for our infrastructure, see also a recent scraping incident. Unfortunately, identifying those scrapers is an open problem. While we have some individual heuristics, they are very noisy and, as a result, there is a high risk of unintentionally also blocking actual user traffic. Therefore, the goal of this project is to identify potential patterns that can more reliably detect media scrapers. We will apply embedding techniques to webrequest data from media scrapers to capture the similarity between individual requests or actors. One starting point is the tooling developed in T415637 and apply to this specific use case. Alternative approaches include integrating the individual signals used in previous heuristics into a common feature space. Applying clustering techniques, we can identify characteristic patterns by manually labeling a small number of known scrapers. The deliverable of this work is a recommendation of a set of new signals that can be used by SRE to detect media scrapers automatically.