Page MenuHomePhabricator

Data analysis request for media rate limiting
Open, In Progress, MediumPublic

Description

We are exploring cost-based rate limits on media files, similar to the current shadow mode cost-based policies in the rest-gateway. In order to fine-tune policies, we would need the following data analysis.

The limits are based on response_size in kbytes and keyed to x-client-ip on a fixed-window of 1 minute.

The current limit is 600000 kbytes/minute, tallied from 0 at the beginning of the window and reset at the end.

The request that would go over-quota is let through, and the following requests during the window would be blocked.

Based on webrequest data for cache status miss and pass on upload.wikimedia.org, for half the limit 300000 kbytes/min, the limit 600000 kbytes/min and twice the limit 1200000 kbytes/min :

Client profile

  • How many distinct clients would be blocked + % of total clients?
  • X-Is-Browser split
  • User-Agent cardinality (are there User-Agent that jump out as getting potentially blocked a lot more than others)
  • res_proxy

Request profile

  • Percentage of total requests blocked
  • X-Is-Browser split
  • res_proxy

Thanks a lot for the help!

Details

Other Assignee
Clement_Goubert

Event Timeline

KCVelaga_WMF changed the task status from Open to In Progress.Aug 12 2026, 2:33 PM
KCVelaga_WMF moved this task from Next 2 weeks to Doing on the Product-Analytics (Kanban) board.

Shared the outputs from initial analysis for questions to @Clement_Goubert for review.

Spreadsheet (internal): https://docs.google.com/spreadsheets/d/1bylF4s2FOo-W5F-Dhfb-RuiOk-dI-sAN5gvJtjEmBLo/edit?usp=sharing

It has the following outputs:

  • Overall clients and requests that'd be blocked
  • Blocks by residential proxy presence, and counts by individual proxy networks.
  • Blocks by spider vs. user categorization.
  • Blocks by file (image, image_thumbnail, image_3d, video, audio and documents)
  • Blocks by browser likelines
    • bucketed scores: negative < 0, low < 30, medium < 100, and high >= 100
    • individual x-is-browser score
  • Clients that would have been most hit by the limits (top 20)

For each breakdown, we have the three proposed limits: 300K kbytes/minute, 600K kbytes/minute, and 1200K kbytes/minute with how many clients would have been blocked and how many requests would have been blocked.

Data period: 13 July to 9 August 2026 (4 weeks) of webrequest logs to upload.wikimedia.org and thumb.wikimedia.org

Let me know if there are any following up questions or there are other dimensions of interest.

Additional parameters for a new run:

  • Correlate with KAPOW scores to see if interesting information pops up
  • Correlate with known_client to see if we need to implement known_client ratelimit bypass
  • Estimate bandwidth savings avg(response_size)*blocked_requests, possibly estimate over X-Is-Browser and top clients