Page MenuHomePhabricator
Create Task
Maniphest T416475

Unify and improve load testing strategy for inference services
Open, Needs TriagePublic
Actions

Description

Our current Locust load tests were configured independently and differently during development of each individual service. This makes it hard to compare the results between services and it might make the number unreliable in some cases. We need to standardize the configuration and fix several issues to make load tests useful for capacity planning.

Current State

  • Global config users 2 concurrent test users
  • Wait times between subsequent requests for each user very between each service anywhere from 0s to 5s
  • Inconsistent patterns across test files (host config, wait times, task weights)

Initial proposals:

  • Standardize wait times to between(0, 0.1) for throughput testing
  • Use a consistent pattern of 2 concurrent users for single worker deployed in a service. Services using multiple workers should scale concurrent users accordingly.
  • Add response validation to all tests (currently only edit_check validates)

Related Objects

Event Timeline

BWojtowicz-WMF created this task.Feb 4 2026, 1:38 PM
Restricted Application added a subscriber: Aklapper. · View Herald TranscriptFeb 4 2026, 1:38 PM
BWojtowicz-WMF added a comment.Mar 24 2026, 9:36 AM

When investigating T420931, I found that my custom async load test script achieves >300 RPS against the same service with 5 replicas, whereas the locust test against 1 replica reports only ~0.67 RPS. The discrepancy comes down to the Locust configuration:

The article topic locust test uses wait_time = between(1, 5) so each simulated user waits 1-5 seconds between requests. Combined with our config of users = 2, the theoretical max throughput is ~0.67RPS:

avg_wait = (1 + 5) / 2 = 3.0s
max_rps_per_user = 1 / avg_wait = ~0.33 RPS
max_rps_locust = 2 users * 0.33 = ~0.67 RPS

With the custom async script, I've noticed the optimal saturation of the service to be around 4 async workers per 1 replica.
This means the current locust configuration and results might be very misleading in some cases.

This week, I'll continue to ensure in that in all tests, we:

  • Reduce wait_time to between(0, 0.1) for throughput testing in all configurations
  • Increase number of maximum concurrent users to increase service saturation.
isarantopoulos edited projects, added: Machine-Learning-Team (Q1 FY2026-27); removed: Machine-Learning-Team.Apr 7 2026, 1:16 PM
Content licensed under Creative Commons Attribution-ShareAlike (CC BY-SA) 4.0 unless otherwise noted; code licensed under GNU General Public License (GPL) 2.0 or later and other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL · Credits