Page MenuHomePhabricator

Load test WDQS v2 with rewritten queries.
Open, Needs TriagePublic

Description

We generated two sample datasets of Blazegraph logged queries, rewritten to WDQS v2 compatible SPARQL.

As a starting point, we have 3_448_194 rewrites from the internal endpoint, and 2_244_361 rewrites for the external endpoint.

We can use these datasets to replay traffic deterministically, and collect request and system metrics on the WDQS v2 pilot endpoint. In particular, we can compare latency (distributions) and throughput between Blazegraph and QLever.

Moreover, we will be able to observe QLever's memory allocation and thread utilization patterns under load and with the system in steady state.

The test plan is roughly as follows:

  1. Rewrite wikidata_main queries in batch, preserving the event produced timestamp.
  2. Produce rewritten queries from HDFS to kafka-test with query and meta.dt fields ({"meta": {"dt": <rewritten_dt>}, "query": <rewritten_query>}).
  3. Replay traffic deterministically via queryhammer to internal WDQS routes (queries won't leave WMF network perimeter). The target will be wikidata_main. We will replay the whole Blazegraph fleet on the 6 wdqs-main replicas currently deployed in DPE.
  4. Collect and report queryhammer stats.
  5. Collect and report Prometheus metrics for WDQS v2 endpoints.

Control

  • We need to test both internal and external endpoint queries
  • We compared performance by submitting queries via the proxy vs. directly against QLever.

Related WIP

Other tasks in flight, with different (but related) angles on perf measurement.

AC

  • we have provided upper/lower throughput bounds to inform rate limit choices

Details

Other Assignee
lerickson
Related Changes in Gerrit:
Related Changes in GitLab:
TitleReferenceAuthorSource BranchDest Branch
stats: move Stats locking into accessor methodsrepos/wikidata-platform/queryhammer!26gmodenarefactor-statsmain
streaming: anchor replay on the earliest buffered eventrepos/wikidata-platform/queryhammer!23gmodenascheduler-anchor-warmupmain
streaming: wait a full reorder buffer instead of droppingrepos/wikidata-platform/queryhammer!22gmodenastreaming-backpressuremain
golang: version bump docker imagerepos/wikidata-platform/queryhammer!17gmodenabump-golangmain
Customize query in GitLab

Event Timeline

WIP docs on how to prepare data and submit rewritten queries to WDQS v2 at https://gitlab.wikimedia.org/gmodena/wdqs-load-testing

tl;dr:

Rewritten queries are avaialble at wikidata.wdqs_external_rewrite_export and wikidata.wdqs_internal_rewrite_export.

To run tests, we need to produce rewrites into dedicated kafka topics (I'm using kafka-test), consumed on wdqs1029 (which can reach both kafka and WDQS v2 via internal routes).

Change #1345098 had a related patch set uploaded (by Gmodena; author: Gmodena):

[operations/deployment-charts@master] wdqs: widen proxy readiness probe timeout/failureThreshold

https://gerrit.wikimedia.org/r/1345098

Change #1345098 merged by jenkins-bot:

[operations/deployment-charts@master] wdqs: widen proxy readiness probe timeout/failureThreshold

https://gerrit.wikimedia.org/r/1345098

Input data (kafka topics and input.json) is generated with this supporting scripts https://gitlab.wikimedia.org/gmodena/wdqs-load-testing

Impact of query load on write performance

Wikidata changes propagate in real time, with median lag under 30 seconds even under heavy read load. Our current SLO is 10 minutes. By saturating QLever with our load-testing tooling, we showed that write performance is unaffected by read load. This addresses the most common WDQS v1 failure scenario, where change-propagation lag correlated with read load.

In this setup the streaming updater is writing to qlever directly. When istio ejected all proxies because of congestion (health probe degradation, 5xx returned to read class queries) the updater lag (write class queries) was not impacted.

Performance on internal traffic (WBQC, mostly)

We replayed deterministically a sample of WBQC query logs and showed that latency and throughput match Blazegraph (as a lower bound). On a (randomly chosen) sub-sample of 101186 queries collected on 2026-09-05, we report a throughput of:

  • Min: 2177.0 req/min
  • p50: 3887.0 req/min
  • p90: 4523.0 req/min
  • p95: 4620.9 req/min
  • p99: 4707.1 req/min
  • Max: 5659.0 req/min

Replicate this test run with:

./queryhammer -streaming     -kafka-topics "wdqs2_internal_traffic_replay_sorted"     -kafka-brokers "kafka-test1010.eqiad.wmnet:9092"     -endpoint "https://wdqs2-main-external.discovery.wmnet:30443/sparql"     -kafka-group "wdqs1029.load_test.internal.29"     -reorder-delay 50s     -reorder-buffer-size 50000     -max-lag 30s     -rate-limit 1000     -parallelism 10 -timeout 6

This value matches the actual throughput recorded in the logs sample. We repeated the same experiment on different subsamples with consistent results.
WBQC can reach 15000 queries/minute at peak across two data centers. To simulate that behavior, we performed a follow-up experiment that pushed that throughput. We forced 6000 - 12000 queries/minute and ensured that traffic was sustained for 2.5 hours on a single data center (eqiad). Service throughput can be observed at https://grafana.wikimedia.org/d/trs2czn/wdqsv2-fancy?from=2026-09-24T08:00:34.946Z&to=2026-09-24T14:36:48.252Z&timezone=utc&var-cluster=P0AF0B00C3C579A2D&var-namespace=wdqs&var-release=main-external&viewPanel=panel-11

Replicate this run with

./queryhammer  -query-files input.jsonl  -endpoint "https://wdqs2-main-external.discovery.wmnet:30443/sparql"     -parallelism 20     -rate-limit 300 --duration  9000 -randomize  -benchmark --failed-log-file wbqc.fail.jsonl

Knows issues that might have impact on perf

I did a new load test with WBCQ queries. After a round of tuning, and increasing cores to eqiad main release.

$  ./queryhammer -endpoint "https://wdqs2-main-external.svc.eqiad.wmnet:30443/sparql" -query-files internal.wbqc.txt -parallelism 100 -benchmark -randomize -rate-limit 1000 --duration 1800 
...
2026/10/08 12:00:34 ============================================================
2026/10/08 12:00:34 Final Statistics
2026/10/08 12:00:34 ============================================================
2026/10/08 12:00:34 Total queries processed: 986216
2026/10/08 12:00:34   Successful:            960495 (97.39%)
2026/10/08 12:00:34   Failed:                25721 (2.61%)
2026/10/08 12:00:34 Total time:              30m0.003s
2026/10/08 12:00:34 Average throughput:      547.90 req/s (533.61 success/s)
2026/10/08 12:00:34                          32873.8 req/min (32016.5 success/min)
2026/10/08 12:00:34 ------------------------------------------------------------
2026/10/08 12:00:34 Throughput Distribution (per-minute, sorted by req/min)
2026/10/08 12:00:34 ------------------------------------------------------------
2026/10/08 12:00:34   Samples: 28
2026/10/08 12:00:34   Min:   18274.3 req/min ( 17838.6 success/min,  97.6%)
2026/10/08 12:00:34   p50:   33159.0 req/min ( 32628.0 success/min,  98.4%)
2026/10/08 12:00:34   p90:   36227.9 req/min ( 35277.9 success/min,  97.4%)
2026/10/08 12:00:34   p95:   37490.8 req/min ( 36603.8 success/min,  97.6%)
2026/10/08 12:00:34   p99:   38288.5 req/min ( 37453.5 success/min,  97.8%)
2026/10/08 12:00:34   Max:   40687.1 req/min ( 38944.4 success/min,  95.7%)
2026/10/08 12:00:34 ------------------------------------------------------------
2026/10/08 12:00:34 Latency Distribution
2026/10/08 12:00:34 ------------------------------------------------------------
2026/10/08 12:00:34   Samples: 100000 (reservoir sampled from 960495 total measurements)
2026/10/08 12:00:34   Min:  3.954048ms
2026/10/08 12:00:34   p50:  20.632241ms
2026/10/08 12:00:34   p95:  167.993893ms
2026/10/08 12:00:34   p99:  448.565684ms
2026/10/08 12:00:34   Max:  20.582857208s
2026/10/08 12:00:34   Mean: 48.304788ms
2026/10/08 12:00:34 ============================================================

And some data on external. Sample run against the 3 eqiad replicas (60 cores each, QLEVER_NUM_QUERIES=120). Set client timeout to 65s to match envoy.

Number of concurrent test workers set to 300 to saturate cores, but not QLever concurrency limit.

$ ./queryhammer -endpoint "https://wdqs2-main-external.svc.eqiad.wmnet:30443/sparql" -query-files external_input.sorted.jsonl -parallelism 300 -benchmark -randomize -rate-limit 200 --duration 1800 -timeout 65 -failed-log-file external_input.sorted.failed.jsonl

2026/10/09 11:00:35 ============================================================
2026/10/09 11:00:35 Final Statistics
2026/10/09 11:00:35 ============================================================
2026/10/09 11:00:35 Total queries processed: 79804
2026/10/09 11:00:35   Successful:            58965 (73.89%)
2026/10/09 11:00:35   Failed:                20839 (26.11%)
2026/10/09 11:00:35 Total time:              30m0.014s
2026/10/09 11:00:35 Average throughput:      44.34 req/s (32.76 success/s)
2026/10/09 11:00:35                          2660.1 req/min (1965.5 success/min)
2026/10/09 11:00:35 ------------------------------------------------------------
2026/10/09 11:00:35 Throughput Distribution (per-minute, sorted by req/min)
2026/10/09 11:00:35 ------------------------------------------------------------
2026/10/09 11:00:35   Samples: 28
2026/10/09 11:00:35   Min:    1630.2 req/min (  1238.8 success/min,  76.0%)
2026/10/09 11:00:35   p50:    2528.3 req/min (  2016.9 success/min,  79.8%)
2026/10/09 11:00:35   p90:    3344.0 req/min (  2353.0 success/min,  70.4%)
2026/10/09 11:00:35   p95:    3393.0 req/min (  2509.0 success/min,  73.9%)
2026/10/09 11:00:35   p99:    3997.0 req/min (  2207.0 success/min,  55.2%)
2026/10/09 11:00:35   Max:    4305.3 req/min (  1612.6 success/min,  37.5%)
2026/10/09 11:00:35 ------------------------------------------------------------
2026/10/09 11:00:35 Latency Distribution
2026/10/09 11:00:35 ------------------------------------------------------------
2026/10/09 11:00:35   Samples: 58965
2026/10/09 11:00:35   Min:  5.059782ms
2026/10/09 11:00:35   p50:  3.577370392s
2026/10/09 11:00:35   p95:  21.9598226s
2026/10/09 11:00:35   p99:  46.825671176s
2026/10/09 11:00:35   Max:  1m4.686128587s
2026/10/09 11:00:35   Mean: 6.469170022s
2026/10/09 11:00:35 ============================================================

This looks ok, but not great. Some stats on failures. The NULL staus code is a client timeout.

gmodena@wdqs1029:~/queryhammer$ nix run nixpkgs#duckdb -- <<'EOF'
SELECT status_code,
       regexp_extract(error_body, '"errorType"\s*:\s*"([^"]*)"', 1) AS error_type,
       count(*) AS n
FROM read_json_auto('external_input.sorted.failed.jsonl')
GROUP BY ALL
ORDER BY n DESC;
EOF


┌─────────────┬───────────────────────┬───────┐
│ status_code │      error_type       │   n   │
│    int64    │        varchar        │ int64 │
├─────────────┼───────────────────────┼───────┤
│         500 │ internal_server_error │ 17050 │
│         503 │                       │  2939 │
│         400 │ parse_error           │  2234 │
│         504 │ backend_timeout       │  2013 │
│        NULL │ NULL                  │  1488 │
│         400 │ federation_error      │   179 │
│         500 │                       │    82 │
│         400 │ bad_request           │    60 │
│         504 │                       │     1 │
└─────────────┴───────────────────────┴───────┘

Lowering parallelism to the number of cores available in eqiad (180) made the error rate, and reported latencies, drop. We need to re-evaluate the QLEVER_NUM_QUERY concurrency parameter.

./queryhammer -endpoint "https://wdqs2-main-external.svc.eqiad.wmnet:30443/sparql" -query-files external_input.sorted.jsonl -parallelism 180 -benchmark -randomize -rate-limit 200 --duration 1800 -timeout 65 -failed-log-file external_input.sorted.failed.jsonl 

2026/10/09 11:55:02 ============================================================
2026/10/09 11:55:02 Final Statistics
2026/10/09 11:55:02 ============================================================
2026/10/09 11:55:02 Total queries processed: 68424
2026/10/09 11:55:02   Successful:            64332 (94.02%)
2026/10/09 11:55:02   Failed:                4092 (5.98%)
2026/10/09 11:55:02 Total time:              30m0.009s
2026/10/09 11:55:02 Average throughput:      38.01 req/s (35.74 success/s)
2026/10/09 11:55:02                          2280.8 req/min (2144.4 success/min)
2026/10/09 11:55:02 ------------------------------------------------------------
2026/10/09 11:55:02 Throughput Distribution (per-minute, sorted by req/min)
2026/10/09 11:55:02 ------------------------------------------------------------
2026/10/09 11:55:02   Samples: 28
2026/10/09 11:55:02   Min:    1412.0 req/min (  1301.0 success/min,  92.1%)
2026/10/09 11:55:02   p50:    2294.0 req/min (  2152.0 success/min,  93.8%)
2026/10/09 11:55:02   p90:    2650.2 req/min (  2516.3 success/min,  94.9%)
2026/10/09 11:55:02   p95:    2666.0 req/min (  2531.0 success/min,  94.9%)
2026/10/09 11:55:02   p99:    2745.0 req/min (  2613.0 success/min,  95.2%)
2026/10/09 11:55:02   Max:    3096.9 req/min (  3001.9 success/min,  96.9%)
2026/10/09 11:55:02 ------------------------------------------------------------
2026/10/09 11:55:02 Latency Distribution
2026/10/09 11:55:02 ------------------------------------------------------------
2026/10/09 11:55:02   Samples: 64332
2026/10/09 11:55:02   Min:  4.442657ms
2026/10/09 11:55:02   p50:  457.74151ms
2026/10/09 11:55:02   p95:  14.321714493s
2026/10/09 11:55:02   p99:  41.613013615s
2026/10/09 11:55:02   Max:  59.917816902s
2026/10/09 11:55:02   Mean: 2.976688765s
2026/10/09 11:55:02 ============================================================

Failures

gmodena@wdqs1029:~/queryhammer$ nix run nixpkgs#duckdb -- <<'EOF'                                                                                                                             SELECT status_code,                                                                                  
       regexp_extract(error_body, '"errorType"\s*:\s*"([^"]*)"', 1) AS error_type,
       count(*) AS n
FROM read_json_auto('external_input.sorted.failed.jsonl')
GROUP BY ALL
ORDER BY n DESC;
EOF
┌─────────────┬───────────────────────┬───────┐
│ status_code │      error_type       │   n   │
│    int64    │        varchar        │ int64 │
├─────────────┼───────────────────────┼───────┤
│         500 │ internal_server_error │ 17387 │
│         504 │ backend_timeout       │  4066 │
│         400 │ parse_error           │  3591 │
│         503 │                       │  2939 │
│        NULL │ NULL                  │  1668 │
│         400 │ federation_error      │   315 │
│         400 │ bad_request           │    87 │
│         500 │                       │    84 │
│         504 │                       │     1 │
└─────────────┴───────────────────────┴───────┘

Qualitatively it would be interesting to see which queries keep failing across experiments, and their original status code. That's beyond the scope of this exercise though, here we only care about measuing overall throughput.

Core utilization and query distribution across pods for both exetrnal test runs is visible in Grafana.

Screenshot From 2026-10-09 14-33-32.png (593×411 px, 64 KB)

Screenshot From 2026-10-09 14-34-32.png (597×396 px, 63 KB)

Screenshot From 2026-10-09 14-35-30.png (613×421 px, 29 KB)

Last experiment for this task. I want to compare perf with/without proxy. This targets our dev node wdqs1029 (40 cores, 256GB ram).

QLever test

I'm running qlever with 30 cpus `QLEVER_NUM_QUERIES=60. Mem/core allocation is bit wonky.

$ docker run --rm     --name wdqs-main-external-qlever     --memory=155g --memory-swap=155g     --cpus=30     -p 7001:7001     -v /srv/wdqs/qlever/index:/srv/data/current     -e SERVICE_IDENTIFIER=wdqs-main-external-qlever     -e QLEVER_PORT=7001     -e QLEVER_INDEX_DIR=/srv/data/current     -e QLEVER_DBNAME=wikidata     -e QLEVER_MAX_MEM=130G     -e QLEVER_MAX_CACHE=32G     -e QLEVER_NUM_QUERIES=60     -e QLEVER_QUERY_TIMEOUT=60s     -e QLEVER_ADDITIONAL_ARGS="--persist-updates --enable-metrics --set-runtime-parameter service-max-redirects=3"     docker-registry.wikimedia.org/repos/wikidata-platform/wdqs/wdqs-qlever:master_2026-08-20T20-33-08Z_1075455
...
QLever server 1075455, compiled on Thu Aug 20 20:45:40 UTC 2026 using git hash 1075455
...
$ gmodena@wdqs1029:~/queryhammer$ ./queryhammer -endpoint "http://localhost:7001/sparql" -query-files external_input.sorted.jsonl -parallelism 30 -benchmark -randomize -rate-limit 100 --duration 1800 -failed-log-file  external_input.sorted.failed.jsonl --inject-prefixes -timeout 65
...
2026/10/09 14:36:47 ============================================================
2026/10/09 14:36:47 Final Statistics
2026/10/09 14:36:47 ============================================================
2026/10/09 14:36:47 Total queries processed: 52694
2026/10/09 14:36:47   Successful:            9279 (17.61%)
2026/10/09 14:36:47   Failed:                43415 (82.39%)
2026/10/09 14:36:47 Total time:              29m59.994s
2026/10/09 14:36:47 Average throughput:      29.27 req/s (5.16 success/s)
2026/10/09 14:36:47                          1756.5 req/min (309.3 success/min)
2026/10/09 14:36:47 ------------------------------------------------------------
2026/10/09 14:36:47 Throughput Distribution (per-minute, sorted by req/min)
2026/10/09 14:36:47 ------------------------------------------------------------
2026/10/09 14:36:47   Samples: 28
2026/10/09 14:36:47   Min:     318.5 req/min (   305.5 success/min,  95.9%)
2026/10/09 14:36:47   p50:     454.0 req/min (   431.0 success/min,  94.9%)
2026/10/09 14:36:47   p90:    6000.0 req/min (     0.0 success/min,   0.0%)
2026/10/09 14:36:47   p95:    6000.0 req/min (     0.0 success/min,   0.0%)
2026/10/09 14:36:47   p99:    6000.0 req/min (     0.0 success/min,   0.0%)
2026/10/09 14:36:47   Max:    6000.1 req/min (     0.0 success/min,   0.0%)
2026/10/09 14:36:47 ------------------------------------------------------------
2026/10/09 14:36:47 Latency Distribution
2026/10/09 14:36:47 ------------------------------------------------------------
2026/10/09 14:36:47   Samples: 9279
2026/10/09 14:36:47   Min:  4.230364ms
2026/10/09 14:36:47   p50:  193.703429ms
2026/10/09 14:36:47   p95:  14.688025784s
2026/10/09 14:36:47   p99:  38.444144151s
2026/10/09 14:36:47   Max:  59.984013961s
2026/10/09 14:36:47   Mean: 2.34172471s
2026/10/09 14:36:47 ============================================================

Load test.

Error rate, mostly triggered by client timeouts, started spiking at the 22 minutes mark.

2026/10/09 14:29:22 Progress: 9751 processed (9204 success, 547 failed) | 7.20 req/s (6.79 success/s) | 431.8 req/min (407.6 success/min) | elapsed: 22m35s
2026/10/09 14:29:27 Progress: 9789 processed (9239 success, 550 failed) | 7.20 req/s (6.79 success/s) | 431.9 req/min (407.6 success/min) | elapsed: 22m40s
2026/10/09 14:29:32 Progress: 9829 processed (9277 success, 552 failed) | 7.20 req/s (6.80 success/s) | 432.0 req/min (407.8 success/min) | elapsed: 22m45s
2026/10/09 14:29:37 Progress: 9832 processed (9279 success, 553 failed) | 7.18 req/s (6.77 success/s) | 430.6 req/min (406.4 success/min) | elapsed: 22m50s
2026/10/09 14:29:42 Progress: 10194 processed (9279 success, 915 failed) | 7.41 req/s (6.75 success/s) | 444.8 req/min (404.9 success/min) | elapsed: 22m55s
2026/10/09 14:29:47 Progress: 10694 processed (9279 success, 1415 failed) | 7.75 req/s (6.72 success/s) | 465.0 req/min (403.4 success/min) | elapsed: 23m0s
2026/10/09 14:29:52 Progress: 11194 processed (9279 success, 1915 failed) | 8.08 req/s (6.70 success/s) | 484.9 req/min (402.0 success/min) | elapsed: 23m5s
2026/10/09 14:29:57 Progress: 11694 processed (9279 success, 2415 failed) | 8.41 req/s (6.68 success/s) | 504.8 req/min (400.5 success/min) | elapsed: 23m10s

Update: upon inspection, qlever process died at 2026/10/09 14:29:37 P

2026-10-09 14:29:32.599 - INFO: 
2026-10-09 14:29:32.600 - INFO: Request received via POST, content type "application/sparql-query"
/usr/bin/wdqs-qlever.sh: line 18:     7 Killed                  /usr/bin/qlever-server --index-basename ${QLEVER_DBNAME} --port ${QLEVER_PORT} --memory-max-size ${QLEVER_MAX_MEM} --cache-max-size ${QLEVER_MAX_CACHE} --cache-max-size-single-entry ${QLEVER_MAX_CACHE_SINGLE_ENTRY} --default-query-timeout ${QLEVER_QUERY_TIMEOUT} -n -j ${QLEVER_NUM_QUERIES} ${QLEVER_ADDITIONAL_ARGS} "$@"
wdqs-proxy test

Matching proxy config (without prefix injection, I'll let queryhammer handle that):

docker run --rm \
    --name wdqs-main-external-proxy \
    --network wdqs \
    --memory=2g --memory-swap=2g \
    --cpus=2 \
    -p 8080:8080 -p 9100:9100 \
    -e SERVICE_IDENTIFIER=wdqs-main-external-proxy \
    -e QUARKUS_REST_CLIENT_DOWNSTREAM_SPARQL_ENDPOINT_URL=http://qlever:7001/sparql \
    -e QUARKUS_REST_CLIENT_DOWNSTREAM_SPARQL_ENDPOINT_CONNECTION_POOL_SIZE=60 \
    -e QUARKUS_REST_CLIENT_DOWNSTREAM_SPARQL_ENDPOINT_READ_TIMEOUT=60000 \
    -e QUARKUS_REST_CLIENT_EVENTGATE_ENDPOINT_URL=http://localhost:6004 \
    -e QUARKUS_MANAGEMENT_PORT=9100 \
    -e 'QUARKUS_LOG_CATEGORY__ORG.WIKIMEDIA.WDQS__LEVEL=DEBUG' \
    -e EVENTGATE_STREAM=wdqs_external.sparql_query.v2 \
    -e EVENTGATE_GRAPH_NAME=wikidata_main \
    -e QUERY_REWRITER_MW_SPARQL_ENABLED=false \
    -e QUERY_REWRITER_FEDERATION_ENABLED=false \
    -e WDQS_PROXY_METRICS_PREFIX=wdqs.proxy. \
    docker-registry.wikimedia.org/repos/wikidata-platform/wdqs/wdqs-proxy:v0.12.0

Load test

$ ./queryhammer -endpoint "http://localhost:8080/sparql" -query-files external_input.sorted.jsonl -parallelism 30 -benchmark -randomize -rate-limit 100 --duration 120 -failed-log-file  external_input.sorted.failed.jsonl -inject-prefixes -timeout 65
...
2026/10/09 15:20:57 ============================================================
2026/10/09 15:20:57 Final Statistics
2026/10/09 15:20:57 ============================================================
2026/10/09 15:20:57 Total queries processed: 13617
2026/10/09 15:20:57   Successful:            12669 (93.04%)
2026/10/09 15:20:57   Failed:                948 (6.96%)
2026/10/09 15:20:57 Total time:              30m0.003s
2026/10/09 15:20:57 Average throughput:      7.56 req/s (7.04 success/s)
2026/10/09 15:20:57                          453.9 req/min (422.3 success/min)
2026/10/09 15:20:57 ------------------------------------------------------------
2026/10/09 15:20:57 Throughput Distribution (per-minute, sorted by req/min)
2026/10/09 15:20:57 ------------------------------------------------------------
2026/10/09 15:20:57   Samples: 28
2026/10/09 15:20:57   Min:     316.6 req/min (   294.5 success/min,  93.0%)
2026/10/09 15:20:57   p50:     451.4 req/min (   421.9 success/min,  93.5%)
2026/10/09 15:20:57   p90:     538.2 req/min (   495.7 success/min,  92.1%)
2026/10/09 15:20:57   p95:     559.4 req/min (   525.2 success/min,  93.9%)
2026/10/09 15:20:57   p99:     570.0 req/min (   536.0 success/min,  94.0%)
2026/10/09 15:20:57   Max:     605.5 req/min (   578.8 success/min,  95.6%)
2026/10/09 15:20:57 ------------------------------------------------------------
2026/10/09 15:20:57 Latency Distribution
2026/10/09 15:20:57 ------------------------------------------------------------
2026/10/09 15:20:57   Samples: 12669
2026/10/09 15:20:57   Min:  6.989843ms
2026/10/09 15:20:57   p50:  197.746715ms
2026/10/09 15:20:57   p95:  13.372154031s
2026/10/09 15:20:57   p99:  35.901067889s
2026/10/09 15:20:57   Max:  59.890693625s
2026/10/09 15:20:57   Mean: 2.267233026s
2026/10/09 15:20:57 ============================================================

I summarized this phab task in https://wikitech.wikimedia.org/wiki/User:GModena_(WMF)/Drafts/2026-10_WDQSv2_Load_Testing#Capacity_recommendation_for_GA. The doc contain recommendations for initial rate limits at GA:

// [...] we propose the following starting position for the public endpoint, served via the REST gateway:

  • Global: 40 qps / 180 concurrent in-flight queries (single DC capacity).
  • Per actor: 4 qps, with a hard ceiling of 18 concurrent in-flight queries.

"In-flight per actor" is how many of that actor's queries are executing at the same time. It follows from the actor's query cost (not just from its request count). [...] //

I created https://phabricator.wikimedia.org/T441651 to track integration with REST Gateway.

Change #1358194 had a related patch set uploaded (by Gmodena; author: Gmodena):

[operations/deployment-charts@master] WIP: rest-gateway: set WDQSv2 GA rate limits.

https://gerrit.wikimedia.org/r/1358194