Page MenuHomePhabricator

Enable vLLM Prometheus metrics for LiftWing LLM inference services
Closed, ResolvedPublic

Description

In T429597, SREs are working on creating grafana dashboards that show LLM inference services metrics.

Our LLM inference services run vLLM via the embedded vllm.LLM engine inside a KServe predictor (not vllm serve). On that offline path, engine stats are off by default and they are not emitted as shown in: P94723

When we set disable_log_stats=False, vLLM metrics are emitted on the same /metrics endpoint that KServe already exposes as shown in: P94724

Below are the vLLM model-servers where we are going to enable metrics:

  • cope_b_a4b
  • cope_a_9b
  • gpt_oss_safeguard_20b
  • embeddings
  • llm-qwen3-14b

NB: vLLM labels its metrics with served_model_name, which defaults to the HF model path e.g model_name="zentropi-ai/cope-b-a4b", not cope-b-a4b. Our own KServe metrics and the legends shown in: https://grafana-rw.wikimedia.org/goto/ffqvoolrpfrwgf?orgId=default use the short isvc name. We'll align this on our side by passing served_model_name=<kserve model name> so vLLM emits model_name="cope-b-a4b", matching the rest as shown in: P94725

Event Timeline

Change #1307668 had a related patch set uploaded (by Kevin Bazira; author: Kevin Bazira):

[machinelearning/liftwing/inference-services@main] cope-b-a4b: Enable vLLM metrics

https://gerrit.wikimedia.org/r/1307668

Change #1307668 merged by jenkins-bot:

[machinelearning/liftwing/inference-services@main] cope-b-a4b: Enable vLLM metrics

https://gerrit.wikimedia.org/r/1307668

Change #1307708 had a related patch set uploaded (by Kevin Bazira; author: Kevin Bazira):

[operations/deployment-charts@master] ml-services: deploy cope-b-a4b isvc image that emits vLLM metrics

https://gerrit.wikimedia.org/r/1307708

Change #1307708 merged by jenkins-bot:

[operations/deployment-charts@master] ml-services: deploy cope-b-a4b isvc image that emits vLLM metrics

https://gerrit.wikimedia.org/r/1307708

The cope-b-a4b isvc now emits vLLM metrics as shown below:

1$ curl "https://inference.svc.eqiad.wmnet:30443/metrics" -H "Host: cope-b-a4b.llm.wikimedia.org"
2# HELP python_gc_objects_collected_total Objects collected during gc
3# TYPE python_gc_objects_collected_total counter
4python_gc_objects_collected_total{generation="0"} 194669.0
5python_gc_objects_collected_total{generation="1"} 23244.0
6python_gc_objects_collected_total{generation="2"} 240.0
7# HELP python_gc_objects_uncollectable_total Uncollectable objects found during GC
8# TYPE python_gc_objects_uncollectable_total counter
9python_gc_objects_uncollectable_total{generation="0"} 0.0
10python_gc_objects_uncollectable_total{generation="1"} 0.0
11python_gc_objects_uncollectable_total{generation="2"} 0.0
12# HELP python_gc_collections_total Number of times this generation was collected
13# TYPE python_gc_collections_total counter
14python_gc_collections_total{generation="0"} 2757.0
15python_gc_collections_total{generation="1"} 250.0
16python_gc_collections_total{generation="2"} 10.0
17# HELP python_info Python platform information
18# TYPE python_info gauge
19python_info{implementation="CPython",major="3",minor="12",patchlevel="13",version="3.12.13"} 1.0
20# HELP process_virtual_memory_bytes Virtual memory size in bytes.
21# TYPE process_virtual_memory_bytes gauge
22process_virtual_memory_bytes 4.6550847488e+010
23# HELP process_resident_memory_bytes Resident memory size in bytes.
24# TYPE process_resident_memory_bytes gauge
25process_resident_memory_bytes 2.295427072e+09
26# HELP process_start_time_seconds Start time of the process since unix epoch in seconds.
27# TYPE process_start_time_seconds gauge
28process_start_time_seconds 1.78333879195e+09
29# HELP process_cpu_seconds_total Total user and system CPU time spent in seconds.
30# TYPE process_cpu_seconds_total counter
31process_cpu_seconds_total 55.12
32# HELP process_open_fds Number of open file descriptors.
33# TYPE process_open_fds gauge
34process_open_fds 53.0
35# HELP process_max_fds Maximum number of open file descriptors.
36# TYPE process_max_fds gauge
37process_max_fds 1.073741816e+09
38# HELP request_preprocess_seconds pre-process request latency
39# TYPE request_preprocess_seconds histogram
40request_preprocess_seconds_bucket{le="0.005",model_name="cope-b-a4b"} 4088.0
41request_preprocess_seconds_bucket{le="0.01",model_name="cope-b-a4b"} 4088.0
42request_preprocess_seconds_bucket{le="0.025",model_name="cope-b-a4b"} 4088.0
43request_preprocess_seconds_bucket{le="0.05",model_name="cope-b-a4b"} 4088.0
44request_preprocess_seconds_bucket{le="0.075",model_name="cope-b-a4b"} 4088.0
45request_preprocess_seconds_bucket{le="0.1",model_name="cope-b-a4b"} 4088.0
46request_preprocess_seconds_bucket{le="0.25",model_name="cope-b-a4b"} 4088.0
47request_preprocess_seconds_bucket{le="0.5",model_name="cope-b-a4b"} 4088.0
48request_preprocess_seconds_bucket{le="0.75",model_name="cope-b-a4b"} 4088.0
49request_preprocess_seconds_bucket{le="1.0",model_name="cope-b-a4b"} 4088.0
50request_preprocess_seconds_bucket{le="2.5",model_name="cope-b-a4b"} 4088.0
51request_preprocess_seconds_bucket{le="5.0",model_name="cope-b-a4b"} 4088.0
52request_preprocess_seconds_bucket{le="7.5",model_name="cope-b-a4b"} 4088.0
53request_preprocess_seconds_bucket{le="10.0",model_name="cope-b-a4b"} 4088.0
54request_preprocess_seconds_bucket{le="+Inf",model_name="cope-b-a4b"} 4088.0
55request_preprocess_seconds_count{model_name="cope-b-a4b"} 4088.0
56request_preprocess_seconds_sum{model_name="cope-b-a4b"} 0.10078066121786833
57# HELP request_preprocess_seconds_created pre-process request latency
58# TYPE request_preprocess_seconds_created gauge
59request_preprocess_seconds_created{model_name="cope-b-a4b"} 1.7833389344787433e+09
60# HELP request_postprocess_seconds post-process request latency
61# TYPE request_postprocess_seconds histogram
62request_postprocess_seconds_bucket{le="0.005",model_name="cope-b-a4b"} 4088.0
63request_postprocess_seconds_bucket{le="0.01",model_name="cope-b-a4b"} 4088.0
64request_postprocess_seconds_bucket{le="0.025",model_name="cope-b-a4b"} 4088.0
65request_postprocess_seconds_bucket{le="0.05",model_name="cope-b-a4b"} 4088.0
66request_postprocess_seconds_bucket{le="0.075",model_name="cope-b-a4b"} 4088.0
67request_postprocess_seconds_bucket{le="0.1",model_name="cope-b-a4b"} 4088.0
68request_postprocess_seconds_bucket{le="0.25",model_name="cope-b-a4b"} 4088.0
69request_postprocess_seconds_bucket{le="0.5",model_name="cope-b-a4b"} 4088.0
70request_postprocess_seconds_bucket{le="0.75",model_name="cope-b-a4b"} 4088.0
71request_postprocess_seconds_bucket{le="1.0",model_name="cope-b-a4b"} 4088.0
72request_postprocess_seconds_bucket{le="2.5",model_name="cope-b-a4b"} 4088.0
73request_postprocess_seconds_bucket{le="5.0",model_name="cope-b-a4b"} 4088.0
74request_postprocess_seconds_bucket{le="7.5",model_name="cope-b-a4b"} 4088.0
75request_postprocess_seconds_bucket{le="10.0",model_name="cope-b-a4b"} 4088.0
76request_postprocess_seconds_bucket{le="+Inf",model_name="cope-b-a4b"} 4088.0
77request_postprocess_seconds_count{model_name="cope-b-a4b"} 4088.0
78request_postprocess_seconds_sum{model_name="cope-b-a4b"} 0.045810746029019356
79# HELP request_postprocess_seconds_created post-process request latency
80# TYPE request_postprocess_seconds_created gauge
81request_postprocess_seconds_created{model_name="cope-b-a4b"} 1.7833389367737012e+09
82# HELP request_predict_seconds predict request latency
83# TYPE request_predict_seconds histogram
84request_predict_seconds_bucket{le="0.005",model_name="cope-b-a4b"} 0.0
85request_predict_seconds_bucket{le="0.01",model_name="cope-b-a4b"} 0.0
86request_predict_seconds_bucket{le="0.025",model_name="cope-b-a4b"} 3.0
87request_predict_seconds_bucket{le="0.05",model_name="cope-b-a4b"} 2259.0
88request_predict_seconds_bucket{le="0.075",model_name="cope-b-a4b"} 3409.0
89request_predict_seconds_bucket{le="0.1",model_name="cope-b-a4b"} 3739.0
90request_predict_seconds_bucket{le="0.25",model_name="cope-b-a4b"} 4086.0
91request_predict_seconds_bucket{le="0.5",model_name="cope-b-a4b"} 4086.0
92request_predict_seconds_bucket{le="0.75",model_name="cope-b-a4b"} 4087.0
93request_predict_seconds_bucket{le="1.0",model_name="cope-b-a4b"} 4087.0
94request_predict_seconds_bucket{le="2.5",model_name="cope-b-a4b"} 4088.0
95request_predict_seconds_bucket{le="5.0",model_name="cope-b-a4b"} 4088.0
96request_predict_seconds_bucket{le="7.5",model_name="cope-b-a4b"} 4088.0
97request_predict_seconds_bucket{le="10.0",model_name="cope-b-a4b"} 4088.0
98request_predict_seconds_bucket{le="+Inf",model_name="cope-b-a4b"} 4088.0
99request_predict_seconds_count{model_name="cope-b-a4b"} 4088.0
100request_predict_seconds_sum{model_name="cope-b-a4b"} 226.2469999799505
101# HELP request_predict_seconds_created predict request latency
102# TYPE request_predict_seconds_created gauge
103request_predict_seconds_created{model_name="cope-b-a4b"} 1.7833389344788766e+09
104# HELP request_explain_seconds explain request latency
105# TYPE request_explain_seconds histogram
106# HELP vllm:estimated_flops_per_gpu_total Estimated number of floating point operations per GPU (for Model Flops Utilization calculations).
107# TYPE vllm:estimated_flops_per_gpu_total counter
108vllm:estimated_flops_per_gpu_total{engine="0",model_name="cope-b-a4b"} 0.0
109# HELP vllm:estimated_flops_per_gpu_created Estimated number of floating point operations per GPU (for Model Flops Utilization calculations).
110# TYPE vllm:estimated_flops_per_gpu_created gauge
111vllm:estimated_flops_per_gpu_created{engine="0",model_name="cope-b-a4b"} 1.783338930332477e+09
112# HELP vllm:estimated_read_bytes_per_gpu_total Estimated number of bytes read from memory per GPU (for Model Flops Utilization calculations).
113# TYPE vllm:estimated_read_bytes_per_gpu_total counter
114vllm:estimated_read_bytes_per_gpu_total{engine="0",model_name="cope-b-a4b"} 0.0
115# HELP vllm:estimated_read_bytes_per_gpu_created Estimated number of bytes read from memory per GPU (for Model Flops Utilization calculations).
116# TYPE vllm:estimated_read_bytes_per_gpu_created gauge
117vllm:estimated_read_bytes_per_gpu_created{engine="0",model_name="cope-b-a4b"} 1.7833389303325064e+09
118# HELP vllm:estimated_write_bytes_per_gpu_total Estimated number of bytes written to memory per GPU (for Model Flops Utilization calculations).
119# TYPE vllm:estimated_write_bytes_per_gpu_total counter
120vllm:estimated_write_bytes_per_gpu_total{engine="0",model_name="cope-b-a4b"} 0.0
121# HELP vllm:estimated_write_bytes_per_gpu_created Estimated number of bytes written to memory per GPU (for Model Flops Utilization calculations).
122# TYPE vllm:estimated_write_bytes_per_gpu_created gauge
123vllm:estimated_write_bytes_per_gpu_created{engine="0",model_name="cope-b-a4b"} 1.783338930332521e+09
124# HELP vllm:num_requests_running Number of requests in model execution batches.
125# TYPE vllm:num_requests_running gauge
126vllm:num_requests_running{engine="0",model_name="cope-b-a4b"} 0.0
127# HELP vllm:num_requests_waiting Number of requests waiting to be processed.
128# TYPE vllm:num_requests_waiting gauge
129vllm:num_requests_waiting{engine="0",model_name="cope-b-a4b"} 0.0
130# HELP vllm:num_requests_waiting_by_reason Number of waiting requests by reason. Reason labels: 'capacity' = waiting for scheduling capacity; 'deferred' = deferred by transient constraints (LoRA budget, KV transfer, blocked status). Sum of all reasons equals vllm:num_requests_waiting.
131# TYPE vllm:num_requests_waiting_by_reason gauge
132vllm:num_requests_waiting_by_reason{engine="0",model_name="cope-b-a4b",reason="capacity"} 0.0
133vllm:num_requests_waiting_by_reason{engine="0",model_name="cope-b-a4b",reason="deferred"} 0.0
134# HELP vllm:engine_sleep_state Engine sleep state; awake = 0 means engine is sleeping; awake = 1 means engine is awake; weights_offloaded = 1 means sleep level 1; discard_all = 1 means sleep level 2.
135# TYPE vllm:engine_sleep_state gauge
136vllm:engine_sleep_state{engine="0",model_name="cope-b-a4b",sleep_state="awake"} 1.0
137vllm:engine_sleep_state{engine="0",model_name="cope-b-a4b",sleep_state="weights_offloaded"} 0.0
138vllm:engine_sleep_state{engine="0",model_name="cope-b-a4b",sleep_state="discard_all"} 0.0
139# HELP vllm:kv_cache_usage_perc KV-cache usage. 1 means 100 percent usage.
140# TYPE vllm:kv_cache_usage_perc gauge
141vllm:kv_cache_usage_perc{engine="0",model_name="cope-b-a4b"} 0.0
142# HELP vllm:prefix_cache_queries_total Prefix cache queries, in terms of number of queried tokens.
143# TYPE vllm:prefix_cache_queries_total counter
144vllm:prefix_cache_queries_total{engine="0",model_name="cope-b-a4b"} 1.017793e+07
145# HELP vllm:prefix_cache_queries_created Prefix cache queries, in terms of number of queried tokens.
146# TYPE vllm:prefix_cache_queries_created gauge
147vllm:prefix_cache_queries_created{engine="0",model_name="cope-b-a4b"} 1.7833389303327112e+09
148# HELP vllm:prefix_cache_hits_total Prefix cache hits, in terms of number of cached tokens.
149# TYPE vllm:prefix_cache_hits_total counter
150vllm:prefix_cache_hits_total{engine="0",model_name="cope-b-a4b"} 6.577568e+06
151# HELP vllm:prefix_cache_hits_created Prefix cache hits, in terms of number of cached tokens.
152# TYPE vllm:prefix_cache_hits_created gauge
153vllm:prefix_cache_hits_created{engine="0",model_name="cope-b-a4b"} 1.7833389303327224e+09
154# HELP vllm:external_prefix_cache_queries_total External prefix cache queries from KV connector cross-instance cache sharing, in terms of number of queried tokens.
155# TYPE vllm:external_prefix_cache_queries_total counter
156vllm:external_prefix_cache_queries_total{engine="0",model_name="cope-b-a4b"} 0.0
157# HELP vllm:external_prefix_cache_queries_created External prefix cache queries from KV connector cross-instance cache sharing, in terms of number of queried tokens.
158# TYPE vllm:external_prefix_cache_queries_created gauge
159vllm:external_prefix_cache_queries_created{engine="0",model_name="cope-b-a4b"} 1.7833389303327353e+09
160# HELP vllm:external_prefix_cache_hits_total External prefix cache hits from KV connector cross-instance cache sharing, in terms of number of cached tokens.
161# TYPE vllm:external_prefix_cache_hits_total counter
162vllm:external_prefix_cache_hits_total{engine="0",model_name="cope-b-a4b"} 0.0
163# HELP vllm:external_prefix_cache_hits_created External prefix cache hits from KV connector cross-instance cache sharing, in terms of number of cached tokens.
164# TYPE vllm:external_prefix_cache_hits_created gauge
165vllm:external_prefix_cache_hits_created{engine="0",model_name="cope-b-a4b"} 1.783338930332746e+09
166# HELP vllm:mm_cache_queries_total Multi-modal cache queries, in terms of number of queried items.
167# TYPE vllm:mm_cache_queries_total counter
168vllm:mm_cache_queries_total{engine="0",model_name="cope-b-a4b"} 0.0
169# HELP vllm:mm_cache_queries_created Multi-modal cache queries, in terms of number of queried items.
170# TYPE vllm:mm_cache_queries_created gauge
171vllm:mm_cache_queries_created{engine="0",model_name="cope-b-a4b"} 1.783338930332757e+09
172# HELP vllm:mm_cache_hits_total Multi-modal cache hits, in terms of number of cached items.
173# TYPE vllm:mm_cache_hits_total counter
174vllm:mm_cache_hits_total{engine="0",model_name="cope-b-a4b"} 0.0
175# HELP vllm:mm_cache_hits_created Multi-modal cache hits, in terms of number of cached items.
176# TYPE vllm:mm_cache_hits_created gauge
177vllm:mm_cache_hits_created{engine="0",model_name="cope-b-a4b"} 1.7833389303327696e+09
178# HELP vllm:num_preemptions_total Cumulative number of preemption from the engine.
179# TYPE vllm:num_preemptions_total counter
180vllm:num_preemptions_total{engine="0",model_name="cope-b-a4b"} 0.0
181# HELP vllm:num_preemptions_created Cumulative number of preemption from the engine.
182# TYPE vllm:num_preemptions_created gauge
183vllm:num_preemptions_created{engine="0",model_name="cope-b-a4b"} 1.7833389303327801e+09
184# HELP vllm:prompt_tokens_total Number of prefill tokens processed.
185# TYPE vllm:prompt_tokens_total counter
186vllm:prompt_tokens_total{engine="0",model_name="cope-b-a4b"} 1.017793e+07
187# HELP vllm:prompt_tokens_created Number of prefill tokens processed.
188# TYPE vllm:prompt_tokens_created gauge
189vllm:prompt_tokens_created{engine="0",model_name="cope-b-a4b"} 1.783338930332789e+09
190# HELP vllm:prompt_tokens_by_source_total Number of prompt tokens by source.
191# TYPE vllm:prompt_tokens_by_source_total counter
192vllm:prompt_tokens_by_source_total{engine="0",model_name="cope-b-a4b",source="local_compute"} 3.600362e+06
193vllm:prompt_tokens_by_source_total{engine="0",model_name="cope-b-a4b",source="local_cache_hit"} 6.577568e+06
194vllm:prompt_tokens_by_source_total{engine="0",model_name="cope-b-a4b",source="external_kv_transfer"} 0.0
195# HELP vllm:prompt_tokens_by_source_created Number of prompt tokens by source.
196# TYPE vllm:prompt_tokens_by_source_created gauge
197vllm:prompt_tokens_by_source_created{engine="0",model_name="cope-b-a4b",source="local_compute"} 1.7833389303328023e+09
198vllm:prompt_tokens_by_source_created{engine="0",model_name="cope-b-a4b",source="local_cache_hit"} 1.7833389303328078e+09
199vllm:prompt_tokens_by_source_created{engine="0",model_name="cope-b-a4b",source="external_kv_transfer"} 1.7833389303328118e+09
200# HELP vllm:prompt_tokens_cached_total Number of cached prompt tokens (local + external).
201# TYPE vllm:prompt_tokens_cached_total counter
202vllm:prompt_tokens_cached_total{engine="0",model_name="cope-b-a4b"} 6.577568e+06
203# HELP vllm:prompt_tokens_cached_created Number of cached prompt tokens (local + external).
204# TYPE vllm:prompt_tokens_cached_created gauge
205vllm:prompt_tokens_cached_created{engine="0",model_name="cope-b-a4b"} 1.7833389303328218e+09
206# HELP vllm:generation_tokens_total Number of generation tokens processed.
207# TYPE vllm:generation_tokens_total counter
208vllm:generation_tokens_total{engine="0",model_name="cope-b-a4b"} 4088.0
209# HELP vllm:generation_tokens_created Number of generation tokens processed.
210# TYPE vllm:generation_tokens_created gauge
211vllm:generation_tokens_created{engine="0",model_name="cope-b-a4b"} 1.7833389303328319e+09
212# HELP vllm:request_success_total Count of successfully processed requests.
213# TYPE vllm:request_success_total counter
214vllm:request_success_total{engine="0",finished_reason="stop",model_name="cope-b-a4b"} 0.0
215vllm:request_success_total{engine="0",finished_reason="length",model_name="cope-b-a4b"} 4088.0
216vllm:request_success_total{engine="0",finished_reason="abort",model_name="cope-b-a4b"} 0.0
217vllm:request_success_total{engine="0",finished_reason="error",model_name="cope-b-a4b"} 0.0
218vllm:request_success_total{engine="0",finished_reason="repetition",model_name="cope-b-a4b"} 0.0
219# HELP vllm:request_success_created Count of successfully processed requests.
220# TYPE vllm:request_success_created gauge
221vllm:request_success_created{engine="0",finished_reason="stop",model_name="cope-b-a4b"} 1.7833389303328555e+09
222vllm:request_success_created{engine="0",finished_reason="length",model_name="cope-b-a4b"} 1.7833389303328636e+09
223vllm:request_success_created{engine="0",finished_reason="abort",model_name="cope-b-a4b"} 1.78333893033287e+09
224vllm:request_success_created{engine="0",finished_reason="error",model_name="cope-b-a4b"} 1.7833389303328755e+09
225vllm:request_success_created{engine="0",finished_reason="repetition",model_name="cope-b-a4b"} 1.783338930332881e+09
226# HELP vllm:request_prompt_tokens Number of prefill tokens processed.
227# TYPE vllm:request_prompt_tokens histogram
228vllm:request_prompt_tokens_bucket{engine="0",le="1.0",model_name="cope-b-a4b"} 0.0
229vllm:request_prompt_tokens_bucket{engine="0",le="2.0",model_name="cope-b-a4b"} 0.0
230vllm:request_prompt_tokens_bucket{engine="0",le="5.0",model_name="cope-b-a4b"} 0.0
231vllm:request_prompt_tokens_bucket{engine="0",le="10.0",model_name="cope-b-a4b"} 0.0
232vllm:request_prompt_tokens_bucket{engine="0",le="20.0",model_name="cope-b-a4b"} 0.0
233vllm:request_prompt_tokens_bucket{engine="0",le="50.0",model_name="cope-b-a4b"} 0.0
234vllm:request_prompt_tokens_bucket{engine="0",le="100.0",model_name="cope-b-a4b"} 2.0
235vllm:request_prompt_tokens_bucket{engine="0",le="200.0",model_name="cope-b-a4b"} 2.0
236vllm:request_prompt_tokens_bucket{engine="0",le="500.0",model_name="cope-b-a4b"} 2.0
237vllm:request_prompt_tokens_bucket{engine="0",le="1000.0",model_name="cope-b-a4b"} 2.0
238vllm:request_prompt_tokens_bucket{engine="0",le="2000.0",model_name="cope-b-a4b"} 1303.0
239vllm:request_prompt_tokens_bucket{engine="0",le="5000.0",model_name="cope-b-a4b"} 4066.0
240vllm:request_prompt_tokens_bucket{engine="0",le="10000.0",model_name="cope-b-a4b"} 4088.0
241vllm:request_prompt_tokens_bucket{engine="0",le="+Inf",model_name="cope-b-a4b"} 4088.0
242vllm:request_prompt_tokens_count{engine="0",model_name="cope-b-a4b"} 4088.0
243vllm:request_prompt_tokens_sum{engine="0",model_name="cope-b-a4b"} 1.017793e+07
244# HELP vllm:request_prompt_tokens_created Number of prefill tokens processed.
245# TYPE vllm:request_prompt_tokens_created gauge
246vllm:request_prompt_tokens_created{engine="0",model_name="cope-b-a4b"} 1.7833389303329148e+09
247# HELP vllm:request_generation_tokens Number of generation tokens processed.
248# TYPE vllm:request_generation_tokens histogram
249vllm:request_generation_tokens_bucket{engine="0",le="1.0",model_name="cope-b-a4b"} 4088.0
250vllm:request_generation_tokens_bucket{engine="0",le="2.0",model_name="cope-b-a4b"} 4088.0
251vllm:request_generation_tokens_bucket{engine="0",le="5.0",model_name="cope-b-a4b"} 4088.0
252vllm:request_generation_tokens_bucket{engine="0",le="10.0",model_name="cope-b-a4b"} 4088.0
253vllm:request_generation_tokens_bucket{engine="0",le="20.0",model_name="cope-b-a4b"} 4088.0
254vllm:request_generation_tokens_bucket{engine="0",le="50.0",model_name="cope-b-a4b"} 4088.0
255vllm:request_generation_tokens_bucket{engine="0",le="100.0",model_name="cope-b-a4b"} 4088.0
256vllm:request_generation_tokens_bucket{engine="0",le="200.0",model_name="cope-b-a4b"} 4088.0
257vllm:request_generation_tokens_bucket{engine="0",le="500.0",model_name="cope-b-a4b"} 4088.0
258vllm:request_generation_tokens_bucket{engine="0",le="1000.0",model_name="cope-b-a4b"} 4088.0
259vllm:request_generation_tokens_bucket{engine="0",le="2000.0",model_name="cope-b-a4b"} 4088.0
260vllm:request_generation_tokens_bucket{engine="0",le="5000.0",model_name="cope-b-a4b"} 4088.0
261vllm:request_generation_tokens_bucket{engine="0",le="10000.0",model_name="cope-b-a4b"} 4088.0
262vllm:request_generation_tokens_bucket{engine="0",le="+Inf",model_name="cope-b-a4b"} 4088.0
263vllm:request_generation_tokens_count{engine="0",model_name="cope-b-a4b"} 4088.0
264vllm:request_generation_tokens_sum{engine="0",model_name="cope-b-a4b"} 4088.0
265# HELP vllm:request_generation_tokens_created Number of generation tokens processed.
266# TYPE vllm:request_generation_tokens_created gauge
267vllm:request_generation_tokens_created{engine="0",model_name="cope-b-a4b"} 1.78333893033296e+09
268# HELP vllm:iteration_tokens_total Histogram of number of tokens per engine_step.
269# TYPE vllm:iteration_tokens_total histogram
270vllm:iteration_tokens_total_bucket{engine="0",le="1.0",model_name="cope-b-a4b"} 0.0
271vllm:iteration_tokens_total_bucket{engine="0",le="8.0",model_name="cope-b-a4b"} 0.0
272vllm:iteration_tokens_total_bucket{engine="0",le="16.0",model_name="cope-b-a4b"} 0.0
273vllm:iteration_tokens_total_bucket{engine="0",le="32.0",model_name="cope-b-a4b"} 0.0
274vllm:iteration_tokens_total_bucket{engine="0",le="64.0",model_name="cope-b-a4b"} 0.0
275vllm:iteration_tokens_total_bucket{engine="0",le="128.0",model_name="cope-b-a4b"} 2.0
276vllm:iteration_tokens_total_bucket{engine="0",le="256.0",model_name="cope-b-a4b"} 2.0
277vllm:iteration_tokens_total_bucket{engine="0",le="512.0",model_name="cope-b-a4b"} 2.0
278vllm:iteration_tokens_total_bucket{engine="0",le="1024.0",model_name="cope-b-a4b"} 2.0
279vllm:iteration_tokens_total_bucket{engine="0",le="2048.0",model_name="cope-b-a4b"} 1487.0
280vllm:iteration_tokens_total_bucket{engine="0",le="4096.0",model_name="cope-b-a4b"} 3831.0
281vllm:iteration_tokens_total_bucket{engine="0",le="8192.0",model_name="cope-b-a4b"} 4088.0
282vllm:iteration_tokens_total_bucket{engine="0",le="16384.0",model_name="cope-b-a4b"} 4088.0
283vllm:iteration_tokens_total_bucket{engine="0",le="+Inf",model_name="cope-b-a4b"} 4088.0
284vllm:iteration_tokens_total_count{engine="0",model_name="cope-b-a4b"} 4088.0
285vllm:iteration_tokens_total_sum{engine="0",model_name="cope-b-a4b"} 1.0182018e+07
286# HELP vllm:iteration_tokens_total_created Histogram of number of tokens per engine_step.
287# TYPE vllm:iteration_tokens_total_created gauge
288vllm:iteration_tokens_total_created{engine="0",model_name="cope-b-a4b"} 1.783338930332987e+09
289# HELP vllm:request_max_num_generation_tokens Histogram of maximum number of requested generation tokens.
290# TYPE vllm:request_max_num_generation_tokens histogram
291vllm:request_max_num_generation_tokens_bucket{engine="0",le="1.0",model_name="cope-b-a4b"} 4088.0
292vllm:request_max_num_generation_tokens_bucket{engine="0",le="2.0",model_name="cope-b-a4b"} 4088.0
293vllm:request_max_num_generation_tokens_bucket{engine="0",le="5.0",model_name="cope-b-a4b"} 4088.0
294vllm:request_max_num_generation_tokens_bucket{engine="0",le="10.0",model_name="cope-b-a4b"} 4088.0
295vllm:request_max_num_generation_tokens_bucket{engine="0",le="20.0",model_name="cope-b-a4b"} 4088.0
296vllm:request_max_num_generation_tokens_bucket{engine="0",le="50.0",model_name="cope-b-a4b"} 4088.0
297vllm:request_max_num_generation_tokens_bucket{engine="0",le="100.0",model_name="cope-b-a4b"} 4088.0
298vllm:request_max_num_generation_tokens_bucket{engine="0",le="200.0",model_name="cope-b-a4b"} 4088.0
299vllm:request_max_num_generation_tokens_bucket{engine="0",le="500.0",model_name="cope-b-a4b"} 4088.0
300vllm:request_max_num_generation_tokens_bucket{engine="0",le="1000.0",model_name="cope-b-a4b"} 4088.0
301vllm:request_max_num_generation_tokens_bucket{engine="0",le="2000.0",model_name="cope-b-a4b"} 4088.0
302vllm:request_max_num_generation_tokens_bucket{engine="0",le="5000.0",model_name="cope-b-a4b"} 4088.0
303vllm:request_max_num_generation_tokens_bucket{engine="0",le="10000.0",model_name="cope-b-a4b"} 4088.0
304vllm:request_max_num_generation_tokens_bucket{engine="0",le="+Inf",model_name="cope-b-a4b"} 4088.0
305vllm:request_max_num_generation_tokens_count{engine="0",model_name="cope-b-a4b"} 4088.0
306vllm:request_max_num_generation_tokens_sum{engine="0",model_name="cope-b-a4b"} 4088.0
307# HELP vllm:request_max_num_generation_tokens_created Histogram of maximum number of requested generation tokens.
308# TYPE vllm:request_max_num_generation_tokens_created gauge
309vllm:request_max_num_generation_tokens_created{engine="0",model_name="cope-b-a4b"} 1.783338930333016e+09
310# HELP vllm:request_params_n Histogram of the n request parameter.
311# TYPE vllm:request_params_n histogram
312vllm:request_params_n_bucket{engine="0",le="1.0",model_name="cope-b-a4b"} 4088.0
313vllm:request_params_n_bucket{engine="0",le="2.0",model_name="cope-b-a4b"} 4088.0
314vllm:request_params_n_bucket{engine="0",le="5.0",model_name="cope-b-a4b"} 4088.0
315vllm:request_params_n_bucket{engine="0",le="10.0",model_name="cope-b-a4b"} 4088.0
316vllm:request_params_n_bucket{engine="0",le="20.0",model_name="cope-b-a4b"} 4088.0
317vllm:request_params_n_bucket{engine="0",le="+Inf",model_name="cope-b-a4b"} 4088.0
318vllm:request_params_n_count{engine="0",model_name="cope-b-a4b"} 4088.0
319vllm:request_params_n_sum{engine="0",model_name="cope-b-a4b"} 4088.0
320# HELP vllm:request_params_n_created Histogram of the n request parameter.
321# TYPE vllm:request_params_n_created gauge
322vllm:request_params_n_created{engine="0",model_name="cope-b-a4b"} 1.7833389303330402e+09
323# HELP vllm:request_params_max_tokens Histogram of the max_tokens request parameter.
324# TYPE vllm:request_params_max_tokens histogram
325vllm:request_params_max_tokens_bucket{engine="0",le="1.0",model_name="cope-b-a4b"} 4088.0
326vllm:request_params_max_tokens_bucket{engine="0",le="2.0",model_name="cope-b-a4b"} 4088.0
327vllm:request_params_max_tokens_bucket{engine="0",le="5.0",model_name="cope-b-a4b"} 4088.0
328vllm:request_params_max_tokens_bucket{engine="0",le="10.0",model_name="cope-b-a4b"} 4088.0
329vllm:request_params_max_tokens_bucket{engine="0",le="20.0",model_name="cope-b-a4b"} 4088.0
330vllm:request_params_max_tokens_bucket{engine="0",le="50.0",model_name="cope-b-a4b"} 4088.0
331vllm:request_params_max_tokens_bucket{engine="0",le="100.0",model_name="cope-b-a4b"} 4088.0
332vllm:request_params_max_tokens_bucket{engine="0",le="200.0",model_name="cope-b-a4b"} 4088.0
333vllm:request_params_max_tokens_bucket{engine="0",le="500.0",model_name="cope-b-a4b"} 4088.0
334vllm:request_params_max_tokens_bucket{engine="0",le="1000.0",model_name="cope-b-a4b"} 4088.0
335vllm:request_params_max_tokens_bucket{engine="0",le="2000.0",model_name="cope-b-a4b"} 4088.0
336vllm:request_params_max_tokens_bucket{engine="0",le="5000.0",model_name="cope-b-a4b"} 4088.0
337vllm:request_params_max_tokens_bucket{engine="0",le="10000.0",model_name="cope-b-a4b"} 4088.0
338vllm:request_params_max_tokens_bucket{engine="0",le="+Inf",model_name="cope-b-a4b"} 4088.0
339vllm:request_params_max_tokens_count{engine="0",model_name="cope-b-a4b"} 4088.0
340vllm:request_params_max_tokens_sum{engine="0",model_name="cope-b-a4b"} 4088.0
341# HELP vllm:request_params_max_tokens_created Histogram of the max_tokens request parameter.
342# TYPE vllm:request_params_max_tokens_created gauge
343vllm:request_params_max_tokens_created{engine="0",model_name="cope-b-a4b"} 1.7833389303330612e+09
344# HELP vllm:time_to_first_token_seconds Histogram of time to first token in seconds.
345# TYPE vllm:time_to_first_token_seconds histogram
346vllm:time_to_first_token_seconds_bucket{engine="0",le="0.001",model_name="cope-b-a4b"} 0.0
347vllm:time_to_first_token_seconds_bucket{engine="0",le="0.005",model_name="cope-b-a4b"} 0.0
348vllm:time_to_first_token_seconds_bucket{engine="0",le="0.01",model_name="cope-b-a4b"} 0.0
349vllm:time_to_first_token_seconds_bucket{engine="0",le="0.02",model_name="cope-b-a4b"} 3.0
350vllm:time_to_first_token_seconds_bucket{engine="0",le="0.04",model_name="cope-b-a4b"} 1746.0
351vllm:time_to_first_token_seconds_bucket{engine="0",le="0.06",model_name="cope-b-a4b"} 3089.0
352vllm:time_to_first_token_seconds_bucket{engine="0",le="0.08",model_name="cope-b-a4b"} 3561.0
353vllm:time_to_first_token_seconds_bucket{engine="0",le="0.1",model_name="cope-b-a4b"} 3782.0
354vllm:time_to_first_token_seconds_bucket{engine="0",le="0.25",model_name="cope-b-a4b"} 4086.0
355vllm:time_to_first_token_seconds_bucket{engine="0",le="0.5",model_name="cope-b-a4b"} 4086.0
356vllm:time_to_first_token_seconds_bucket{engine="0",le="0.75",model_name="cope-b-a4b"} 4087.0
357vllm:time_to_first_token_seconds_bucket{engine="0",le="1.0",model_name="cope-b-a4b"} 4087.0
358vllm:time_to_first_token_seconds_bucket{engine="0",le="2.5",model_name="cope-b-a4b"} 4088.0
359vllm:time_to_first_token_seconds_bucket{engine="0",le="5.0",model_name="cope-b-a4b"} 4088.0
360vllm:time_to_first_token_seconds_bucket{engine="0",le="7.5",model_name="cope-b-a4b"} 4088.0
361vllm:time_to_first_token_seconds_bucket{engine="0",le="10.0",model_name="cope-b-a4b"} 4088.0
362vllm:time_to_first_token_seconds_bucket{engine="0",le="20.0",model_name="cope-b-a4b"} 4088.0
363vllm:time_to_first_token_seconds_bucket{engine="0",le="40.0",model_name="cope-b-a4b"} 4088.0
364vllm:time_to_first_token_seconds_bucket{engine="0",le="80.0",model_name="cope-b-a4b"} 4088.0
365vllm:time_to_first_token_seconds_bucket{engine="0",le="160.0",model_name="cope-b-a4b"} 4088.0
366vllm:time_to_first_token_seconds_bucket{engine="0",le="640.0",model_name="cope-b-a4b"} 4088.0
367vllm:time_to_first_token_seconds_bucket{engine="0",le="2560.0",model_name="cope-b-a4b"} 4088.0
368vllm:time_to_first_token_seconds_bucket{engine="0",le="+Inf",model_name="cope-b-a4b"} 4088.0
369vllm:time_to_first_token_seconds_count{engine="0",model_name="cope-b-a4b"} 4088.0
370vllm:time_to_first_token_seconds_sum{engine="0",model_name="cope-b-a4b"} 215.77645802497864
371# HELP vllm:time_to_first_token_seconds_created Histogram of time to first token in seconds.
372# TYPE vllm:time_to_first_token_seconds_created gauge
373vllm:time_to_first_token_seconds_created{engine="0",model_name="cope-b-a4b"} 1.783338930333143e+09
374# HELP vllm:inter_token_latency_seconds Histogram of inter-token latency in seconds.
375# TYPE vllm:inter_token_latency_seconds histogram
376vllm:inter_token_latency_seconds_bucket{engine="0",le="0.01",model_name="cope-b-a4b"} 0.0
377vllm:inter_token_latency_seconds_bucket{engine="0",le="0.025",model_name="cope-b-a4b"} 0.0
378vllm:inter_token_latency_seconds_bucket{engine="0",le="0.05",model_name="cope-b-a4b"} 0.0
379vllm:inter_token_latency_seconds_bucket{engine="0",le="0.075",model_name="cope-b-a4b"} 0.0
380vllm:inter_token_latency_seconds_bucket{engine="0",le="0.1",model_name="cope-b-a4b"} 0.0
381vllm:inter_token_latency_seconds_bucket{engine="0",le="0.15",model_name="cope-b-a4b"} 0.0
382vllm:inter_token_latency_seconds_bucket{engine="0",le="0.2",model_name="cope-b-a4b"} 0.0
383vllm:inter_token_latency_seconds_bucket{engine="0",le="0.3",model_name="cope-b-a4b"} 0.0
384vllm:inter_token_latency_seconds_bucket{engine="0",le="0.4",model_name="cope-b-a4b"} 0.0
385vllm:inter_token_latency_seconds_bucket{engine="0",le="0.5",model_name="cope-b-a4b"} 0.0
386vllm:inter_token_latency_seconds_bucket{engine="0",le="0.75",model_name="cope-b-a4b"} 0.0
387vllm:inter_token_latency_seconds_bucket{engine="0",le="1.0",model_name="cope-b-a4b"} 0.0
388vllm:inter_token_latency_seconds_bucket{engine="0",le="2.5",model_name="cope-b-a4b"} 0.0
389vllm:inter_token_latency_seconds_bucket{engine="0",le="5.0",model_name="cope-b-a4b"} 0.0
390vllm:inter_token_latency_seconds_bucket{engine="0",le="7.5",model_name="cope-b-a4b"} 0.0
391vllm:inter_token_latency_seconds_bucket{engine="0",le="10.0",model_name="cope-b-a4b"} 0.0
392vllm:inter_token_latency_seconds_bucket{engine="0",le="20.0",model_name="cope-b-a4b"} 0.0
393vllm:inter_token_latency_seconds_bucket{engine="0",le="40.0",model_name="cope-b-a4b"} 0.0
394vllm:inter_token_latency_seconds_bucket{engine="0",le="80.0",model_name="cope-b-a4b"} 0.0
395vllm:inter_token_latency_seconds_bucket{engine="0",le="+Inf",model_name="cope-b-a4b"} 0.0
396vllm:inter_token_latency_seconds_count{engine="0",model_name="cope-b-a4b"} 0.0
397vllm:inter_token_latency_seconds_sum{engine="0",model_name="cope-b-a4b"} 0.0
398# HELP vllm:inter_token_latency_seconds_created Histogram of inter-token latency in seconds.
399# TYPE vllm:inter_token_latency_seconds_created gauge
400vllm:inter_token_latency_seconds_created{engine="0",model_name="cope-b-a4b"} 1.7833389303331869e+09
401# HELP vllm:request_time_per_output_token_seconds Histogram of time_per_output_token_seconds per request.
402# TYPE vllm:request_time_per_output_token_seconds histogram
403vllm:request_time_per_output_token_seconds_bucket{engine="0",le="0.01",model_name="cope-b-a4b"} 4088.0
404vllm:request_time_per_output_token_seconds_bucket{engine="0",le="0.025",model_name="cope-b-a4b"} 4088.0
405vllm:request_time_per_output_token_seconds_bucket{engine="0",le="0.05",model_name="cope-b-a4b"} 4088.0
406vllm:request_time_per_output_token_seconds_bucket{engine="0",le="0.075",model_name="cope-b-a4b"} 4088.0
407vllm:request_time_per_output_token_seconds_bucket{engine="0",le="0.1",model_name="cope-b-a4b"} 4088.0
408vllm:request_time_per_output_token_seconds_bucket{engine="0",le="0.15",model_name="cope-b-a4b"} 4088.0
409vllm:request_time_per_output_token_seconds_bucket{engine="0",le="0.2",model_name="cope-b-a4b"} 4088.0
410vllm:request_time_per_output_token_seconds_bucket{engine="0",le="0.3",model_name="cope-b-a4b"} 4088.0
411vllm:request_time_per_output_token_seconds_bucket{engine="0",le="0.4",model_name="cope-b-a4b"} 4088.0
412vllm:request_time_per_output_token_seconds_bucket{engine="0",le="0.5",model_name="cope-b-a4b"} 4088.0
413vllm:request_time_per_output_token_seconds_bucket{engine="0",le="0.75",model_name="cope-b-a4b"} 4088.0
414vllm:request_time_per_output_token_seconds_bucket{engine="0",le="1.0",model_name="cope-b-a4b"} 4088.0
415vllm:request_time_per_output_token_seconds_bucket{engine="0",le="2.5",model_name="cope-b-a4b"} 4088.0
416vllm:request_time_per_output_token_seconds_bucket{engine="0",le="5.0",model_name="cope-b-a4b"} 4088.0
417vllm:request_time_per_output_token_seconds_bucket{engine="0",le="7.5",model_name="cope-b-a4b"} 4088.0
418vllm:request_time_per_output_token_seconds_bucket{engine="0",le="10.0",model_name="cope-b-a4b"} 4088.0
419vllm:request_time_per_output_token_seconds_bucket{engine="0",le="20.0",model_name="cope-b-a4b"} 4088.0
420vllm:request_time_per_output_token_seconds_bucket{engine="0",le="40.0",model_name="cope-b-a4b"} 4088.0
421vllm:request_time_per_output_token_seconds_bucket{engine="0",le="80.0",model_name="cope-b-a4b"} 4088.0
422vllm:request_time_per_output_token_seconds_bucket{engine="0",le="+Inf",model_name="cope-b-a4b"} 4088.0
423vllm:request_time_per_output_token_seconds_count{engine="0",model_name="cope-b-a4b"} 4088.0
424vllm:request_time_per_output_token_seconds_sum{engine="0",model_name="cope-b-a4b"} 0.0
425# HELP vllm:request_time_per_output_token_seconds_created Histogram of time_per_output_token_seconds per request.
426# TYPE vllm:request_time_per_output_token_seconds_created gauge
427vllm:request_time_per_output_token_seconds_created{engine="0",model_name="cope-b-a4b"} 1.7833389303332179e+09
428# HELP vllm:e2e_request_latency_seconds Histogram of e2e request latency in seconds.
429# TYPE vllm:e2e_request_latency_seconds histogram
430vllm:e2e_request_latency_seconds_bucket{engine="0",le="0.3",model_name="cope-b-a4b"} 4086.0
431vllm:e2e_request_latency_seconds_bucket{engine="0",le="0.5",model_name="cope-b-a4b"} 4086.0
432vllm:e2e_request_latency_seconds_bucket{engine="0",le="0.8",model_name="cope-b-a4b"} 4087.0
433vllm:e2e_request_latency_seconds_bucket{engine="0",le="1.0",model_name="cope-b-a4b"} 4087.0
434vllm:e2e_request_latency_seconds_bucket{engine="0",le="1.5",model_name="cope-b-a4b"} 4087.0
435vllm:e2e_request_latency_seconds_bucket{engine="0",le="2.0",model_name="cope-b-a4b"} 4087.0
436vllm:e2e_request_latency_seconds_bucket{engine="0",le="2.5",model_name="cope-b-a4b"} 4088.0
437vllm:e2e_request_latency_seconds_bucket{engine="0",le="5.0",model_name="cope-b-a4b"} 4088.0
438vllm:e2e_request_latency_seconds_bucket{engine="0",le="10.0",model_name="cope-b-a4b"} 4088.0
439vllm:e2e_request_latency_seconds_bucket{engine="0",le="15.0",model_name="cope-b-a4b"} 4088.0
440vllm:e2e_request_latency_seconds_bucket{engine="0",le="20.0",model_name="cope-b-a4b"} 4088.0
441vllm:e2e_request_latency_seconds_bucket{engine="0",le="30.0",model_name="cope-b-a4b"} 4088.0
442vllm:e2e_request_latency_seconds_bucket{engine="0",le="40.0",model_name="cope-b-a4b"} 4088.0
443vllm:e2e_request_latency_seconds_bucket{engine="0",le="50.0",model_name="cope-b-a4b"} 4088.0
444vllm:e2e_request_latency_seconds_bucket{engine="0",le="60.0",model_name="cope-b-a4b"} 4088.0
445vllm:e2e_request_latency_seconds_bucket{engine="0",le="120.0",model_name="cope-b-a4b"} 4088.0
446vllm:e2e_request_latency_seconds_bucket{engine="0",le="240.0",model_name="cope-b-a4b"} 4088.0
447vllm:e2e_request_latency_seconds_bucket{engine="0",le="480.0",model_name="cope-b-a4b"} 4088.0
448vllm:e2e_request_latency_seconds_bucket{engine="0",le="960.0",model_name="cope-b-a4b"} 4088.0
449vllm:e2e_request_latency_seconds_bucket{engine="0",le="1920.0",model_name="cope-b-a4b"} 4088.0
450vllm:e2e_request_latency_seconds_bucket{engine="0",le="7680.0",model_name="cope-b-a4b"} 4088.0
451vllm:e2e_request_latency_seconds_bucket{engine="0",le="+Inf",model_name="cope-b-a4b"} 4088.0
452vllm:e2e_request_latency_seconds_count{engine="0",model_name="cope-b-a4b"} 4088.0
453vllm:e2e_request_latency_seconds_sum{engine="0",model_name="cope-b-a4b"} 215.77645802497864
454# HELP vllm:e2e_request_latency_seconds_created Histogram of e2e request latency in seconds.
455# TYPE vllm:e2e_request_latency_seconds_created gauge
456vllm:e2e_request_latency_seconds_created{engine="0",model_name="cope-b-a4b"} 1.78333893033325e+09
457# HELP vllm:request_queue_time_seconds Histogram of time spent in WAITING phase for request.
458# TYPE vllm:request_queue_time_seconds histogram
459vllm:request_queue_time_seconds_bucket{engine="0",le="0.3",model_name="cope-b-a4b"} 4088.0
460vllm:request_queue_time_seconds_bucket{engine="0",le="0.5",model_name="cope-b-a4b"} 4088.0
461vllm:request_queue_time_seconds_bucket{engine="0",le="0.8",model_name="cope-b-a4b"} 4088.0
462vllm:request_queue_time_seconds_bucket{engine="0",le="1.0",model_name="cope-b-a4b"} 4088.0
463vllm:request_queue_time_seconds_bucket{engine="0",le="1.5",model_name="cope-b-a4b"} 4088.0
464vllm:request_queue_time_seconds_bucket{engine="0",le="2.0",model_name="cope-b-a4b"} 4088.0
465vllm:request_queue_time_seconds_bucket{engine="0",le="2.5",model_name="cope-b-a4b"} 4088.0
466vllm:request_queue_time_seconds_bucket{engine="0",le="5.0",model_name="cope-b-a4b"} 4088.0
467vllm:request_queue_time_seconds_bucket{engine="0",le="10.0",model_name="cope-b-a4b"} 4088.0
468vllm:request_queue_time_seconds_bucket{engine="0",le="15.0",model_name="cope-b-a4b"} 4088.0
469vllm:request_queue_time_seconds_bucket{engine="0",le="20.0",model_name="cope-b-a4b"} 4088.0
470vllm:request_queue_time_seconds_bucket{engine="0",le="30.0",model_name="cope-b-a4b"} 4088.0
471vllm:request_queue_time_seconds_bucket{engine="0",le="40.0",model_name="cope-b-a4b"} 4088.0
472vllm:request_queue_time_seconds_bucket{engine="0",le="50.0",model_name="cope-b-a4b"} 4088.0
473vllm:request_queue_time_seconds_bucket{engine="0",le="60.0",model_name="cope-b-a4b"} 4088.0
474vllm:request_queue_time_seconds_bucket{engine="0",le="120.0",model_name="cope-b-a4b"} 4088.0
475vllm:request_queue_time_seconds_bucket{engine="0",le="240.0",model_name="cope-b-a4b"} 4088.0
476vllm:request_queue_time_seconds_bucket{engine="0",le="480.0",model_name="cope-b-a4b"} 4088.0
477vllm:request_queue_time_seconds_bucket{engine="0",le="960.0",model_name="cope-b-a4b"} 4088.0
478vllm:request_queue_time_seconds_bucket{engine="0",le="1920.0",model_name="cope-b-a4b"} 4088.0
479vllm:request_queue_time_seconds_bucket{engine="0",le="7680.0",model_name="cope-b-a4b"} 4088.0
480vllm:request_queue_time_seconds_bucket{engine="0",le="+Inf",model_name="cope-b-a4b"} 4088.0
481vllm:request_queue_time_seconds_count{engine="0",model_name="cope-b-a4b"} 4088.0
482vllm:request_queue_time_seconds_sum{engine="0",model_name="cope-b-a4b"} 0.03302581515163183
483# HELP vllm:request_queue_time_seconds_created Histogram of time spent in WAITING phase for request.
484# TYPE vllm:request_queue_time_seconds_created gauge
485vllm:request_queue_time_seconds_created{engine="0",model_name="cope-b-a4b"} 1.7833389303332808e+09
486# HELP vllm:request_inference_time_seconds Histogram of time spent in RUNNING phase for request.
487# TYPE vllm:request_inference_time_seconds histogram
488vllm:request_inference_time_seconds_bucket{engine="0",le="0.3",model_name="cope-b-a4b"} 4086.0
489vllm:request_inference_time_seconds_bucket{engine="0",le="0.5",model_name="cope-b-a4b"} 4086.0
490vllm:request_inference_time_seconds_bucket{engine="0",le="0.8",model_name="cope-b-a4b"} 4087.0
491vllm:request_inference_time_seconds_bucket{engine="0",le="1.0",model_name="cope-b-a4b"} 4087.0
492vllm:request_inference_time_seconds_bucket{engine="0",le="1.5",model_name="cope-b-a4b"} 4087.0
493vllm:request_inference_time_seconds_bucket{engine="0",le="2.0",model_name="cope-b-a4b"} 4087.0
494vllm:request_inference_time_seconds_bucket{engine="0",le="2.5",model_name="cope-b-a4b"} 4088.0
495vllm:request_inference_time_seconds_bucket{engine="0",le="5.0",model_name="cope-b-a4b"} 4088.0
496vllm:request_inference_time_seconds_bucket{engine="0",le="10.0",model_name="cope-b-a4b"} 4088.0
497vllm:request_inference_time_seconds_bucket{engine="0",le="15.0",model_name="cope-b-a4b"} 4088.0
498vllm:request_inference_time_seconds_bucket{engine="0",le="20.0",model_name="cope-b-a4b"} 4088.0
499vllm:request_inference_time_seconds_bucket{engine="0",le="30.0",model_name="cope-b-a4b"} 4088.0
500vllm:request_inference_time_seconds_bucket{engine="0",le="40.0",model_name="cope-b-a4b"} 4088.0
501vllm:request_inference_time_seconds_bucket{engine="0",le="50.0",model_name="cope-b-a4b"} 4088.0
502vllm:request_inference_time_seconds_bucket{engine="0",le="60.0",model_name="cope-b-a4b"} 4088.0
503vllm:request_inference_time_seconds_bucket{engine="0",le="120.0",model_name="cope-b-a4b"} 4088.0
504vllm:request_inference_time_seconds_bucket{engine="0",le="240.0",model_name="cope-b-a4b"} 4088.0
505vllm:request_inference_time_seconds_bucket{engine="0",le="480.0",model_name="cope-b-a4b"} 4088.0
506vllm:request_inference_time_seconds_bucket{engine="0",le="960.0",model_name="cope-b-a4b"} 4088.0
507vllm:request_inference_time_seconds_bucket{engine="0",le="1920.0",model_name="cope-b-a4b"} 4088.0
508vllm:request_inference_time_seconds_bucket{engine="0",le="7680.0",model_name="cope-b-a4b"} 4088.0
509vllm:request_inference_time_seconds_bucket{engine="0",le="+Inf",model_name="cope-b-a4b"} 4088.0
510vllm:request_inference_time_seconds_count{engine="0",model_name="cope-b-a4b"} 4088.0
511vllm:request_inference_time_seconds_sum{engine="0",model_name="cope-b-a4b"} 193.81161683239043
512# HELP vllm:request_inference_time_seconds_created Histogram of time spent in RUNNING phase for request.
513# TYPE vllm:request_inference_time_seconds_created gauge
514vllm:request_inference_time_seconds_created{engine="0",model_name="cope-b-a4b"} 1.7833389303333106e+09
515# HELP vllm:request_prefill_time_seconds Histogram of time spent in PREFILL phase for request.
516# TYPE vllm:request_prefill_time_seconds histogram
517vllm:request_prefill_time_seconds_bucket{engine="0",le="0.3",model_name="cope-b-a4b"} 4086.0
518vllm:request_prefill_time_seconds_bucket{engine="0",le="0.5",model_name="cope-b-a4b"} 4086.0
519vllm:request_prefill_time_seconds_bucket{engine="0",le="0.8",model_name="cope-b-a4b"} 4087.0
520vllm:request_prefill_time_seconds_bucket{engine="0",le="1.0",model_name="cope-b-a4b"} 4087.0
521vllm:request_prefill_time_seconds_bucket{engine="0",le="1.5",model_name="cope-b-a4b"} 4087.0
522vllm:request_prefill_time_seconds_bucket{engine="0",le="2.0",model_name="cope-b-a4b"} 4087.0
523vllm:request_prefill_time_seconds_bucket{engine="0",le="2.5",model_name="cope-b-a4b"} 4088.0
524vllm:request_prefill_time_seconds_bucket{engine="0",le="5.0",model_name="cope-b-a4b"} 4088.0
525vllm:request_prefill_time_seconds_bucket{engine="0",le="10.0",model_name="cope-b-a4b"} 4088.0
526vllm:request_prefill_time_seconds_bucket{engine="0",le="15.0",model_name="cope-b-a4b"} 4088.0
527vllm:request_prefill_time_seconds_bucket{engine="0",le="20.0",model_name="cope-b-a4b"} 4088.0
528vllm:request_prefill_time_seconds_bucket{engine="0",le="30.0",model_name="cope-b-a4b"} 4088.0
529vllm:request_prefill_time_seconds_bucket{engine="0",le="40.0",model_name="cope-b-a4b"} 4088.0
530vllm:request_prefill_time_seconds_bucket{engine="0",le="50.0",model_name="cope-b-a4b"} 4088.0
531vllm:request_prefill_time_seconds_bucket{engine="0",le="60.0",model_name="cope-b-a4b"} 4088.0
532vllm:request_prefill_time_seconds_bucket{engine="0",le="120.0",model_name="cope-b-a4b"} 4088.0
533vllm:request_prefill_time_seconds_bucket{engine="0",le="240.0",model_name="cope-b-a4b"} 4088.0
534vllm:request_prefill_time_seconds_bucket{engine="0",le="480.0",model_name="cope-b-a4b"} 4088.0
535vllm:request_prefill_time_seconds_bucket{engine="0",le="960.0",model_name="cope-b-a4b"} 4088.0
536vllm:request_prefill_time_seconds_bucket{engine="0",le="1920.0",model_name="cope-b-a4b"} 4088.0
537vllm:request_prefill_time_seconds_bucket{engine="0",le="7680.0",model_name="cope-b-a4b"} 4088.0
538vllm:request_prefill_time_seconds_bucket{engine="0",le="+Inf",model_name="cope-b-a4b"} 4088.0
539vllm:request_prefill_time_seconds_count{engine="0",model_name="cope-b-a4b"} 4088.0
540vllm:request_prefill_time_seconds_sum{engine="0",model_name="cope-b-a4b"} 193.81161683239043
541# HELP vllm:request_prefill_time_seconds_created Histogram of time spent in PREFILL phase for request.
542# TYPE vllm:request_prefill_time_seconds_created gauge
543vllm:request_prefill_time_seconds_created{engine="0",model_name="cope-b-a4b"} 1.7833389303333404e+09
544# HELP vllm:request_decode_time_seconds Histogram of time spent in DECODE phase for request.
545# TYPE vllm:request_decode_time_seconds histogram
546vllm:request_decode_time_seconds_bucket{engine="0",le="0.3",model_name="cope-b-a4b"} 4088.0
547vllm:request_decode_time_seconds_bucket{engine="0",le="0.5",model_name="cope-b-a4b"} 4088.0
548vllm:request_decode_time_seconds_bucket{engine="0",le="0.8",model_name="cope-b-a4b"} 4088.0
549vllm:request_decode_time_seconds_bucket{engine="0",le="1.0",model_name="cope-b-a4b"} 4088.0
550vllm:request_decode_time_seconds_bucket{engine="0",le="1.5",model_name="cope-b-a4b"} 4088.0
551vllm:request_decode_time_seconds_bucket{engine="0",le="2.0",model_name="cope-b-a4b"} 4088.0
552vllm:request_decode_time_seconds_bucket{engine="0",le="2.5",model_name="cope-b-a4b"} 4088.0
553vllm:request_decode_time_seconds_bucket{engine="0",le="5.0",model_name="cope-b-a4b"} 4088.0
554vllm:request_decode_time_seconds_bucket{engine="0",le="10.0",model_name="cope-b-a4b"} 4088.0
555vllm:request_decode_time_seconds_bucket{engine="0",le="15.0",model_name="cope-b-a4b"} 4088.0
556vllm:request_decode_time_seconds_bucket{engine="0",le="20.0",model_name="cope-b-a4b"} 4088.0
557vllm:request_decode_time_seconds_bucket{engine="0",le="30.0",model_name="cope-b-a4b"} 4088.0
558vllm:request_decode_time_seconds_bucket{engine="0",le="40.0",model_name="cope-b-a4b"} 4088.0
559vllm:request_decode_time_seconds_bucket{engine="0",le="50.0",model_name="cope-b-a4b"} 4088.0
560vllm:request_decode_time_seconds_bucket{engine="0",le="60.0",model_name="cope-b-a4b"} 4088.0
561vllm:request_decode_time_seconds_bucket{engine="0",le="120.0",model_name="cope-b-a4b"} 4088.0
562vllm:request_decode_time_seconds_bucket{engine="0",le="240.0",model_name="cope-b-a4b"} 4088.0
563vllm:request_decode_time_seconds_bucket{engine="0",le="480.0",model_name="cope-b-a4b"} 4088.0
564vllm:request_decode_time_seconds_bucket{engine="0",le="960.0",model_name="cope-b-a4b"} 4088.0
565vllm:request_decode_time_seconds_bucket{engine="0",le="1920.0",model_name="cope-b-a4b"} 4088.0
566vllm:request_decode_time_seconds_bucket{engine="0",le="7680.0",model_name="cope-b-a4b"} 4088.0
567vllm:request_decode_time_seconds_bucket{engine="0",le="+Inf",model_name="cope-b-a4b"} 4088.0
568vllm:request_decode_time_seconds_count{engine="0",model_name="cope-b-a4b"} 4088.0
569vllm:request_decode_time_seconds_sum{engine="0",model_name="cope-b-a4b"} 0.0
570# HELP vllm:request_decode_time_seconds_created Histogram of time spent in DECODE phase for request.
571# TYPE vllm:request_decode_time_seconds_created gauge
572vllm:request_decode_time_seconds_created{engine="0",model_name="cope-b-a4b"} 1.7833389303333695e+09
573# HELP vllm:request_prefill_kv_computed_tokens Histogram of new KV tokens computed during prefill (excluding cached tokens).
574# TYPE vllm:request_prefill_kv_computed_tokens histogram
575vllm:request_prefill_kv_computed_tokens_bucket{engine="0",le="1.0",model_name="cope-b-a4b"} 0.0
576vllm:request_prefill_kv_computed_tokens_bucket{engine="0",le="2.0",model_name="cope-b-a4b"} 0.0
577vllm:request_prefill_kv_computed_tokens_bucket{engine="0",le="5.0",model_name="cope-b-a4b"} 0.0
578vllm:request_prefill_kv_computed_tokens_bucket{engine="0",le="10.0",model_name="cope-b-a4b"} 0.0
579vllm:request_prefill_kv_computed_tokens_bucket{engine="0",le="20.0",model_name="cope-b-a4b"} 0.0
580vllm:request_prefill_kv_computed_tokens_bucket{engine="0",le="50.0",model_name="cope-b-a4b"} 2.0
581vllm:request_prefill_kv_computed_tokens_bucket{engine="0",le="100.0",model_name="cope-b-a4b"} 26.0
582vllm:request_prefill_kv_computed_tokens_bucket{engine="0",le="200.0",model_name="cope-b-a4b"} 203.0
583vllm:request_prefill_kv_computed_tokens_bucket{engine="0",le="500.0",model_name="cope-b-a4b"} 1705.0
584vllm:request_prefill_kv_computed_tokens_bucket{engine="0",le="1000.0",model_name="cope-b-a4b"} 2850.0
585vllm:request_prefill_kv_computed_tokens_bucket{engine="0",le="2000.0",model_name="cope-b-a4b"} 3655.0
586vllm:request_prefill_kv_computed_tokens_bucket{engine="0",le="5000.0",model_name="cope-b-a4b"} 4088.0
587vllm:request_prefill_kv_computed_tokens_bucket{engine="0",le="10000.0",model_name="cope-b-a4b"} 4088.0
588vllm:request_prefill_kv_computed_tokens_bucket{engine="0",le="+Inf",model_name="cope-b-a4b"} 4088.0
589vllm:request_prefill_kv_computed_tokens_count{engine="0",model_name="cope-b-a4b"} 4088.0
590vllm:request_prefill_kv_computed_tokens_sum{engine="0",model_name="cope-b-a4b"} 3.600362e+06
591# HELP vllm:request_prefill_kv_computed_tokens_created Histogram of new KV tokens computed during prefill (excluding cached tokens).
592# TYPE vllm:request_prefill_kv_computed_tokens_created gauge
593vllm:request_prefill_kv_computed_tokens_created{engine="0",model_name="cope-b-a4b"} 1.7833389303334012e+09
594# HELP vllm:cache_config_info Information of the LLMEngine CacheConfig
595# TYPE vllm:cache_config_info gauge
596vllm:cache_config_info{_block_size_resolved="True",block_size="16",cache_dtype="auto",calculate_kv_scales="False",enable_prefix_caching="True",engine="0",gpu_memory_utilization="0.92",hash_block_size="None",is_attention_free="False",kv_cache_dtype_skip_layers="[]",kv_cache_memory_bytes="None",kv_offloading_backend="native",kv_offloading_size="None",kv_sharing_fast_prefill="False",mamba_block_size="None",mamba_cache_dtype="auto",mamba_cache_mode="none",mamba_page_size_padded="None",mamba_ssm_cache_dtype="auto",num_cpu_blocks="None",num_gpu_blocks="201897",num_gpu_blocks_override="None",prefix_caching_hash_algo="sha256",sliding_window="None",user_specified_block_size="False",user_specified_mamba_block_size="False"} 1.0

Change #1307897 had a related patch set uploaded (by Kevin Bazira; author: Kevin Bazira):

[machinelearning/liftwing/inference-services@main] cope-a-9b: Enable vLLM metrics

https://gerrit.wikimedia.org/r/1307897

Change #1308001 had a related patch set uploaded (by Kevin Bazira; author: Kevin Bazira):

[machinelearning/liftwing/inference-services@main] gpt-oss-safeguard-20b: Enable vLLM metrics

https://gerrit.wikimedia.org/r/1308001

Change #1308002 had a related patch set uploaded (by Kevin Bazira; author: Kevin Bazira):

[machinelearning/liftwing/inference-services@main] embeddings: Enable vLLM metrics

https://gerrit.wikimedia.org/r/1308002

Change #1308003 had a related patch set uploaded (by Kevin Bazira; author: Kevin Bazira):

[machinelearning/liftwing/inference-services@main] qwen36: Enable vLLM metrics

https://gerrit.wikimedia.org/r/1308003

Change #1307897 merged by jenkins-bot:

[machinelearning/liftwing/inference-services@main] cope-a-9b: Enable vLLM metrics

https://gerrit.wikimedia.org/r/1307897

Change #1308001 merged by jenkins-bot:

[machinelearning/liftwing/inference-services@main] gpt-oss-safeguard-20b: Enable vLLM metrics

https://gerrit.wikimedia.org/r/1308001

Change #1308002 merged by jenkins-bot:

[machinelearning/liftwing/inference-services@main] embeddings: Enable vLLM metrics

https://gerrit.wikimedia.org/r/1308002

Change #1308003 merged by jenkins-bot:

[machinelearning/liftwing/inference-services@main] qwen36: Enable vLLM metrics

https://gerrit.wikimedia.org/r/1308003

Change #1308049 had a related patch set uploaded (by Kevin Bazira; author: Kevin Bazira):

[operations/deployment-charts@master] ml-services: deploy cope-a-9b isvc image that emits vLLM metrics

https://gerrit.wikimedia.org/r/1308049

Change #1308050 had a related patch set uploaded (by Kevin Bazira; author: Kevin Bazira):

[operations/deployment-charts@master] ml-services: deploy gpt-oss-safeguard-20b isvc image that emits vLLM metrics

https://gerrit.wikimedia.org/r/1308050

Change #1308051 had a related patch set uploaded (by Kevin Bazira; author: Kevin Bazira):

[operations/deployment-charts@master] ml-services: deploy embeddings isvc image that emits vLLM metrics

https://gerrit.wikimedia.org/r/1308051

Change #1308052 had a related patch set uploaded (by Kevin Bazira; author: Kevin Bazira):

[operations/deployment-charts@master] ml-services: deploy qwen36 isvc image that emits vLLM metrics

https://gerrit.wikimedia.org/r/1308052

Change #1308049 merged by jenkins-bot:

[operations/deployment-charts@master] ml-services: deploy cope-a-9b isvc image that emits vLLM metrics

https://gerrit.wikimedia.org/r/1308049

The cope-a-9b isvc now emits vLLM metrics as shown below:

1$ curl "https://inference.svc.eqiad.wmnet:30443/metrics" -H "Host: cope-a-9b.experimental.wikimedia.org"
2# HELP python_gc_objects_collected_total Objects collected during gc
3# TYPE python_gc_objects_collected_total counter
4python_gc_objects_collected_total{generation="0"} 17222.0
5python_gc_objects_collected_total{generation="1"} 2967.0
6python_gc_objects_collected_total{generation="2"} 189.0
7# HELP python_gc_objects_uncollectable_total Uncollectable objects found during GC
8# TYPE python_gc_objects_uncollectable_total counter
9python_gc_objects_uncollectable_total{generation="0"} 0.0
10python_gc_objects_uncollectable_total{generation="1"} 0.0
11python_gc_objects_uncollectable_total{generation="2"} 0.0
12# HELP python_gc_collections_total Number of times this generation was collected
13# TYPE python_gc_collections_total counter
14python_gc_collections_total{generation="0"} 1891.0
15python_gc_collections_total{generation="1"} 171.0
16python_gc_collections_total{generation="2"} 10.0
17# HELP python_info Python platform information
18# TYPE python_info gauge
19python_info{implementation="CPython",major="3",minor="11",patchlevel="2",version="3.11.2"} 1.0
20# HELP process_virtual_memory_bytes Virtual memory size in bytes.
21# TYPE process_virtual_memory_bytes gauge
22process_virtual_memory_bytes 1.478488064e+010
23# HELP process_resident_memory_bytes Resident memory size in bytes.
24# TYPE process_resident_memory_bytes gauge
25process_resident_memory_bytes 1.892458496e+09
26# HELP process_start_time_seconds Start time of the process since unix epoch in seconds.
27# TYPE process_start_time_seconds gauge
28process_start_time_seconds 1.78341858234e+09
29# HELP process_cpu_seconds_total Total user and system CPU time spent in seconds.
30# TYPE process_cpu_seconds_total counter
31process_cpu_seconds_total 13.399999999999999
32# HELP process_open_fds Number of open file descriptors.
33# TYPE process_open_fds gauge
34process_open_fds 36.0
35# HELP process_max_fds Maximum number of open file descriptors.
36# TYPE process_max_fds gauge
37process_max_fds 1.073741816e+09
38# HELP request_preprocess_seconds pre-process request latency
39# TYPE request_preprocess_seconds histogram
40request_preprocess_seconds_bucket{le="0.005",model_name="cope-a-9b"} 2.0
41request_preprocess_seconds_bucket{le="0.01",model_name="cope-a-9b"} 2.0
42request_preprocess_seconds_bucket{le="0.025",model_name="cope-a-9b"} 2.0
43request_preprocess_seconds_bucket{le="0.05",model_name="cope-a-9b"} 2.0
44request_preprocess_seconds_bucket{le="0.075",model_name="cope-a-9b"} 2.0
45request_preprocess_seconds_bucket{le="0.1",model_name="cope-a-9b"} 2.0
46request_preprocess_seconds_bucket{le="0.25",model_name="cope-a-9b"} 2.0
47request_preprocess_seconds_bucket{le="0.5",model_name="cope-a-9b"} 2.0
48request_preprocess_seconds_bucket{le="0.75",model_name="cope-a-9b"} 2.0
49request_preprocess_seconds_bucket{le="1.0",model_name="cope-a-9b"} 2.0
50request_preprocess_seconds_bucket{le="2.5",model_name="cope-a-9b"} 2.0
51request_preprocess_seconds_bucket{le="5.0",model_name="cope-a-9b"} 2.0
52request_preprocess_seconds_bucket{le="7.5",model_name="cope-a-9b"} 2.0
53request_preprocess_seconds_bucket{le="10.0",model_name="cope-a-9b"} 2.0
54request_preprocess_seconds_bucket{le="+Inf",model_name="cope-a-9b"} 2.0
55request_preprocess_seconds_count{model_name="cope-a-9b"} 2.0
56request_preprocess_seconds_sum{model_name="cope-a-9b"} 7.705437019467354e-05
57# HELP request_preprocess_seconds_created pre-process request latency
58# TYPE request_preprocess_seconds_created gauge
59request_preprocess_seconds_created{model_name="cope-a-9b"} 1.7834187804908519e+09
60# HELP request_postprocess_seconds post-process request latency
61# TYPE request_postprocess_seconds histogram
62request_postprocess_seconds_bucket{le="0.005",model_name="cope-a-9b"} 2.0
63request_postprocess_seconds_bucket{le="0.01",model_name="cope-a-9b"} 2.0
64request_postprocess_seconds_bucket{le="0.025",model_name="cope-a-9b"} 2.0
65request_postprocess_seconds_bucket{le="0.05",model_name="cope-a-9b"} 2.0
66request_postprocess_seconds_bucket{le="0.075",model_name="cope-a-9b"} 2.0
67request_postprocess_seconds_bucket{le="0.1",model_name="cope-a-9b"} 2.0
68request_postprocess_seconds_bucket{le="0.25",model_name="cope-a-9b"} 2.0
69request_postprocess_seconds_bucket{le="0.5",model_name="cope-a-9b"} 2.0
70request_postprocess_seconds_bucket{le="0.75",model_name="cope-a-9b"} 2.0
71request_postprocess_seconds_bucket{le="1.0",model_name="cope-a-9b"} 2.0
72request_postprocess_seconds_bucket{le="2.5",model_name="cope-a-9b"} 2.0
73request_postprocess_seconds_bucket{le="5.0",model_name="cope-a-9b"} 2.0
74request_postprocess_seconds_bucket{le="7.5",model_name="cope-a-9b"} 2.0
75request_postprocess_seconds_bucket{le="10.0",model_name="cope-a-9b"} 2.0
76request_postprocess_seconds_bucket{le="+Inf",model_name="cope-a-9b"} 2.0
77request_postprocess_seconds_count{model_name="cope-a-9b"} 2.0
78request_postprocess_seconds_sum{model_name="cope-a-9b"} 2.2774096578359604e-05
79# HELP request_postprocess_seconds_created post-process request latency
80# TYPE request_postprocess_seconds_created gauge
81request_postprocess_seconds_created{model_name="cope-a-9b"} 1.7834187815370963e+09
82# HELP request_predict_seconds predict request latency
83# TYPE request_predict_seconds histogram
84request_predict_seconds_bucket{le="0.005",model_name="cope-a-9b"} 0.0
85request_predict_seconds_bucket{le="0.01",model_name="cope-a-9b"} 0.0
86request_predict_seconds_bucket{le="0.025",model_name="cope-a-9b"} 0.0
87request_predict_seconds_bucket{le="0.05",model_name="cope-a-9b"} 1.0
88request_predict_seconds_bucket{le="0.075",model_name="cope-a-9b"} 1.0
89request_predict_seconds_bucket{le="0.1",model_name="cope-a-9b"} 1.0
90request_predict_seconds_bucket{le="0.25",model_name="cope-a-9b"} 1.0
91request_predict_seconds_bucket{le="0.5",model_name="cope-a-9b"} 1.0
92request_predict_seconds_bucket{le="0.75",model_name="cope-a-9b"} 1.0
93request_predict_seconds_bucket{le="1.0",model_name="cope-a-9b"} 1.0
94request_predict_seconds_bucket{le="2.5",model_name="cope-a-9b"} 2.0
95request_predict_seconds_bucket{le="5.0",model_name="cope-a-9b"} 2.0
96request_predict_seconds_bucket{le="7.5",model_name="cope-a-9b"} 2.0
97request_predict_seconds_bucket{le="10.0",model_name="cope-a-9b"} 2.0
98request_predict_seconds_bucket{le="+Inf",model_name="cope-a-9b"} 2.0
99request_predict_seconds_count{model_name="cope-a-9b"} 2.0
100request_predict_seconds_sum{model_name="cope-a-9b"} 1.0732316481880844
101# HELP request_predict_seconds_created predict request latency
102# TYPE request_predict_seconds_created gauge
103request_predict_seconds_created{model_name="cope-a-9b"} 1.7834187804909766e+09
104# HELP request_explain_seconds explain request latency
105# TYPE request_explain_seconds histogram
106# HELP vllm:num_requests_running Number of requests in model execution batches.
107# TYPE vllm:num_requests_running gauge
108vllm:num_requests_running{engine="0",model_name="cope-a-9b"} 0.0
109# HELP vllm:num_requests_waiting Number of requests waiting to be processed.
110# TYPE vllm:num_requests_waiting gauge
111vllm:num_requests_waiting{engine="0",model_name="cope-a-9b"} 0.0
112# HELP vllm:engine_sleep_state Engine sleep state; awake = 0 means engine is sleeping; awake = 1 means engine is awake; weights_offloaded = 1 means sleep level 1; discard_all = 1 means sleep level 2.
113# TYPE vllm:engine_sleep_state gauge
114vllm:engine_sleep_state{engine="0",model_name="cope-a-9b",sleep_state="awake"} 1.0
115vllm:engine_sleep_state{engine="0",model_name="cope-a-9b",sleep_state="weights_offloaded"} 0.0
116vllm:engine_sleep_state{engine="0",model_name="cope-a-9b",sleep_state="discard_all"} 0.0
117# HELP vllm:kv_cache_usage_perc KV-cache usage. 1 means 100 percent usage.
118# TYPE vllm:kv_cache_usage_perc gauge
119vllm:kv_cache_usage_perc{engine="0",model_name="cope-a-9b"} 0.0
120# HELP vllm:prefix_cache_queries_total Prefix cache queries, in terms of number of queried tokens.
121# TYPE vllm:prefix_cache_queries_total counter
122vllm:prefix_cache_queries_total{engine="0",model_name="cope-a-9b"} 176.0
123# HELP vllm:prefix_cache_queries_created Prefix cache queries, in terms of number of queried tokens.
124# TYPE vllm:prefix_cache_queries_created gauge
125vllm:prefix_cache_queries_created{engine="0",model_name="cope-a-9b"} 1.7834186890806415e+09
126# HELP vllm:prefix_cache_hits_total Prefix cache hits, in terms of number of cached tokens.
127# TYPE vllm:prefix_cache_hits_total counter
128vllm:prefix_cache_hits_total{engine="0",model_name="cope-a-9b"} 80.0
129# HELP vllm:prefix_cache_hits_created Prefix cache hits, in terms of number of cached tokens.
130# TYPE vllm:prefix_cache_hits_created gauge
131vllm:prefix_cache_hits_created{engine="0",model_name="cope-a-9b"} 1.7834186890806594e+09
132# HELP vllm:external_prefix_cache_queries_total External prefix cache queries from KV connector cross-instance cache sharing, in terms of number of queried tokens.
133# TYPE vllm:external_prefix_cache_queries_total counter
134vllm:external_prefix_cache_queries_total{engine="0",model_name="cope-a-9b"} 0.0
135# HELP vllm:external_prefix_cache_queries_created External prefix cache queries from KV connector cross-instance cache sharing, in terms of number of queried tokens.
136# TYPE vllm:external_prefix_cache_queries_created gauge
137vllm:external_prefix_cache_queries_created{engine="0",model_name="cope-a-9b"} 1.783418689080672e+09
138# HELP vllm:external_prefix_cache_hits_total External prefix cache hits from KV connector cross-instance cache sharing, in terms of number of cached tokens.
139# TYPE vllm:external_prefix_cache_hits_total counter
140vllm:external_prefix_cache_hits_total{engine="0",model_name="cope-a-9b"} 0.0
141# HELP vllm:external_prefix_cache_hits_created External prefix cache hits from KV connector cross-instance cache sharing, in terms of number of cached tokens.
142# TYPE vllm:external_prefix_cache_hits_created gauge
143vllm:external_prefix_cache_hits_created{engine="0",model_name="cope-a-9b"} 1.783418689080687e+09
144# HELP vllm:mm_cache_queries_total Multi-modal cache queries, in terms of number of queried items.
145# TYPE vllm:mm_cache_queries_total counter
146vllm:mm_cache_queries_total{engine="0",model_name="cope-a-9b"} 0.0
147# HELP vllm:mm_cache_queries_created Multi-modal cache queries, in terms of number of queried items.
148# TYPE vllm:mm_cache_queries_created gauge
149vllm:mm_cache_queries_created{engine="0",model_name="cope-a-9b"} 1.783418689080699e+09
150# HELP vllm:mm_cache_hits_total Multi-modal cache hits, in terms of number of cached items.
151# TYPE vllm:mm_cache_hits_total counter
152vllm:mm_cache_hits_total{engine="0",model_name="cope-a-9b"} 0.0
153# HELP vllm:mm_cache_hits_created Multi-modal cache hits, in terms of number of cached items.
154# TYPE vllm:mm_cache_hits_created gauge
155vllm:mm_cache_hits_created{engine="0",model_name="cope-a-9b"} 1.7834186890807097e+09
156# HELP vllm:num_preemptions_total Cumulative number of preemption from the engine.
157# TYPE vllm:num_preemptions_total counter
158vllm:num_preemptions_total{engine="0",model_name="cope-a-9b"} 0.0
159# HELP vllm:num_preemptions_created Cumulative number of preemption from the engine.
160# TYPE vllm:num_preemptions_created gauge
161vllm:num_preemptions_created{engine="0",model_name="cope-a-9b"} 1.783418689080721e+09
162# HELP vllm:prompt_tokens_total Number of prefill tokens processed.
163# TYPE vllm:prompt_tokens_total counter
164vllm:prompt_tokens_total{engine="0",model_name="cope-a-9b"} 176.0
165# HELP vllm:prompt_tokens_created Number of prefill tokens processed.
166# TYPE vllm:prompt_tokens_created gauge
167vllm:prompt_tokens_created{engine="0",model_name="cope-a-9b"} 1.7834186890807347e+09
168# HELP vllm:generation_tokens_total Number of generation tokens processed.
169# TYPE vllm:generation_tokens_total counter
170vllm:generation_tokens_total{engine="0",model_name="cope-a-9b"} 2.0
171# HELP vllm:generation_tokens_created Number of generation tokens processed.
172# TYPE vllm:generation_tokens_created gauge
173vllm:generation_tokens_created{engine="0",model_name="cope-a-9b"} 1.783418689080749e+09
174# HELP vllm:request_success_total Count of successfully processed requests.
175# TYPE vllm:request_success_total counter
176vllm:request_success_total{engine="0",finished_reason="stop",model_name="cope-a-9b"} 0.0
177vllm:request_success_total{engine="0",finished_reason="length",model_name="cope-a-9b"} 2.0
178vllm:request_success_total{engine="0",finished_reason="abort",model_name="cope-a-9b"} 0.0
179vllm:request_success_total{engine="0",finished_reason="error",model_name="cope-a-9b"} 0.0
180# HELP vllm:request_success_created Count of successfully processed requests.
181# TYPE vllm:request_success_created gauge
182vllm:request_success_created{engine="0",finished_reason="stop",model_name="cope-a-9b"} 1.7834186890807736e+09
183vllm:request_success_created{engine="0",finished_reason="length",model_name="cope-a-9b"} 1.7834186890807807e+09
184vllm:request_success_created{engine="0",finished_reason="abort",model_name="cope-a-9b"} 1.7834186890807893e+09
185vllm:request_success_created{engine="0",finished_reason="error",model_name="cope-a-9b"} 1.7834186890807948e+09
186# HELP vllm:request_prompt_tokens Number of prefill tokens processed.
187# TYPE vllm:request_prompt_tokens histogram
188vllm:request_prompt_tokens_bucket{engine="0",le="1.0",model_name="cope-a-9b"} 0.0
189vllm:request_prompt_tokens_bucket{engine="0",le="2.0",model_name="cope-a-9b"} 0.0
190vllm:request_prompt_tokens_bucket{engine="0",le="5.0",model_name="cope-a-9b"} 0.0
191vllm:request_prompt_tokens_bucket{engine="0",le="10.0",model_name="cope-a-9b"} 0.0
192vllm:request_prompt_tokens_bucket{engine="0",le="20.0",model_name="cope-a-9b"} 0.0
193vllm:request_prompt_tokens_bucket{engine="0",le="50.0",model_name="cope-a-9b"} 0.0
194vllm:request_prompt_tokens_bucket{engine="0",le="100.0",model_name="cope-a-9b"} 2.0
195vllm:request_prompt_tokens_bucket{engine="0",le="200.0",model_name="cope-a-9b"} 2.0
196vllm:request_prompt_tokens_bucket{engine="0",le="500.0",model_name="cope-a-9b"} 2.0
197vllm:request_prompt_tokens_bucket{engine="0",le="1000.0",model_name="cope-a-9b"} 2.0
198vllm:request_prompt_tokens_bucket{engine="0",le="2000.0",model_name="cope-a-9b"} 2.0
199vllm:request_prompt_tokens_bucket{engine="0",le="+Inf",model_name="cope-a-9b"} 2.0
200vllm:request_prompt_tokens_count{engine="0",model_name="cope-a-9b"} 2.0
201vllm:request_prompt_tokens_sum{engine="0",model_name="cope-a-9b"} 176.0
202# HELP vllm:request_prompt_tokens_created Number of prefill tokens processed.
203# TYPE vllm:request_prompt_tokens_created gauge
204vllm:request_prompt_tokens_created{engine="0",model_name="cope-a-9b"} 1.7834186890808346e+09
205# HELP vllm:request_generation_tokens Number of generation tokens processed.
206# TYPE vllm:request_generation_tokens histogram
207vllm:request_generation_tokens_bucket{engine="0",le="1.0",model_name="cope-a-9b"} 2.0
208vllm:request_generation_tokens_bucket{engine="0",le="2.0",model_name="cope-a-9b"} 2.0
209vllm:request_generation_tokens_bucket{engine="0",le="5.0",model_name="cope-a-9b"} 2.0
210vllm:request_generation_tokens_bucket{engine="0",le="10.0",model_name="cope-a-9b"} 2.0
211vllm:request_generation_tokens_bucket{engine="0",le="20.0",model_name="cope-a-9b"} 2.0
212vllm:request_generation_tokens_bucket{engine="0",le="50.0",model_name="cope-a-9b"} 2.0
213vllm:request_generation_tokens_bucket{engine="0",le="100.0",model_name="cope-a-9b"} 2.0
214vllm:request_generation_tokens_bucket{engine="0",le="200.0",model_name="cope-a-9b"} 2.0
215vllm:request_generation_tokens_bucket{engine="0",le="500.0",model_name="cope-a-9b"} 2.0
216vllm:request_generation_tokens_bucket{engine="0",le="1000.0",model_name="cope-a-9b"} 2.0
217vllm:request_generation_tokens_bucket{engine="0",le="2000.0",model_name="cope-a-9b"} 2.0
218vllm:request_generation_tokens_bucket{engine="0",le="+Inf",model_name="cope-a-9b"} 2.0
219vllm:request_generation_tokens_count{engine="0",model_name="cope-a-9b"} 2.0
220vllm:request_generation_tokens_sum{engine="0",model_name="cope-a-9b"} 2.0
221# HELP vllm:request_generation_tokens_created Number of generation tokens processed.
222# TYPE vllm:request_generation_tokens_created gauge
223vllm:request_generation_tokens_created{engine="0",model_name="cope-a-9b"} 1.783418689080885e+09
224# HELP vllm:iteration_tokens_total Histogram of number of tokens per engine_step.
225# TYPE vllm:iteration_tokens_total histogram
226vllm:iteration_tokens_total_bucket{engine="0",le="1.0",model_name="cope-a-9b"} 2.0
227vllm:iteration_tokens_total_bucket{engine="0",le="8.0",model_name="cope-a-9b"} 2.0
228vllm:iteration_tokens_total_bucket{engine="0",le="16.0",model_name="cope-a-9b"} 2.0
229vllm:iteration_tokens_total_bucket{engine="0",le="32.0",model_name="cope-a-9b"} 2.0
230vllm:iteration_tokens_total_bucket{engine="0",le="64.0",model_name="cope-a-9b"} 2.0
231vllm:iteration_tokens_total_bucket{engine="0",le="128.0",model_name="cope-a-9b"} 4.0
232vllm:iteration_tokens_total_bucket{engine="0",le="256.0",model_name="cope-a-9b"} 4.0
233vllm:iteration_tokens_total_bucket{engine="0",le="512.0",model_name="cope-a-9b"} 4.0
234vllm:iteration_tokens_total_bucket{engine="0",le="1024.0",model_name="cope-a-9b"} 4.0
235vllm:iteration_tokens_total_bucket{engine="0",le="2048.0",model_name="cope-a-9b"} 4.0
236vllm:iteration_tokens_total_bucket{engine="0",le="4096.0",model_name="cope-a-9b"} 4.0
237vllm:iteration_tokens_total_bucket{engine="0",le="8192.0",model_name="cope-a-9b"} 4.0
238vllm:iteration_tokens_total_bucket{engine="0",le="16384.0",model_name="cope-a-9b"} 4.0
239vllm:iteration_tokens_total_bucket{engine="0",le="+Inf",model_name="cope-a-9b"} 4.0
240vllm:iteration_tokens_total_count{engine="0",model_name="cope-a-9b"} 4.0
241vllm:iteration_tokens_total_sum{engine="0",model_name="cope-a-9b"} 178.0
242# HELP vllm:iteration_tokens_total_created Histogram of number of tokens per engine_step.
243# TYPE vllm:iteration_tokens_total_created gauge
244vllm:iteration_tokens_total_created{engine="0",model_name="cope-a-9b"} 1.7834186890809145e+09
245# HELP vllm:request_max_num_generation_tokens Histogram of maximum number of requested generation tokens.
246# TYPE vllm:request_max_num_generation_tokens histogram
247vllm:request_max_num_generation_tokens_bucket{engine="0",le="1.0",model_name="cope-a-9b"} 2.0
248vllm:request_max_num_generation_tokens_bucket{engine="0",le="2.0",model_name="cope-a-9b"} 2.0
249vllm:request_max_num_generation_tokens_bucket{engine="0",le="5.0",model_name="cope-a-9b"} 2.0
250vllm:request_max_num_generation_tokens_bucket{engine="0",le="10.0",model_name="cope-a-9b"} 2.0
251vllm:request_max_num_generation_tokens_bucket{engine="0",le="20.0",model_name="cope-a-9b"} 2.0
252vllm:request_max_num_generation_tokens_bucket{engine="0",le="50.0",model_name="cope-a-9b"} 2.0
253vllm:request_max_num_generation_tokens_bucket{engine="0",le="100.0",model_name="cope-a-9b"} 2.0
254vllm:request_max_num_generation_tokens_bucket{engine="0",le="200.0",model_name="cope-a-9b"} 2.0
255vllm:request_max_num_generation_tokens_bucket{engine="0",le="500.0",model_name="cope-a-9b"} 2.0
256vllm:request_max_num_generation_tokens_bucket{engine="0",le="1000.0",model_name="cope-a-9b"} 2.0
257vllm:request_max_num_generation_tokens_bucket{engine="0",le="2000.0",model_name="cope-a-9b"} 2.0
258vllm:request_max_num_generation_tokens_bucket{engine="0",le="+Inf",model_name="cope-a-9b"} 2.0
259vllm:request_max_num_generation_tokens_count{engine="0",model_name="cope-a-9b"} 2.0
260vllm:request_max_num_generation_tokens_sum{engine="0",model_name="cope-a-9b"} 2.0
261# HELP vllm:request_max_num_generation_tokens_created Histogram of maximum number of requested generation tokens.
262# TYPE vllm:request_max_num_generation_tokens_created gauge
263vllm:request_max_num_generation_tokens_created{engine="0",model_name="cope-a-9b"} 1.7834186890809476e+09
264# HELP vllm:request_params_n Histogram of the n request parameter.
265# TYPE vllm:request_params_n histogram
266vllm:request_params_n_bucket{engine="0",le="1.0",model_name="cope-a-9b"} 2.0
267vllm:request_params_n_bucket{engine="0",le="2.0",model_name="cope-a-9b"} 2.0
268vllm:request_params_n_bucket{engine="0",le="5.0",model_name="cope-a-9b"} 2.0
269vllm:request_params_n_bucket{engine="0",le="10.0",model_name="cope-a-9b"} 2.0
270vllm:request_params_n_bucket{engine="0",le="20.0",model_name="cope-a-9b"} 2.0
271vllm:request_params_n_bucket{engine="0",le="+Inf",model_name="cope-a-9b"} 2.0
272vllm:request_params_n_count{engine="0",model_name="cope-a-9b"} 2.0
273vllm:request_params_n_sum{engine="0",model_name="cope-a-9b"} 2.0
274# HELP vllm:request_params_n_created Histogram of the n request parameter.
275# TYPE vllm:request_params_n_created gauge
276vllm:request_params_n_created{engine="0",model_name="cope-a-9b"} 1.783418689081072e+09
277# HELP vllm:request_params_max_tokens Histogram of the max_tokens request parameter.
278# TYPE vllm:request_params_max_tokens histogram
279vllm:request_params_max_tokens_bucket{engine="0",le="1.0",model_name="cope-a-9b"} 2.0
280vllm:request_params_max_tokens_bucket{engine="0",le="2.0",model_name="cope-a-9b"} 2.0
281vllm:request_params_max_tokens_bucket{engine="0",le="5.0",model_name="cope-a-9b"} 2.0
282vllm:request_params_max_tokens_bucket{engine="0",le="10.0",model_name="cope-a-9b"} 2.0
283vllm:request_params_max_tokens_bucket{engine="0",le="20.0",model_name="cope-a-9b"} 2.0
284vllm:request_params_max_tokens_bucket{engine="0",le="50.0",model_name="cope-a-9b"} 2.0
285vllm:request_params_max_tokens_bucket{engine="0",le="100.0",model_name="cope-a-9b"} 2.0
286vllm:request_params_max_tokens_bucket{engine="0",le="200.0",model_name="cope-a-9b"} 2.0
287vllm:request_params_max_tokens_bucket{engine="0",le="500.0",model_name="cope-a-9b"} 2.0
288vllm:request_params_max_tokens_bucket{engine="0",le="1000.0",model_name="cope-a-9b"} 2.0
289vllm:request_params_max_tokens_bucket{engine="0",le="2000.0",model_name="cope-a-9b"} 2.0
290vllm:request_params_max_tokens_bucket{engine="0",le="+Inf",model_name="cope-a-9b"} 2.0
291vllm:request_params_max_tokens_count{engine="0",model_name="cope-a-9b"} 2.0
292vllm:request_params_max_tokens_sum{engine="0",model_name="cope-a-9b"} 2.0
293# HELP vllm:request_params_max_tokens_created Histogram of the max_tokens request parameter.
294# TYPE vllm:request_params_max_tokens_created gauge
295vllm:request_params_max_tokens_created{engine="0",model_name="cope-a-9b"} 1.7834186890810933e+09
296# HELP vllm:time_to_first_token_seconds Histogram of time to first token in seconds.
297# TYPE vllm:time_to_first_token_seconds histogram
298vllm:time_to_first_token_seconds_bucket{engine="0",le="0.001",model_name="cope-a-9b"} 0.0
299vllm:time_to_first_token_seconds_bucket{engine="0",le="0.005",model_name="cope-a-9b"} 0.0
300vllm:time_to_first_token_seconds_bucket{engine="0",le="0.01",model_name="cope-a-9b"} 0.0
301vllm:time_to_first_token_seconds_bucket{engine="0",le="0.02",model_name="cope-a-9b"} 0.0
302vllm:time_to_first_token_seconds_bucket{engine="0",le="0.04",model_name="cope-a-9b"} 1.0
303vllm:time_to_first_token_seconds_bucket{engine="0",le="0.06",model_name="cope-a-9b"} 1.0
304vllm:time_to_first_token_seconds_bucket{engine="0",le="0.08",model_name="cope-a-9b"} 1.0
305vllm:time_to_first_token_seconds_bucket{engine="0",le="0.1",model_name="cope-a-9b"} 1.0
306vllm:time_to_first_token_seconds_bucket{engine="0",le="0.25",model_name="cope-a-9b"} 1.0
307vllm:time_to_first_token_seconds_bucket{engine="0",le="0.5",model_name="cope-a-9b"} 1.0
308vllm:time_to_first_token_seconds_bucket{engine="0",le="0.75",model_name="cope-a-9b"} 1.0
309vllm:time_to_first_token_seconds_bucket{engine="0",le="1.0",model_name="cope-a-9b"} 1.0
310vllm:time_to_first_token_seconds_bucket{engine="0",le="2.5",model_name="cope-a-9b"} 2.0
311vllm:time_to_first_token_seconds_bucket{engine="0",le="5.0",model_name="cope-a-9b"} 2.0
312vllm:time_to_first_token_seconds_bucket{engine="0",le="7.5",model_name="cope-a-9b"} 2.0
313vllm:time_to_first_token_seconds_bucket{engine="0",le="10.0",model_name="cope-a-9b"} 2.0
314vllm:time_to_first_token_seconds_bucket{engine="0",le="20.0",model_name="cope-a-9b"} 2.0
315vllm:time_to_first_token_seconds_bucket{engine="0",le="40.0",model_name="cope-a-9b"} 2.0
316vllm:time_to_first_token_seconds_bucket{engine="0",le="80.0",model_name="cope-a-9b"} 2.0
317vllm:time_to_first_token_seconds_bucket{engine="0",le="160.0",model_name="cope-a-9b"} 2.0
318vllm:time_to_first_token_seconds_bucket{engine="0",le="640.0",model_name="cope-a-9b"} 2.0
319vllm:time_to_first_token_seconds_bucket{engine="0",le="2560.0",model_name="cope-a-9b"} 2.0
320vllm:time_to_first_token_seconds_bucket{engine="0",le="+Inf",model_name="cope-a-9b"} 2.0
321vllm:time_to_first_token_seconds_count{engine="0",model_name="cope-a-9b"} 2.0
322vllm:time_to_first_token_seconds_sum{engine="0",model_name="cope-a-9b"} 1.0706405639648438
323# HELP vllm:time_to_first_token_seconds_created Histogram of time to first token in seconds.
324# TYPE vllm:time_to_first_token_seconds_created gauge
325vllm:time_to_first_token_seconds_created{engine="0",model_name="cope-a-9b"} 1.7834186890811222e+09
326# HELP vllm:inter_token_latency_seconds Histogram of inter-token latency in seconds.
327# TYPE vllm:inter_token_latency_seconds histogram
328vllm:inter_token_latency_seconds_bucket{engine="0",le="0.01",model_name="cope-a-9b"} 0.0
329vllm:inter_token_latency_seconds_bucket{engine="0",le="0.025",model_name="cope-a-9b"} 0.0
330vllm:inter_token_latency_seconds_bucket{engine="0",le="0.05",model_name="cope-a-9b"} 0.0
331vllm:inter_token_latency_seconds_bucket{engine="0",le="0.075",model_name="cope-a-9b"} 0.0
332vllm:inter_token_latency_seconds_bucket{engine="0",le="0.1",model_name="cope-a-9b"} 0.0
333vllm:inter_token_latency_seconds_bucket{engine="0",le="0.15",model_name="cope-a-9b"} 0.0
334vllm:inter_token_latency_seconds_bucket{engine="0",le="0.2",model_name="cope-a-9b"} 0.0
335vllm:inter_token_latency_seconds_bucket{engine="0",le="0.3",model_name="cope-a-9b"} 0.0
336vllm:inter_token_latency_seconds_bucket{engine="0",le="0.4",model_name="cope-a-9b"} 0.0
337vllm:inter_token_latency_seconds_bucket{engine="0",le="0.5",model_name="cope-a-9b"} 0.0
338vllm:inter_token_latency_seconds_bucket{engine="0",le="0.75",model_name="cope-a-9b"} 0.0
339vllm:inter_token_latency_seconds_bucket{engine="0",le="1.0",model_name="cope-a-9b"} 0.0
340vllm:inter_token_latency_seconds_bucket{engine="0",le="2.5",model_name="cope-a-9b"} 0.0
341vllm:inter_token_latency_seconds_bucket{engine="0",le="5.0",model_name="cope-a-9b"} 0.0
342vllm:inter_token_latency_seconds_bucket{engine="0",le="7.5",model_name="cope-a-9b"} 0.0
343vllm:inter_token_latency_seconds_bucket{engine="0",le="10.0",model_name="cope-a-9b"} 0.0
344vllm:inter_token_latency_seconds_bucket{engine="0",le="20.0",model_name="cope-a-9b"} 0.0
345vllm:inter_token_latency_seconds_bucket{engine="0",le="40.0",model_name="cope-a-9b"} 0.0
346vllm:inter_token_latency_seconds_bucket{engine="0",le="80.0",model_name="cope-a-9b"} 0.0
347vllm:inter_token_latency_seconds_bucket{engine="0",le="+Inf",model_name="cope-a-9b"} 0.0
348vllm:inter_token_latency_seconds_count{engine="0",model_name="cope-a-9b"} 0.0
349vllm:inter_token_latency_seconds_sum{engine="0",model_name="cope-a-9b"} 0.0
350# HELP vllm:inter_token_latency_seconds_created Histogram of inter-token latency in seconds.
351# TYPE vllm:inter_token_latency_seconds_created gauge
352vllm:inter_token_latency_seconds_created{engine="0",model_name="cope-a-9b"} 1.7834186890811582e+09
353# HELP vllm:request_time_per_output_token_seconds Histogram of time_per_output_token_seconds per request.
354# TYPE vllm:request_time_per_output_token_seconds histogram
355vllm:request_time_per_output_token_seconds_bucket{engine="0",le="0.01",model_name="cope-a-9b"} 2.0
356vllm:request_time_per_output_token_seconds_bucket{engine="0",le="0.025",model_name="cope-a-9b"} 2.0
357vllm:request_time_per_output_token_seconds_bucket{engine="0",le="0.05",model_name="cope-a-9b"} 2.0
358vllm:request_time_per_output_token_seconds_bucket{engine="0",le="0.075",model_name="cope-a-9b"} 2.0
359vllm:request_time_per_output_token_seconds_bucket{engine="0",le="0.1",model_name="cope-a-9b"} 2.0
360vllm:request_time_per_output_token_seconds_bucket{engine="0",le="0.15",model_name="cope-a-9b"} 2.0
361vllm:request_time_per_output_token_seconds_bucket{engine="0",le="0.2",model_name="cope-a-9b"} 2.0
362vllm:request_time_per_output_token_seconds_bucket{engine="0",le="0.3",model_name="cope-a-9b"} 2.0
363vllm:request_time_per_output_token_seconds_bucket{engine="0",le="0.4",model_name="cope-a-9b"} 2.0
364vllm:request_time_per_output_token_seconds_bucket{engine="0",le="0.5",model_name="cope-a-9b"} 2.0
365vllm:request_time_per_output_token_seconds_bucket{engine="0",le="0.75",model_name="cope-a-9b"} 2.0
366vllm:request_time_per_output_token_seconds_bucket{engine="0",le="1.0",model_name="cope-a-9b"} 2.0
367vllm:request_time_per_output_token_seconds_bucket{engine="0",le="2.5",model_name="cope-a-9b"} 2.0
368vllm:request_time_per_output_token_seconds_bucket{engine="0",le="5.0",model_name="cope-a-9b"} 2.0
369vllm:request_time_per_output_token_seconds_bucket{engine="0",le="7.5",model_name="cope-a-9b"} 2.0
370vllm:request_time_per_output_token_seconds_bucket{engine="0",le="10.0",model_name="cope-a-9b"} 2.0
371vllm:request_time_per_output_token_seconds_bucket{engine="0",le="20.0",model_name="cope-a-9b"} 2.0
372vllm:request_time_per_output_token_seconds_bucket{engine="0",le="40.0",model_name="cope-a-9b"} 2.0
373vllm:request_time_per_output_token_seconds_bucket{engine="0",le="80.0",model_name="cope-a-9b"} 2.0
374vllm:request_time_per_output_token_seconds_bucket{engine="0",le="+Inf",model_name="cope-a-9b"} 2.0
375vllm:request_time_per_output_token_seconds_count{engine="0",model_name="cope-a-9b"} 2.0
376vllm:request_time_per_output_token_seconds_sum{engine="0",model_name="cope-a-9b"} 0.0
377# HELP vllm:request_time_per_output_token_seconds_created Histogram of time_per_output_token_seconds per request.
378# TYPE vllm:request_time_per_output_token_seconds_created gauge
379vllm:request_time_per_output_token_seconds_created{engine="0",model_name="cope-a-9b"} 1.7834186890811908e+09
380# HELP vllm:e2e_request_latency_seconds Histogram of e2e request latency in seconds.
381# TYPE vllm:e2e_request_latency_seconds histogram
382vllm:e2e_request_latency_seconds_bucket{engine="0",le="0.3",model_name="cope-a-9b"} 1.0
383vllm:e2e_request_latency_seconds_bucket{engine="0",le="0.5",model_name="cope-a-9b"} 1.0
384vllm:e2e_request_latency_seconds_bucket{engine="0",le="0.8",model_name="cope-a-9b"} 1.0
385vllm:e2e_request_latency_seconds_bucket{engine="0",le="1.0",model_name="cope-a-9b"} 1.0
386vllm:e2e_request_latency_seconds_bucket{engine="0",le="1.5",model_name="cope-a-9b"} 2.0
387vllm:e2e_request_latency_seconds_bucket{engine="0",le="2.0",model_name="cope-a-9b"} 2.0
388vllm:e2e_request_latency_seconds_bucket{engine="0",le="2.5",model_name="cope-a-9b"} 2.0
389vllm:e2e_request_latency_seconds_bucket{engine="0",le="5.0",model_name="cope-a-9b"} 2.0
390vllm:e2e_request_latency_seconds_bucket{engine="0",le="10.0",model_name="cope-a-9b"} 2.0
391vllm:e2e_request_latency_seconds_bucket{engine="0",le="15.0",model_name="cope-a-9b"} 2.0
392vllm:e2e_request_latency_seconds_bucket{engine="0",le="20.0",model_name="cope-a-9b"} 2.0
393vllm:e2e_request_latency_seconds_bucket{engine="0",le="30.0",model_name="cope-a-9b"} 2.0
394vllm:e2e_request_latency_seconds_bucket{engine="0",le="40.0",model_name="cope-a-9b"} 2.0
395vllm:e2e_request_latency_seconds_bucket{engine="0",le="50.0",model_name="cope-a-9b"} 2.0
396vllm:e2e_request_latency_seconds_bucket{engine="0",le="60.0",model_name="cope-a-9b"} 2.0
397vllm:e2e_request_latency_seconds_bucket{engine="0",le="120.0",model_name="cope-a-9b"} 2.0
398vllm:e2e_request_latency_seconds_bucket{engine="0",le="240.0",model_name="cope-a-9b"} 2.0
399vllm:e2e_request_latency_seconds_bucket{engine="0",le="480.0",model_name="cope-a-9b"} 2.0
400vllm:e2e_request_latency_seconds_bucket{engine="0",le="960.0",model_name="cope-a-9b"} 2.0
401vllm:e2e_request_latency_seconds_bucket{engine="0",le="1920.0",model_name="cope-a-9b"} 2.0
402vllm:e2e_request_latency_seconds_bucket{engine="0",le="7680.0",model_name="cope-a-9b"} 2.0
403vllm:e2e_request_latency_seconds_bucket{engine="0",le="+Inf",model_name="cope-a-9b"} 2.0
404vllm:e2e_request_latency_seconds_count{engine="0",model_name="cope-a-9b"} 2.0
405vllm:e2e_request_latency_seconds_sum{engine="0",model_name="cope-a-9b"} 1.0706405639648438
406# HELP vllm:e2e_request_latency_seconds_created Histogram of e2e request latency in seconds.
407# TYPE vllm:e2e_request_latency_seconds_created gauge
408vllm:e2e_request_latency_seconds_created{engine="0",model_name="cope-a-9b"} 1.7834186890812263e+09
409# HELP vllm:request_queue_time_seconds Histogram of time spent in WAITING phase for request.
410# TYPE vllm:request_queue_time_seconds histogram
411vllm:request_queue_time_seconds_bucket{engine="0",le="0.3",model_name="cope-a-9b"} 2.0
412vllm:request_queue_time_seconds_bucket{engine="0",le="0.5",model_name="cope-a-9b"} 2.0
413vllm:request_queue_time_seconds_bucket{engine="0",le="0.8",model_name="cope-a-9b"} 2.0
414vllm:request_queue_time_seconds_bucket{engine="0",le="1.0",model_name="cope-a-9b"} 2.0
415vllm:request_queue_time_seconds_bucket{engine="0",le="1.5",model_name="cope-a-9b"} 2.0
416vllm:request_queue_time_seconds_bucket{engine="0",le="2.0",model_name="cope-a-9b"} 2.0
417vllm:request_queue_time_seconds_bucket{engine="0",le="2.5",model_name="cope-a-9b"} 2.0
418vllm:request_queue_time_seconds_bucket{engine="0",le="5.0",model_name="cope-a-9b"} 2.0
419vllm:request_queue_time_seconds_bucket{engine="0",le="10.0",model_name="cope-a-9b"} 2.0
420vllm:request_queue_time_seconds_bucket{engine="0",le="15.0",model_name="cope-a-9b"} 2.0
421vllm:request_queue_time_seconds_bucket{engine="0",le="20.0",model_name="cope-a-9b"} 2.0
422vllm:request_queue_time_seconds_bucket{engine="0",le="30.0",model_name="cope-a-9b"} 2.0
423vllm:request_queue_time_seconds_bucket{engine="0",le="40.0",model_name="cope-a-9b"} 2.0
424vllm:request_queue_time_seconds_bucket{engine="0",le="50.0",model_name="cope-a-9b"} 2.0
425vllm:request_queue_time_seconds_bucket{engine="0",le="60.0",model_name="cope-a-9b"} 2.0
426vllm:request_queue_time_seconds_bucket{engine="0",le="120.0",model_name="cope-a-9b"} 2.0
427vllm:request_queue_time_seconds_bucket{engine="0",le="240.0",model_name="cope-a-9b"} 2.0
428vllm:request_queue_time_seconds_bucket{engine="0",le="480.0",model_name="cope-a-9b"} 2.0
429vllm:request_queue_time_seconds_bucket{engine="0",le="960.0",model_name="cope-a-9b"} 2.0
430vllm:request_queue_time_seconds_bucket{engine="0",le="1920.0",model_name="cope-a-9b"} 2.0
431vllm:request_queue_time_seconds_bucket{engine="0",le="7680.0",model_name="cope-a-9b"} 2.0
432vllm:request_queue_time_seconds_bucket{engine="0",le="+Inf",model_name="cope-a-9b"} 2.0
433vllm:request_queue_time_seconds_count{engine="0",model_name="cope-a-9b"} 2.0
434vllm:request_queue_time_seconds_sum{engine="0",model_name="cope-a-9b"} 4.561804234981537e-05
435# HELP vllm:request_queue_time_seconds_created Histogram of time spent in WAITING phase for request.
436# TYPE vllm:request_queue_time_seconds_created gauge
437vllm:request_queue_time_seconds_created{engine="0",model_name="cope-a-9b"} 1.7834186890812624e+09
438# HELP vllm:request_inference_time_seconds Histogram of time spent in RUNNING phase for request.
439# TYPE vllm:request_inference_time_seconds histogram
440vllm:request_inference_time_seconds_bucket{engine="0",le="0.3",model_name="cope-a-9b"} 1.0
441vllm:request_inference_time_seconds_bucket{engine="0",le="0.5",model_name="cope-a-9b"} 1.0
442vllm:request_inference_time_seconds_bucket{engine="0",le="0.8",model_name="cope-a-9b"} 1.0
443vllm:request_inference_time_seconds_bucket{engine="0",le="1.0",model_name="cope-a-9b"} 1.0
444vllm:request_inference_time_seconds_bucket{engine="0",le="1.5",model_name="cope-a-9b"} 2.0
445vllm:request_inference_time_seconds_bucket{engine="0",le="2.0",model_name="cope-a-9b"} 2.0
446vllm:request_inference_time_seconds_bucket{engine="0",le="2.5",model_name="cope-a-9b"} 2.0
447vllm:request_inference_time_seconds_bucket{engine="0",le="5.0",model_name="cope-a-9b"} 2.0
448vllm:request_inference_time_seconds_bucket{engine="0",le="10.0",model_name="cope-a-9b"} 2.0
449vllm:request_inference_time_seconds_bucket{engine="0",le="15.0",model_name="cope-a-9b"} 2.0
450vllm:request_inference_time_seconds_bucket{engine="0",le="20.0",model_name="cope-a-9b"} 2.0
451vllm:request_inference_time_seconds_bucket{engine="0",le="30.0",model_name="cope-a-9b"} 2.0
452vllm:request_inference_time_seconds_bucket{engine="0",le="40.0",model_name="cope-a-9b"} 2.0
453vllm:request_inference_time_seconds_bucket{engine="0",le="50.0",model_name="cope-a-9b"} 2.0
454vllm:request_inference_time_seconds_bucket{engine="0",le="60.0",model_name="cope-a-9b"} 2.0
455vllm:request_inference_time_seconds_bucket{engine="0",le="120.0",model_name="cope-a-9b"} 2.0
456vllm:request_inference_time_seconds_bucket{engine="0",le="240.0",model_name="cope-a-9b"} 2.0
457vllm:request_inference_time_seconds_bucket{engine="0",le="480.0",model_name="cope-a-9b"} 2.0
458vllm:request_inference_time_seconds_bucket{engine="0",le="960.0",model_name="cope-a-9b"} 2.0
459vllm:request_inference_time_seconds_bucket{engine="0",le="1920.0",model_name="cope-a-9b"} 2.0
460vllm:request_inference_time_seconds_bucket{engine="0",le="7680.0",model_name="cope-a-9b"} 2.0
461vllm:request_inference_time_seconds_bucket{engine="0",le="+Inf",model_name="cope-a-9b"} 2.0
462vllm:request_inference_time_seconds_count{engine="0",model_name="cope-a-9b"} 2.0
463vllm:request_inference_time_seconds_sum{engine="0",model_name="cope-a-9b"} 1.0646274825558066
464# HELP vllm:request_inference_time_seconds_created Histogram of time spent in RUNNING phase for request.
465# TYPE vllm:request_inference_time_seconds_created gauge
466vllm:request_inference_time_seconds_created{engine="0",model_name="cope-a-9b"} 1.7834186890813036e+09
467# HELP vllm:request_prefill_time_seconds Histogram of time spent in PREFILL phase for request.
468# TYPE vllm:request_prefill_time_seconds histogram
469vllm:request_prefill_time_seconds_bucket{engine="0",le="0.3",model_name="cope-a-9b"} 1.0
470vllm:request_prefill_time_seconds_bucket{engine="0",le="0.5",model_name="cope-a-9b"} 1.0
471vllm:request_prefill_time_seconds_bucket{engine="0",le="0.8",model_name="cope-a-9b"} 1.0
472vllm:request_prefill_time_seconds_bucket{engine="0",le="1.0",model_name="cope-a-9b"} 1.0
473vllm:request_prefill_time_seconds_bucket{engine="0",le="1.5",model_name="cope-a-9b"} 2.0
474vllm:request_prefill_time_seconds_bucket{engine="0",le="2.0",model_name="cope-a-9b"} 2.0
475vllm:request_prefill_time_seconds_bucket{engine="0",le="2.5",model_name="cope-a-9b"} 2.0
476vllm:request_prefill_time_seconds_bucket{engine="0",le="5.0",model_name="cope-a-9b"} 2.0
477vllm:request_prefill_time_seconds_bucket{engine="0",le="10.0",model_name="cope-a-9b"} 2.0
478vllm:request_prefill_time_seconds_bucket{engine="0",le="15.0",model_name="cope-a-9b"} 2.0
479vllm:request_prefill_time_seconds_bucket{engine="0",le="20.0",model_name="cope-a-9b"} 2.0
480vllm:request_prefill_time_seconds_bucket{engine="0",le="30.0",model_name="cope-a-9b"} 2.0
481vllm:request_prefill_time_seconds_bucket{engine="0",le="40.0",model_name="cope-a-9b"} 2.0
482vllm:request_prefill_time_seconds_bucket{engine="0",le="50.0",model_name="cope-a-9b"} 2.0
483vllm:request_prefill_time_seconds_bucket{engine="0",le="60.0",model_name="cope-a-9b"} 2.0
484vllm:request_prefill_time_seconds_bucket{engine="0",le="120.0",model_name="cope-a-9b"} 2.0
485vllm:request_prefill_time_seconds_bucket{engine="0",le="240.0",model_name="cope-a-9b"} 2.0
486vllm:request_prefill_time_seconds_bucket{engine="0",le="480.0",model_name="cope-a-9b"} 2.0
487vllm:request_prefill_time_seconds_bucket{engine="0",le="960.0",model_name="cope-a-9b"} 2.0
488vllm:request_prefill_time_seconds_bucket{engine="0",le="1920.0",model_name="cope-a-9b"} 2.0
489vllm:request_prefill_time_seconds_bucket{engine="0",le="7680.0",model_name="cope-a-9b"} 2.0
490vllm:request_prefill_time_seconds_bucket{engine="0",le="+Inf",model_name="cope-a-9b"} 2.0
491vllm:request_prefill_time_seconds_count{engine="0",model_name="cope-a-9b"} 2.0
492vllm:request_prefill_time_seconds_sum{engine="0",model_name="cope-a-9b"} 1.0646274825558066
493# HELP vllm:request_prefill_time_seconds_created Histogram of time spent in PREFILL phase for request.
494# TYPE vllm:request_prefill_time_seconds_created gauge
495vllm:request_prefill_time_seconds_created{engine="0",model_name="cope-a-9b"} 1.7834186890813413e+09
496# HELP vllm:request_decode_time_seconds Histogram of time spent in DECODE phase for request.
497# TYPE vllm:request_decode_time_seconds histogram
498vllm:request_decode_time_seconds_bucket{engine="0",le="0.3",model_name="cope-a-9b"} 2.0
499vllm:request_decode_time_seconds_bucket{engine="0",le="0.5",model_name="cope-a-9b"} 2.0
500vllm:request_decode_time_seconds_bucket{engine="0",le="0.8",model_name="cope-a-9b"} 2.0
501vllm:request_decode_time_seconds_bucket{engine="0",le="1.0",model_name="cope-a-9b"} 2.0
502vllm:request_decode_time_seconds_bucket{engine="0",le="1.5",model_name="cope-a-9b"} 2.0
503vllm:request_decode_time_seconds_bucket{engine="0",le="2.0",model_name="cope-a-9b"} 2.0
504vllm:request_decode_time_seconds_bucket{engine="0",le="2.5",model_name="cope-a-9b"} 2.0
505vllm:request_decode_time_seconds_bucket{engine="0",le="5.0",model_name="cope-a-9b"} 2.0
506vllm:request_decode_time_seconds_bucket{engine="0",le="10.0",model_name="cope-a-9b"} 2.0
507vllm:request_decode_time_seconds_bucket{engine="0",le="15.0",model_name="cope-a-9b"} 2.0
508vllm:request_decode_time_seconds_bucket{engine="0",le="20.0",model_name="cope-a-9b"} 2.0
509vllm:request_decode_time_seconds_bucket{engine="0",le="30.0",model_name="cope-a-9b"} 2.0
510vllm:request_decode_time_seconds_bucket{engine="0",le="40.0",model_name="cope-a-9b"} 2.0
511vllm:request_decode_time_seconds_bucket{engine="0",le="50.0",model_name="cope-a-9b"} 2.0
512vllm:request_decode_time_seconds_bucket{engine="0",le="60.0",model_name="cope-a-9b"} 2.0
513vllm:request_decode_time_seconds_bucket{engine="0",le="120.0",model_name="cope-a-9b"} 2.0
514vllm:request_decode_time_seconds_bucket{engine="0",le="240.0",model_name="cope-a-9b"} 2.0
515vllm:request_decode_time_seconds_bucket{engine="0",le="480.0",model_name="cope-a-9b"} 2.0
516vllm:request_decode_time_seconds_bucket{engine="0",le="960.0",model_name="cope-a-9b"} 2.0
517vllm:request_decode_time_seconds_bucket{engine="0",le="1920.0",model_name="cope-a-9b"} 2.0
518vllm:request_decode_time_seconds_bucket{engine="0",le="7680.0",model_name="cope-a-9b"} 2.0
519vllm:request_decode_time_seconds_bucket{engine="0",le="+Inf",model_name="cope-a-9b"} 2.0
520vllm:request_decode_time_seconds_count{engine="0",model_name="cope-a-9b"} 2.0
521vllm:request_decode_time_seconds_sum{engine="0",model_name="cope-a-9b"} 0.0
522# HELP vllm:request_decode_time_seconds_created Histogram of time spent in DECODE phase for request.
523# TYPE vllm:request_decode_time_seconds_created gauge
524vllm:request_decode_time_seconds_created{engine="0",model_name="cope-a-9b"} 1.7834186890813763e+09
525# HELP vllm:request_prefill_kv_computed_tokens Histogram of new KV tokens computed during prefill (excluding cached tokens).
526# TYPE vllm:request_prefill_kv_computed_tokens histogram
527vllm:request_prefill_kv_computed_tokens_bucket{engine="0",le="1.0",model_name="cope-a-9b"} 0.0
528vllm:request_prefill_kv_computed_tokens_bucket{engine="0",le="2.0",model_name="cope-a-9b"} 0.0
529vllm:request_prefill_kv_computed_tokens_bucket{engine="0",le="5.0",model_name="cope-a-9b"} 0.0
530vllm:request_prefill_kv_computed_tokens_bucket{engine="0",le="10.0",model_name="cope-a-9b"} 1.0
531vllm:request_prefill_kv_computed_tokens_bucket{engine="0",le="20.0",model_name="cope-a-9b"} 1.0
532vllm:request_prefill_kv_computed_tokens_bucket{engine="0",le="50.0",model_name="cope-a-9b"} 1.0
533vllm:request_prefill_kv_computed_tokens_bucket{engine="0",le="100.0",model_name="cope-a-9b"} 2.0
534vllm:request_prefill_kv_computed_tokens_bucket{engine="0",le="200.0",model_name="cope-a-9b"} 2.0
535vllm:request_prefill_kv_computed_tokens_bucket{engine="0",le="500.0",model_name="cope-a-9b"} 2.0
536vllm:request_prefill_kv_computed_tokens_bucket{engine="0",le="1000.0",model_name="cope-a-9b"} 2.0
537vllm:request_prefill_kv_computed_tokens_bucket{engine="0",le="2000.0",model_name="cope-a-9b"} 2.0
538vllm:request_prefill_kv_computed_tokens_bucket{engine="0",le="+Inf",model_name="cope-a-9b"} 2.0
539vllm:request_prefill_kv_computed_tokens_count{engine="0",model_name="cope-a-9b"} 2.0
540vllm:request_prefill_kv_computed_tokens_sum{engine="0",model_name="cope-a-9b"} 96.0
541# HELP vllm:request_prefill_kv_computed_tokens_created Histogram of new KV tokens computed during prefill (excluding cached tokens).
542# TYPE vllm:request_prefill_kv_computed_tokens_created gauge
543vllm:request_prefill_kv_computed_tokens_created{engine="0",model_name="cope-a-9b"} 1.7834186890814133e+09
544# HELP vllm:cache_config_info Information of the LLMEngine CacheConfig
545# TYPE vllm:cache_config_info gauge
546vllm:cache_config_info{block_size="16",cache_dtype="auto",calculate_kv_scales="False",cpu_kvcache_space_bytes="None",cpu_offload_gb="0.0",enable_prefix_caching="True",engine="0",gpu_memory_utilization="0.9",is_attention_free="False",kv_cache_memory_bytes="None",kv_offloading_backend="native",kv_offloading_size="None",kv_sharing_fast_prefill="False",mamba_block_size="None",mamba_cache_dtype="auto",mamba_cache_mode="none",mamba_page_size_padded="None",mamba_ssm_cache_dtype="auto",num_cpu_blocks="None",num_gpu_blocks="548",num_gpu_blocks_override="None",prefix_caching_hash_algo="sha256",sliding_window="None",swap_space="4.0"} 1.0

Change #1308050 merged by jenkins-bot:

[operations/deployment-charts@master] ml-services: deploy gpt-oss-safeguard-20b isvc image that emits vLLM metrics

https://gerrit.wikimedia.org/r/1308050

The gpt-oss-safeguard-20b isvc now emits vLLM metrics as shown below:

1$ curl "https://inference.svc.eqiad.wmnet:30443/metrics" -H "Host: gpt-oss-safeguard-20b.llm.wikimedia.org"
2# HELP python_gc_objects_collected_total Objects collected during gc
3# TYPE python_gc_objects_collected_total counter
4python_gc_objects_collected_total{generation="0"} 17910.0
5python_gc_objects_collected_total{generation="1"} 2773.0
6python_gc_objects_collected_total{generation="2"} 152.0
7# HELP python_gc_objects_uncollectable_total Uncollectable objects found during GC
8# TYPE python_gc_objects_uncollectable_total counter
9python_gc_objects_uncollectable_total{generation="0"} 0.0
10python_gc_objects_uncollectable_total{generation="1"} 0.0
11python_gc_objects_uncollectable_total{generation="2"} 0.0
12# HELP python_gc_collections_total Number of times this generation was collected
13# TYPE python_gc_collections_total counter
14python_gc_collections_total{generation="0"} 1897.0
15python_gc_collections_total{generation="1"} 172.0
16python_gc_collections_total{generation="2"} 10.0
17# HELP python_info Python platform information
18# TYPE python_info gauge
19python_info{implementation="CPython",major="3",minor="11",patchlevel="2",version="3.11.2"} 1.0
20# HELP process_virtual_memory_bytes Virtual memory size in bytes.
21# TYPE process_virtual_memory_bytes gauge
22process_virtual_memory_bytes 8.0953356288e+010
23# HELP process_resident_memory_bytes Resident memory size in bytes.
24# TYPE process_resident_memory_bytes gauge
25process_resident_memory_bytes 1.856602112e+09
26# HELP process_start_time_seconds Start time of the process since unix epoch in seconds.
27# TYPE process_start_time_seconds gauge
28process_start_time_seconds 1.78341965191e+09
29# HELP process_cpu_seconds_total Total user and system CPU time spent in seconds.
30# TYPE process_cpu_seconds_total counter
31process_cpu_seconds_total 12.24
32# HELP process_open_fds Number of open file descriptors.
33# TYPE process_open_fds gauge
34process_open_fds 36.0
35# HELP process_max_fds Maximum number of open file descriptors.
36# TYPE process_max_fds gauge
37process_max_fds 1.073741816e+09
38# HELP request_preprocess_seconds pre-process request latency
39# TYPE request_preprocess_seconds histogram
40request_preprocess_seconds_bucket{le="0.005",model_name="gpt-oss-safeguard-20b"} 2.0
41request_preprocess_seconds_bucket{le="0.01",model_name="gpt-oss-safeguard-20b"} 2.0
42request_preprocess_seconds_bucket{le="0.025",model_name="gpt-oss-safeguard-20b"} 2.0
43request_preprocess_seconds_bucket{le="0.05",model_name="gpt-oss-safeguard-20b"} 2.0
44request_preprocess_seconds_bucket{le="0.075",model_name="gpt-oss-safeguard-20b"} 2.0
45request_preprocess_seconds_bucket{le="0.1",model_name="gpt-oss-safeguard-20b"} 2.0
46request_preprocess_seconds_bucket{le="0.25",model_name="gpt-oss-safeguard-20b"} 2.0
47request_preprocess_seconds_bucket{le="0.5",model_name="gpt-oss-safeguard-20b"} 2.0
48request_preprocess_seconds_bucket{le="0.75",model_name="gpt-oss-safeguard-20b"} 2.0
49request_preprocess_seconds_bucket{le="1.0",model_name="gpt-oss-safeguard-20b"} 2.0
50request_preprocess_seconds_bucket{le="2.5",model_name="gpt-oss-safeguard-20b"} 2.0
51request_preprocess_seconds_bucket{le="5.0",model_name="gpt-oss-safeguard-20b"} 2.0
52request_preprocess_seconds_bucket{le="7.5",model_name="gpt-oss-safeguard-20b"} 2.0
53request_preprocess_seconds_bucket{le="10.0",model_name="gpt-oss-safeguard-20b"} 2.0
54request_preprocess_seconds_bucket{le="+Inf",model_name="gpt-oss-safeguard-20b"} 2.0
55request_preprocess_seconds_count{model_name="gpt-oss-safeguard-20b"} 2.0
56request_preprocess_seconds_sum{model_name="gpt-oss-safeguard-20b"} 0.004773342050611973
57# HELP request_preprocess_seconds_created pre-process request latency
58# TYPE request_preprocess_seconds_created gauge
59request_preprocess_seconds_created{model_name="gpt-oss-safeguard-20b"} 1.7834199889188945e+09
60# HELP request_postprocess_seconds post-process request latency
61# TYPE request_postprocess_seconds histogram
62request_postprocess_seconds_bucket{le="0.005",model_name="gpt-oss-safeguard-20b"} 2.0
63request_postprocess_seconds_bucket{le="0.01",model_name="gpt-oss-safeguard-20b"} 2.0
64request_postprocess_seconds_bucket{le="0.025",model_name="gpt-oss-safeguard-20b"} 2.0
65request_postprocess_seconds_bucket{le="0.05",model_name="gpt-oss-safeguard-20b"} 2.0
66request_postprocess_seconds_bucket{le="0.075",model_name="gpt-oss-safeguard-20b"} 2.0
67request_postprocess_seconds_bucket{le="0.1",model_name="gpt-oss-safeguard-20b"} 2.0
68request_postprocess_seconds_bucket{le="0.25",model_name="gpt-oss-safeguard-20b"} 2.0
69request_postprocess_seconds_bucket{le="0.5",model_name="gpt-oss-safeguard-20b"} 2.0
70request_postprocess_seconds_bucket{le="0.75",model_name="gpt-oss-safeguard-20b"} 2.0
71request_postprocess_seconds_bucket{le="1.0",model_name="gpt-oss-safeguard-20b"} 2.0
72request_postprocess_seconds_bucket{le="2.5",model_name="gpt-oss-safeguard-20b"} 2.0
73request_postprocess_seconds_bucket{le="5.0",model_name="gpt-oss-safeguard-20b"} 2.0
74request_postprocess_seconds_bucket{le="7.5",model_name="gpt-oss-safeguard-20b"} 2.0
75request_postprocess_seconds_bucket{le="10.0",model_name="gpt-oss-safeguard-20b"} 2.0
76request_postprocess_seconds_bucket{le="+Inf",model_name="gpt-oss-safeguard-20b"} 2.0
77request_postprocess_seconds_count{model_name="gpt-oss-safeguard-20b"} 2.0
78request_postprocess_seconds_sum{model_name="gpt-oss-safeguard-20b"} 0.0020898962393403053
79# HELP request_postprocess_seconds_created post-process request latency
80# TYPE request_postprocess_seconds_created gauge
81request_postprocess_seconds_created{model_name="gpt-oss-safeguard-20b"} 1.783419990027361e+09
82# HELP request_predict_seconds predict request latency
83# TYPE request_predict_seconds histogram
84request_predict_seconds_bucket{le="0.005",model_name="gpt-oss-safeguard-20b"} 0.0
85request_predict_seconds_bucket{le="0.01",model_name="gpt-oss-safeguard-20b"} 0.0
86request_predict_seconds_bucket{le="0.025",model_name="gpt-oss-safeguard-20b"} 0.0
87request_predict_seconds_bucket{le="0.05",model_name="gpt-oss-safeguard-20b"} 0.0
88request_predict_seconds_bucket{le="0.075",model_name="gpt-oss-safeguard-20b"} 0.0
89request_predict_seconds_bucket{le="0.1",model_name="gpt-oss-safeguard-20b"} 0.0
90request_predict_seconds_bucket{le="0.25",model_name="gpt-oss-safeguard-20b"} 0.0
91request_predict_seconds_bucket{le="0.5",model_name="gpt-oss-safeguard-20b"} 0.0
92request_predict_seconds_bucket{le="0.75",model_name="gpt-oss-safeguard-20b"} 1.0
93request_predict_seconds_bucket{le="1.0",model_name="gpt-oss-safeguard-20b"} 1.0
94request_predict_seconds_bucket{le="2.5",model_name="gpt-oss-safeguard-20b"} 2.0
95request_predict_seconds_bucket{le="5.0",model_name="gpt-oss-safeguard-20b"} 2.0
96request_predict_seconds_bucket{le="7.5",model_name="gpt-oss-safeguard-20b"} 2.0
97request_predict_seconds_bucket{le="10.0",model_name="gpt-oss-safeguard-20b"} 2.0
98request_predict_seconds_bucket{le="+Inf",model_name="gpt-oss-safeguard-20b"} 2.0
99request_predict_seconds_count{model_name="gpt-oss-safeguard-20b"} 2.0
100request_predict_seconds_sum{model_name="gpt-oss-safeguard-20b"} 1.6090969983488321
101# HELP request_predict_seconds_created predict request latency
102# TYPE request_predict_seconds_created gauge
103request_predict_seconds_created{model_name="gpt-oss-safeguard-20b"} 1.7834199889229088e+09
104# HELP request_explain_seconds explain request latency
105# TYPE request_explain_seconds histogram
106# HELP vllm:num_requests_running Number of requests in model execution batches.
107# TYPE vllm:num_requests_running gauge
108vllm:num_requests_running{engine="0",model_name="gpt-oss-safeguard-20b"} 0.0
109# HELP vllm:num_requests_waiting Number of requests waiting to be processed.
110# TYPE vllm:num_requests_waiting gauge
111vllm:num_requests_waiting{engine="0",model_name="gpt-oss-safeguard-20b"} 0.0
112# HELP vllm:engine_sleep_state Engine sleep state; awake = 0 means engine is sleeping; awake = 1 means engine is awake; weights_offloaded = 1 means sleep level 1; discard_all = 1 means sleep level 2.
113# TYPE vllm:engine_sleep_state gauge
114vllm:engine_sleep_state{engine="0",model_name="gpt-oss-safeguard-20b",sleep_state="awake"} 1.0
115vllm:engine_sleep_state{engine="0",model_name="gpt-oss-safeguard-20b",sleep_state="weights_offloaded"} 0.0
116vllm:engine_sleep_state{engine="0",model_name="gpt-oss-safeguard-20b",sleep_state="discard_all"} 0.0
117# HELP vllm:kv_cache_usage_perc KV-cache usage. 1 means 100 percent usage.
118# TYPE vllm:kv_cache_usage_perc gauge
119vllm:kv_cache_usage_perc{engine="0",model_name="gpt-oss-safeguard-20b"} 0.0
120# HELP vllm:prefix_cache_queries_total Prefix cache queries, in terms of number of queried tokens.
121# TYPE vllm:prefix_cache_queries_total counter
122vllm:prefix_cache_queries_total{engine="0",model_name="gpt-oss-safeguard-20b"} 384.0
123# HELP vllm:prefix_cache_queries_created Prefix cache queries, in terms of number of queried tokens.
124# TYPE vllm:prefix_cache_queries_created gauge
125vllm:prefix_cache_queries_created{engine="0",model_name="gpt-oss-safeguard-20b"} 1.783419765514034e+09
126# HELP vllm:prefix_cache_hits_total Prefix cache hits, in terms of number of cached tokens.
127# TYPE vllm:prefix_cache_hits_total counter
128vllm:prefix_cache_hits_total{engine="0",model_name="gpt-oss-safeguard-20b"} 128.0
129# HELP vllm:prefix_cache_hits_created Prefix cache hits, in terms of number of cached tokens.
130# TYPE vllm:prefix_cache_hits_created gauge
131vllm:prefix_cache_hits_created{engine="0",model_name="gpt-oss-safeguard-20b"} 1.7834197655140493e+09
132# HELP vllm:external_prefix_cache_queries_total External prefix cache queries from KV connector cross-instance cache sharing, in terms of number of queried tokens.
133# TYPE vllm:external_prefix_cache_queries_total counter
134vllm:external_prefix_cache_queries_total{engine="0",model_name="gpt-oss-safeguard-20b"} 0.0
135# HELP vllm:external_prefix_cache_queries_created External prefix cache queries from KV connector cross-instance cache sharing, in terms of number of queried tokens.
136# TYPE vllm:external_prefix_cache_queries_created gauge
137vllm:external_prefix_cache_queries_created{engine="0",model_name="gpt-oss-safeguard-20b"} 1.7834197655140636e+09
138# HELP vllm:external_prefix_cache_hits_total External prefix cache hits from KV connector cross-instance cache sharing, in terms of number of cached tokens.
139# TYPE vllm:external_prefix_cache_hits_total counter
140vllm:external_prefix_cache_hits_total{engine="0",model_name="gpt-oss-safeguard-20b"} 0.0
141# HELP vllm:external_prefix_cache_hits_created External prefix cache hits from KV connector cross-instance cache sharing, in terms of number of cached tokens.
142# TYPE vllm:external_prefix_cache_hits_created gauge
143vllm:external_prefix_cache_hits_created{engine="0",model_name="gpt-oss-safeguard-20b"} 1.7834197655140743e+09
144# HELP vllm:mm_cache_queries_total Multi-modal cache queries, in terms of number of queried items.
145# TYPE vllm:mm_cache_queries_total counter
146vllm:mm_cache_queries_total{engine="0",model_name="gpt-oss-safeguard-20b"} 0.0
147# HELP vllm:mm_cache_queries_created Multi-modal cache queries, in terms of number of queried items.
148# TYPE vllm:mm_cache_queries_created gauge
149vllm:mm_cache_queries_created{engine="0",model_name="gpt-oss-safeguard-20b"} 1.7834197655140846e+09
150# HELP vllm:mm_cache_hits_total Multi-modal cache hits, in terms of number of cached items.
151# TYPE vllm:mm_cache_hits_total counter
152vllm:mm_cache_hits_total{engine="0",model_name="gpt-oss-safeguard-20b"} 0.0
153# HELP vllm:mm_cache_hits_created Multi-modal cache hits, in terms of number of cached items.
154# TYPE vllm:mm_cache_hits_created gauge
155vllm:mm_cache_hits_created{engine="0",model_name="gpt-oss-safeguard-20b"} 1.7834197655140948e+09
156# HELP vllm:num_preemptions_total Cumulative number of preemption from the engine.
157# TYPE vllm:num_preemptions_total counter
158vllm:num_preemptions_total{engine="0",model_name="gpt-oss-safeguard-20b"} 0.0
159# HELP vllm:num_preemptions_created Cumulative number of preemption from the engine.
160# TYPE vllm:num_preemptions_created gauge
161vllm:num_preemptions_created{engine="0",model_name="gpt-oss-safeguard-20b"} 1.7834197655141072e+09
162# HELP vllm:prompt_tokens_total Number of prefill tokens processed.
163# TYPE vllm:prompt_tokens_total counter
164vllm:prompt_tokens_total{engine="0",model_name="gpt-oss-safeguard-20b"} 384.0
165# HELP vllm:prompt_tokens_created Number of prefill tokens processed.
166# TYPE vllm:prompt_tokens_created gauge
167vllm:prompt_tokens_created{engine="0",model_name="gpt-oss-safeguard-20b"} 1.7834197655141358e+09
168# HELP vllm:generation_tokens_total Number of generation tokens processed.
169# TYPE vllm:generation_tokens_total counter
170vllm:generation_tokens_total{engine="0",model_name="gpt-oss-safeguard-20b"} 230.0
171# HELP vllm:generation_tokens_created Number of generation tokens processed.
172# TYPE vllm:generation_tokens_created gauge
173vllm:generation_tokens_created{engine="0",model_name="gpt-oss-safeguard-20b"} 1.7834197655141468e+09
174# HELP vllm:request_success_total Count of successfully processed requests.
175# TYPE vllm:request_success_total counter
176vllm:request_success_total{engine="0",finished_reason="stop",model_name="gpt-oss-safeguard-20b"} 2.0
177vllm:request_success_total{engine="0",finished_reason="length",model_name="gpt-oss-safeguard-20b"} 0.0
178vllm:request_success_total{engine="0",finished_reason="abort",model_name="gpt-oss-safeguard-20b"} 0.0
179vllm:request_success_total{engine="0",finished_reason="error",model_name="gpt-oss-safeguard-20b"} 0.0
180# HELP vllm:request_success_created Count of successfully processed requests.
181# TYPE vllm:request_success_created gauge
182vllm:request_success_created{engine="0",finished_reason="stop",model_name="gpt-oss-safeguard-20b"} 1.7834197655141695e+09
183vllm:request_success_created{engine="0",finished_reason="length",model_name="gpt-oss-safeguard-20b"} 1.783419765514176e+09
184vllm:request_success_created{engine="0",finished_reason="abort",model_name="gpt-oss-safeguard-20b"} 1.7834197655141842e+09
185vllm:request_success_created{engine="0",finished_reason="error",model_name="gpt-oss-safeguard-20b"} 1.783419765514189e+09
186# HELP vllm:request_prompt_tokens Number of prefill tokens processed.
187# TYPE vllm:request_prompt_tokens histogram
188vllm:request_prompt_tokens_bucket{engine="0",le="1.0",model_name="gpt-oss-safeguard-20b"} 0.0
189vllm:request_prompt_tokens_bucket{engine="0",le="2.0",model_name="gpt-oss-safeguard-20b"} 0.0
190vllm:request_prompt_tokens_bucket{engine="0",le="5.0",model_name="gpt-oss-safeguard-20b"} 0.0
191vllm:request_prompt_tokens_bucket{engine="0",le="10.0",model_name="gpt-oss-safeguard-20b"} 0.0
192vllm:request_prompt_tokens_bucket{engine="0",le="20.0",model_name="gpt-oss-safeguard-20b"} 0.0
193vllm:request_prompt_tokens_bucket{engine="0",le="50.0",model_name="gpt-oss-safeguard-20b"} 0.0
194vllm:request_prompt_tokens_bucket{engine="0",le="100.0",model_name="gpt-oss-safeguard-20b"} 0.0
195vllm:request_prompt_tokens_bucket{engine="0",le="200.0",model_name="gpt-oss-safeguard-20b"} 2.0
196vllm:request_prompt_tokens_bucket{engine="0",le="500.0",model_name="gpt-oss-safeguard-20b"} 2.0
197vllm:request_prompt_tokens_bucket{engine="0",le="1000.0",model_name="gpt-oss-safeguard-20b"} 2.0
198vllm:request_prompt_tokens_bucket{engine="0",le="2000.0",model_name="gpt-oss-safeguard-20b"} 2.0
199vllm:request_prompt_tokens_bucket{engine="0",le="5000.0",model_name="gpt-oss-safeguard-20b"} 2.0
200vllm:request_prompt_tokens_bucket{engine="0",le="10000.0",model_name="gpt-oss-safeguard-20b"} 2.0
201vllm:request_prompt_tokens_bucket{engine="0",le="+Inf",model_name="gpt-oss-safeguard-20b"} 2.0
202vllm:request_prompt_tokens_count{engine="0",model_name="gpt-oss-safeguard-20b"} 2.0
203vllm:request_prompt_tokens_sum{engine="0",model_name="gpt-oss-safeguard-20b"} 384.0
204# HELP vllm:request_prompt_tokens_created Number of prefill tokens processed.
205# TYPE vllm:request_prompt_tokens_created gauge
206vllm:request_prompt_tokens_created{engine="0",model_name="gpt-oss-safeguard-20b"} 1.7834197655142324e+09
207# HELP vllm:request_generation_tokens Number of generation tokens processed.
208# TYPE vllm:request_generation_tokens histogram
209vllm:request_generation_tokens_bucket{engine="0",le="1.0",model_name="gpt-oss-safeguard-20b"} 0.0
210vllm:request_generation_tokens_bucket{engine="0",le="2.0",model_name="gpt-oss-safeguard-20b"} 0.0
211vllm:request_generation_tokens_bucket{engine="0",le="5.0",model_name="gpt-oss-safeguard-20b"} 0.0
212vllm:request_generation_tokens_bucket{engine="0",le="10.0",model_name="gpt-oss-safeguard-20b"} 0.0
213vllm:request_generation_tokens_bucket{engine="0",le="20.0",model_name="gpt-oss-safeguard-20b"} 0.0
214vllm:request_generation_tokens_bucket{engine="0",le="50.0",model_name="gpt-oss-safeguard-20b"} 0.0
215vllm:request_generation_tokens_bucket{engine="0",le="100.0",model_name="gpt-oss-safeguard-20b"} 0.0
216vllm:request_generation_tokens_bucket{engine="0",le="200.0",model_name="gpt-oss-safeguard-20b"} 2.0
217vllm:request_generation_tokens_bucket{engine="0",le="500.0",model_name="gpt-oss-safeguard-20b"} 2.0
218vllm:request_generation_tokens_bucket{engine="0",le="1000.0",model_name="gpt-oss-safeguard-20b"} 2.0
219vllm:request_generation_tokens_bucket{engine="0",le="2000.0",model_name="gpt-oss-safeguard-20b"} 2.0
220vllm:request_generation_tokens_bucket{engine="0",le="5000.0",model_name="gpt-oss-safeguard-20b"} 2.0
221vllm:request_generation_tokens_bucket{engine="0",le="10000.0",model_name="gpt-oss-safeguard-20b"} 2.0
222vllm:request_generation_tokens_bucket{engine="0",le="+Inf",model_name="gpt-oss-safeguard-20b"} 2.0
223vllm:request_generation_tokens_count{engine="0",model_name="gpt-oss-safeguard-20b"} 2.0
224vllm:request_generation_tokens_sum{engine="0",model_name="gpt-oss-safeguard-20b"} 230.0
225# HELP vllm:request_generation_tokens_created Number of generation tokens processed.
226# TYPE vllm:request_generation_tokens_created gauge
227vllm:request_generation_tokens_created{engine="0",model_name="gpt-oss-safeguard-20b"} 1.7834197655142865e+09
228# HELP vllm:iteration_tokens_total Histogram of number of tokens per engine_step.
229# TYPE vllm:iteration_tokens_total histogram
230vllm:iteration_tokens_total_bucket{engine="0",le="1.0",model_name="gpt-oss-safeguard-20b"} 228.0
231vllm:iteration_tokens_total_bucket{engine="0",le="8.0",model_name="gpt-oss-safeguard-20b"} 228.0
232vllm:iteration_tokens_total_bucket{engine="0",le="16.0",model_name="gpt-oss-safeguard-20b"} 228.0
233vllm:iteration_tokens_total_bucket{engine="0",le="32.0",model_name="gpt-oss-safeguard-20b"} 228.0
234vllm:iteration_tokens_total_bucket{engine="0",le="64.0",model_name="gpt-oss-safeguard-20b"} 228.0
235vllm:iteration_tokens_total_bucket{engine="0",le="128.0",model_name="gpt-oss-safeguard-20b"} 228.0
236vllm:iteration_tokens_total_bucket{engine="0",le="256.0",model_name="gpt-oss-safeguard-20b"} 230.0
237vllm:iteration_tokens_total_bucket{engine="0",le="512.0",model_name="gpt-oss-safeguard-20b"} 230.0
238vllm:iteration_tokens_total_bucket{engine="0",le="1024.0",model_name="gpt-oss-safeguard-20b"} 230.0
239vllm:iteration_tokens_total_bucket{engine="0",le="2048.0",model_name="gpt-oss-safeguard-20b"} 230.0
240vllm:iteration_tokens_total_bucket{engine="0",le="4096.0",model_name="gpt-oss-safeguard-20b"} 230.0
241vllm:iteration_tokens_total_bucket{engine="0",le="8192.0",model_name="gpt-oss-safeguard-20b"} 230.0
242vllm:iteration_tokens_total_bucket{engine="0",le="16384.0",model_name="gpt-oss-safeguard-20b"} 230.0
243vllm:iteration_tokens_total_bucket{engine="0",le="+Inf",model_name="gpt-oss-safeguard-20b"} 230.0
244vllm:iteration_tokens_total_count{engine="0",model_name="gpt-oss-safeguard-20b"} 230.0
245vllm:iteration_tokens_total_sum{engine="0",model_name="gpt-oss-safeguard-20b"} 614.0
246# HELP vllm:iteration_tokens_total_created Histogram of number of tokens per engine_step.
247# TYPE vllm:iteration_tokens_total_created gauge
248vllm:iteration_tokens_total_created{engine="0",model_name="gpt-oss-safeguard-20b"} 1.7834197655143168e+09
249# HELP vllm:request_max_num_generation_tokens Histogram of maximum number of requested generation tokens.
250# TYPE vllm:request_max_num_generation_tokens histogram
251vllm:request_max_num_generation_tokens_bucket{engine="0",le="1.0",model_name="gpt-oss-safeguard-20b"} 0.0
252vllm:request_max_num_generation_tokens_bucket{engine="0",le="2.0",model_name="gpt-oss-safeguard-20b"} 0.0
253vllm:request_max_num_generation_tokens_bucket{engine="0",le="5.0",model_name="gpt-oss-safeguard-20b"} 0.0
254vllm:request_max_num_generation_tokens_bucket{engine="0",le="10.0",model_name="gpt-oss-safeguard-20b"} 0.0
255vllm:request_max_num_generation_tokens_bucket{engine="0",le="20.0",model_name="gpt-oss-safeguard-20b"} 0.0
256vllm:request_max_num_generation_tokens_bucket{engine="0",le="50.0",model_name="gpt-oss-safeguard-20b"} 0.0
257vllm:request_max_num_generation_tokens_bucket{engine="0",le="100.0",model_name="gpt-oss-safeguard-20b"} 0.0
258vllm:request_max_num_generation_tokens_bucket{engine="0",le="200.0",model_name="gpt-oss-safeguard-20b"} 2.0
259vllm:request_max_num_generation_tokens_bucket{engine="0",le="500.0",model_name="gpt-oss-safeguard-20b"} 2.0
260vllm:request_max_num_generation_tokens_bucket{engine="0",le="1000.0",model_name="gpt-oss-safeguard-20b"} 2.0
261vllm:request_max_num_generation_tokens_bucket{engine="0",le="2000.0",model_name="gpt-oss-safeguard-20b"} 2.0
262vllm:request_max_num_generation_tokens_bucket{engine="0",le="5000.0",model_name="gpt-oss-safeguard-20b"} 2.0
263vllm:request_max_num_generation_tokens_bucket{engine="0",le="10000.0",model_name="gpt-oss-safeguard-20b"} 2.0
264vllm:request_max_num_generation_tokens_bucket{engine="0",le="+Inf",model_name="gpt-oss-safeguard-20b"} 2.0
265vllm:request_max_num_generation_tokens_count{engine="0",model_name="gpt-oss-safeguard-20b"} 2.0
266vllm:request_max_num_generation_tokens_sum{engine="0",model_name="gpt-oss-safeguard-20b"} 230.0
267# HELP vllm:request_max_num_generation_tokens_created Histogram of maximum number of requested generation tokens.
268# TYPE vllm:request_max_num_generation_tokens_created gauge
269vllm:request_max_num_generation_tokens_created{engine="0",model_name="gpt-oss-safeguard-20b"} 1.7834197655143461e+09
270# HELP vllm:request_params_n Histogram of the n request parameter.
271# TYPE vllm:request_params_n histogram
272vllm:request_params_n_bucket{engine="0",le="1.0",model_name="gpt-oss-safeguard-20b"} 2.0
273vllm:request_params_n_bucket{engine="0",le="2.0",model_name="gpt-oss-safeguard-20b"} 2.0
274vllm:request_params_n_bucket{engine="0",le="5.0",model_name="gpt-oss-safeguard-20b"} 2.0
275vllm:request_params_n_bucket{engine="0",le="10.0",model_name="gpt-oss-safeguard-20b"} 2.0
276vllm:request_params_n_bucket{engine="0",le="20.0",model_name="gpt-oss-safeguard-20b"} 2.0
277vllm:request_params_n_bucket{engine="0",le="+Inf",model_name="gpt-oss-safeguard-20b"} 2.0
278vllm:request_params_n_count{engine="0",model_name="gpt-oss-safeguard-20b"} 2.0
279vllm:request_params_n_sum{engine="0",model_name="gpt-oss-safeguard-20b"} 2.0
280# HELP vllm:request_params_n_created Histogram of the n request parameter.
281# TYPE vllm:request_params_n_created gauge
282vllm:request_params_n_created{engine="0",model_name="gpt-oss-safeguard-20b"} 1.7834197655143735e+09
283# HELP vllm:request_params_max_tokens Histogram of the max_tokens request parameter.
284# TYPE vllm:request_params_max_tokens histogram
285vllm:request_params_max_tokens_bucket{engine="0",le="1.0",model_name="gpt-oss-safeguard-20b"} 0.0
286vllm:request_params_max_tokens_bucket{engine="0",le="2.0",model_name="gpt-oss-safeguard-20b"} 0.0
287vllm:request_params_max_tokens_bucket{engine="0",le="5.0",model_name="gpt-oss-safeguard-20b"} 0.0
288vllm:request_params_max_tokens_bucket{engine="0",le="10.0",model_name="gpt-oss-safeguard-20b"} 0.0
289vllm:request_params_max_tokens_bucket{engine="0",le="20.0",model_name="gpt-oss-safeguard-20b"} 0.0
290vllm:request_params_max_tokens_bucket{engine="0",le="50.0",model_name="gpt-oss-safeguard-20b"} 0.0
291vllm:request_params_max_tokens_bucket{engine="0",le="100.0",model_name="gpt-oss-safeguard-20b"} 0.0
292vllm:request_params_max_tokens_bucket{engine="0",le="200.0",model_name="gpt-oss-safeguard-20b"} 0.0
293vllm:request_params_max_tokens_bucket{engine="0",le="500.0",model_name="gpt-oss-safeguard-20b"} 0.0
294vllm:request_params_max_tokens_bucket{engine="0",le="1000.0",model_name="gpt-oss-safeguard-20b"} 0.0
295vllm:request_params_max_tokens_bucket{engine="0",le="2000.0",model_name="gpt-oss-safeguard-20b"} 0.0
296vllm:request_params_max_tokens_bucket{engine="0",le="5000.0",model_name="gpt-oss-safeguard-20b"} 2.0
297vllm:request_params_max_tokens_bucket{engine="0",le="10000.0",model_name="gpt-oss-safeguard-20b"} 2.0
298vllm:request_params_max_tokens_bucket{engine="0",le="+Inf",model_name="gpt-oss-safeguard-20b"} 2.0
299vllm:request_params_max_tokens_count{engine="0",model_name="gpt-oss-safeguard-20b"} 2.0
300vllm:request_params_max_tokens_sum{engine="0",model_name="gpt-oss-safeguard-20b"} 8192.0
301# HELP vllm:request_params_max_tokens_created Histogram of the max_tokens request parameter.
302# TYPE vllm:request_params_max_tokens_created gauge
303vllm:request_params_max_tokens_created{engine="0",model_name="gpt-oss-safeguard-20b"} 1.783419765514398e+09
304# HELP vllm:time_to_first_token_seconds Histogram of time to first token in seconds.
305# TYPE vllm:time_to_first_token_seconds histogram
306vllm:time_to_first_token_seconds_bucket{engine="0",le="0.001",model_name="gpt-oss-safeguard-20b"} 0.0
307vllm:time_to_first_token_seconds_bucket{engine="0",le="0.005",model_name="gpt-oss-safeguard-20b"} 0.0
308vllm:time_to_first_token_seconds_bucket{engine="0",le="0.01",model_name="gpt-oss-safeguard-20b"} 0.0
309vllm:time_to_first_token_seconds_bucket{engine="0",le="0.02",model_name="gpt-oss-safeguard-20b"} 1.0
310vllm:time_to_first_token_seconds_bucket{engine="0",le="0.04",model_name="gpt-oss-safeguard-20b"} 1.0
311vllm:time_to_first_token_seconds_bucket{engine="0",le="0.06",model_name="gpt-oss-safeguard-20b"} 1.0
312vllm:time_to_first_token_seconds_bucket{engine="0",le="0.08",model_name="gpt-oss-safeguard-20b"} 1.0
313vllm:time_to_first_token_seconds_bucket{engine="0",le="0.1",model_name="gpt-oss-safeguard-20b"} 1.0
314vllm:time_to_first_token_seconds_bucket{engine="0",le="0.25",model_name="gpt-oss-safeguard-20b"} 1.0
315vllm:time_to_first_token_seconds_bucket{engine="0",le="0.5",model_name="gpt-oss-safeguard-20b"} 1.0
316vllm:time_to_first_token_seconds_bucket{engine="0",le="0.75",model_name="gpt-oss-safeguard-20b"} 2.0
317vllm:time_to_first_token_seconds_bucket{engine="0",le="1.0",model_name="gpt-oss-safeguard-20b"} 2.0
318vllm:time_to_first_token_seconds_bucket{engine="0",le="2.5",model_name="gpt-oss-safeguard-20b"} 2.0
319vllm:time_to_first_token_seconds_bucket{engine="0",le="5.0",model_name="gpt-oss-safeguard-20b"} 2.0
320vllm:time_to_first_token_seconds_bucket{engine="0",le="7.5",model_name="gpt-oss-safeguard-20b"} 2.0
321vllm:time_to_first_token_seconds_bucket{engine="0",le="10.0",model_name="gpt-oss-safeguard-20b"} 2.0
322vllm:time_to_first_token_seconds_bucket{engine="0",le="20.0",model_name="gpt-oss-safeguard-20b"} 2.0
323vllm:time_to_first_token_seconds_bucket{engine="0",le="40.0",model_name="gpt-oss-safeguard-20b"} 2.0
324vllm:time_to_first_token_seconds_bucket{engine="0",le="80.0",model_name="gpt-oss-safeguard-20b"} 2.0
325vllm:time_to_first_token_seconds_bucket{engine="0",le="160.0",model_name="gpt-oss-safeguard-20b"} 2.0
326vllm:time_to_first_token_seconds_bucket{engine="0",le="640.0",model_name="gpt-oss-safeguard-20b"} 2.0
327vllm:time_to_first_token_seconds_bucket{engine="0",le="2560.0",model_name="gpt-oss-safeguard-20b"} 2.0
328vllm:time_to_first_token_seconds_bucket{engine="0",le="+Inf",model_name="gpt-oss-safeguard-20b"} 2.0
329vllm:time_to_first_token_seconds_count{engine="0",model_name="gpt-oss-safeguard-20b"} 2.0
330vllm:time_to_first_token_seconds_sum{engine="0",model_name="gpt-oss-safeguard-20b"} 0.6833171844482422
331# HELP vllm:time_to_first_token_seconds_created Histogram of time to first token in seconds.
332# TYPE vllm:time_to_first_token_seconds_created gauge
333vllm:time_to_first_token_seconds_created{engine="0",model_name="gpt-oss-safeguard-20b"} 1.7834197655144267e+09
334# HELP vllm:inter_token_latency_seconds Histogram of inter-token latency in seconds.
335# TYPE vllm:inter_token_latency_seconds histogram
336vllm:inter_token_latency_seconds_bucket{engine="0",le="0.01",model_name="gpt-oss-safeguard-20b"} 228.0
337vllm:inter_token_latency_seconds_bucket{engine="0",le="0.025",model_name="gpt-oss-safeguard-20b"} 228.0
338vllm:inter_token_latency_seconds_bucket{engine="0",le="0.05",model_name="gpt-oss-safeguard-20b"} 228.0
339vllm:inter_token_latency_seconds_bucket{engine="0",le="0.075",model_name="gpt-oss-safeguard-20b"} 228.0
340vllm:inter_token_latency_seconds_bucket{engine="0",le="0.1",model_name="gpt-oss-safeguard-20b"} 228.0
341vllm:inter_token_latency_seconds_bucket{engine="0",le="0.15",model_name="gpt-oss-safeguard-20b"} 228.0
342vllm:inter_token_latency_seconds_bucket{engine="0",le="0.2",model_name="gpt-oss-safeguard-20b"} 228.0
343vllm:inter_token_latency_seconds_bucket{engine="0",le="0.3",model_name="gpt-oss-safeguard-20b"} 228.0
344vllm:inter_token_latency_seconds_bucket{engine="0",le="0.4",model_name="gpt-oss-safeguard-20b"} 228.0
345vllm:inter_token_latency_seconds_bucket{engine="0",le="0.5",model_name="gpt-oss-safeguard-20b"} 228.0
346vllm:inter_token_latency_seconds_bucket{engine="0",le="0.75",model_name="gpt-oss-safeguard-20b"} 228.0
347vllm:inter_token_latency_seconds_bucket{engine="0",le="1.0",model_name="gpt-oss-safeguard-20b"} 228.0
348vllm:inter_token_latency_seconds_bucket{engine="0",le="2.5",model_name="gpt-oss-safeguard-20b"} 228.0
349vllm:inter_token_latency_seconds_bucket{engine="0",le="5.0",model_name="gpt-oss-safeguard-20b"} 228.0
350vllm:inter_token_latency_seconds_bucket{engine="0",le="7.5",model_name="gpt-oss-safeguard-20b"} 228.0
351vllm:inter_token_latency_seconds_bucket{engine="0",le="10.0",model_name="gpt-oss-safeguard-20b"} 228.0
352vllm:inter_token_latency_seconds_bucket{engine="0",le="20.0",model_name="gpt-oss-safeguard-20b"} 228.0
353vllm:inter_token_latency_seconds_bucket{engine="0",le="40.0",model_name="gpt-oss-safeguard-20b"} 228.0
354vllm:inter_token_latency_seconds_bucket{engine="0",le="80.0",model_name="gpt-oss-safeguard-20b"} 228.0
355vllm:inter_token_latency_seconds_bucket{engine="0",le="+Inf",model_name="gpt-oss-safeguard-20b"} 228.0
356vllm:inter_token_latency_seconds_count{engine="0",model_name="gpt-oss-safeguard-20b"} 228.0
357vllm:inter_token_latency_seconds_sum{engine="0",model_name="gpt-oss-safeguard-20b"} 0.9259680965915322
358# HELP vllm:inter_token_latency_seconds_created Histogram of inter-token latency in seconds.
359# TYPE vllm:inter_token_latency_seconds_created gauge
360vllm:inter_token_latency_seconds_created{engine="0",model_name="gpt-oss-safeguard-20b"} 1.783419765514462e+09
361# HELP vllm:request_time_per_output_token_seconds Histogram of time_per_output_token_seconds per request.
362# TYPE vllm:request_time_per_output_token_seconds histogram
363vllm:request_time_per_output_token_seconds_bucket{engine="0",le="0.01",model_name="gpt-oss-safeguard-20b"} 2.0
364vllm:request_time_per_output_token_seconds_bucket{engine="0",le="0.025",model_name="gpt-oss-safeguard-20b"} 2.0
365vllm:request_time_per_output_token_seconds_bucket{engine="0",le="0.05",model_name="gpt-oss-safeguard-20b"} 2.0
366vllm:request_time_per_output_token_seconds_bucket{engine="0",le="0.075",model_name="gpt-oss-safeguard-20b"} 2.0
367vllm:request_time_per_output_token_seconds_bucket{engine="0",le="0.1",model_name="gpt-oss-safeguard-20b"} 2.0
368vllm:request_time_per_output_token_seconds_bucket{engine="0",le="0.15",model_name="gpt-oss-safeguard-20b"} 2.0
369vllm:request_time_per_output_token_seconds_bucket{engine="0",le="0.2",model_name="gpt-oss-safeguard-20b"} 2.0
370vllm:request_time_per_output_token_seconds_bucket{engine="0",le="0.3",model_name="gpt-oss-safeguard-20b"} 2.0
371vllm:request_time_per_output_token_seconds_bucket{engine="0",le="0.4",model_name="gpt-oss-safeguard-20b"} 2.0
372vllm:request_time_per_output_token_seconds_bucket{engine="0",le="0.5",model_name="gpt-oss-safeguard-20b"} 2.0
373vllm:request_time_per_output_token_seconds_bucket{engine="0",le="0.75",model_name="gpt-oss-safeguard-20b"} 2.0
374vllm:request_time_per_output_token_seconds_bucket{engine="0",le="1.0",model_name="gpt-oss-safeguard-20b"} 2.0
375vllm:request_time_per_output_token_seconds_bucket{engine="0",le="2.5",model_name="gpt-oss-safeguard-20b"} 2.0
376vllm:request_time_per_output_token_seconds_bucket{engine="0",le="5.0",model_name="gpt-oss-safeguard-20b"} 2.0
377vllm:request_time_per_output_token_seconds_bucket{engine="0",le="7.5",model_name="gpt-oss-safeguard-20b"} 2.0
378vllm:request_time_per_output_token_seconds_bucket{engine="0",le="10.0",model_name="gpt-oss-safeguard-20b"} 2.0
379vllm:request_time_per_output_token_seconds_bucket{engine="0",le="20.0",model_name="gpt-oss-safeguard-20b"} 2.0
380vllm:request_time_per_output_token_seconds_bucket{engine="0",le="40.0",model_name="gpt-oss-safeguard-20b"} 2.0
381vllm:request_time_per_output_token_seconds_bucket{engine="0",le="80.0",model_name="gpt-oss-safeguard-20b"} 2.0
382vllm:request_time_per_output_token_seconds_bucket{engine="0",le="+Inf",model_name="gpt-oss-safeguard-20b"} 2.0
383vllm:request_time_per_output_token_seconds_count{engine="0",model_name="gpt-oss-safeguard-20b"} 2.0
384vllm:request_time_per_output_token_seconds_sum{engine="0",model_name="gpt-oss-safeguard-20b"} 0.008122761428832082
385# HELP vllm:request_time_per_output_token_seconds_created Histogram of time_per_output_token_seconds per request.
386# TYPE vllm:request_time_per_output_token_seconds_created gauge
387vllm:request_time_per_output_token_seconds_created{engine="0",model_name="gpt-oss-safeguard-20b"} 1.783419765514494e+09
388# HELP vllm:e2e_request_latency_seconds Histogram of e2e request latency in seconds.
389# TYPE vllm:e2e_request_latency_seconds histogram
390vllm:e2e_request_latency_seconds_bucket{engine="0",le="0.3",model_name="gpt-oss-safeguard-20b"} 0.0
391vllm:e2e_request_latency_seconds_bucket{engine="0",le="0.5",model_name="gpt-oss-safeguard-20b"} 0.0
392vllm:e2e_request_latency_seconds_bucket{engine="0",le="0.8",model_name="gpt-oss-safeguard-20b"} 1.0
393vllm:e2e_request_latency_seconds_bucket{engine="0",le="1.0",model_name="gpt-oss-safeguard-20b"} 1.0
394vllm:e2e_request_latency_seconds_bucket{engine="0",le="1.5",model_name="gpt-oss-safeguard-20b"} 2.0
395vllm:e2e_request_latency_seconds_bucket{engine="0",le="2.0",model_name="gpt-oss-safeguard-20b"} 2.0
396vllm:e2e_request_latency_seconds_bucket{engine="0",le="2.5",model_name="gpt-oss-safeguard-20b"} 2.0
397vllm:e2e_request_latency_seconds_bucket{engine="0",le="5.0",model_name="gpt-oss-safeguard-20b"} 2.0
398vllm:e2e_request_latency_seconds_bucket{engine="0",le="10.0",model_name="gpt-oss-safeguard-20b"} 2.0
399vllm:e2e_request_latency_seconds_bucket{engine="0",le="15.0",model_name="gpt-oss-safeguard-20b"} 2.0
400vllm:e2e_request_latency_seconds_bucket{engine="0",le="20.0",model_name="gpt-oss-safeguard-20b"} 2.0
401vllm:e2e_request_latency_seconds_bucket{engine="0",le="30.0",model_name="gpt-oss-safeguard-20b"} 2.0
402vllm:e2e_request_latency_seconds_bucket{engine="0",le="40.0",model_name="gpt-oss-safeguard-20b"} 2.0
403vllm:e2e_request_latency_seconds_bucket{engine="0",le="50.0",model_name="gpt-oss-safeguard-20b"} 2.0
404vllm:e2e_request_latency_seconds_bucket{engine="0",le="60.0",model_name="gpt-oss-safeguard-20b"} 2.0
405vllm:e2e_request_latency_seconds_bucket{engine="0",le="120.0",model_name="gpt-oss-safeguard-20b"} 2.0
406vllm:e2e_request_latency_seconds_bucket{engine="0",le="240.0",model_name="gpt-oss-safeguard-20b"} 2.0
407vllm:e2e_request_latency_seconds_bucket{engine="0",le="480.0",model_name="gpt-oss-safeguard-20b"} 2.0
408vllm:e2e_request_latency_seconds_bucket{engine="0",le="960.0",model_name="gpt-oss-safeguard-20b"} 2.0
409vllm:e2e_request_latency_seconds_bucket{engine="0",le="1920.0",model_name="gpt-oss-safeguard-20b"} 2.0
410vllm:e2e_request_latency_seconds_bucket{engine="0",le="7680.0",model_name="gpt-oss-safeguard-20b"} 2.0
411vllm:e2e_request_latency_seconds_bucket{engine="0",le="+Inf",model_name="gpt-oss-safeguard-20b"} 2.0
412vllm:e2e_request_latency_seconds_count{engine="0",model_name="gpt-oss-safeguard-20b"} 2.0
413vllm:e2e_request_latency_seconds_sum{engine="0",model_name="gpt-oss-safeguard-20b"} 1.6084058284759521
414# HELP vllm:e2e_request_latency_seconds_created Histogram of e2e request latency in seconds.
415# TYPE vllm:e2e_request_latency_seconds_created gauge
416vllm:e2e_request_latency_seconds_created{engine="0",model_name="gpt-oss-safeguard-20b"} 1.7834197655145977e+09
417# HELP vllm:request_queue_time_seconds Histogram of time spent in WAITING phase for request.
418# TYPE vllm:request_queue_time_seconds histogram
419vllm:request_queue_time_seconds_bucket{engine="0",le="0.3",model_name="gpt-oss-safeguard-20b"} 2.0
420vllm:request_queue_time_seconds_bucket{engine="0",le="0.5",model_name="gpt-oss-safeguard-20b"} 2.0
421vllm:request_queue_time_seconds_bucket{engine="0",le="0.8",model_name="gpt-oss-safeguard-20b"} 2.0
422vllm:request_queue_time_seconds_bucket{engine="0",le="1.0",model_name="gpt-oss-safeguard-20b"} 2.0
423vllm:request_queue_time_seconds_bucket{engine="0",le="1.5",model_name="gpt-oss-safeguard-20b"} 2.0
424vllm:request_queue_time_seconds_bucket{engine="0",le="2.0",model_name="gpt-oss-safeguard-20b"} 2.0
425vllm:request_queue_time_seconds_bucket{engine="0",le="2.5",model_name="gpt-oss-safeguard-20b"} 2.0
426vllm:request_queue_time_seconds_bucket{engine="0",le="5.0",model_name="gpt-oss-safeguard-20b"} 2.0
427vllm:request_queue_time_seconds_bucket{engine="0",le="10.0",model_name="gpt-oss-safeguard-20b"} 2.0
428vllm:request_queue_time_seconds_bucket{engine="0",le="15.0",model_name="gpt-oss-safeguard-20b"} 2.0
429vllm:request_queue_time_seconds_bucket{engine="0",le="20.0",model_name="gpt-oss-safeguard-20b"} 2.0
430vllm:request_queue_time_seconds_bucket{engine="0",le="30.0",model_name="gpt-oss-safeguard-20b"} 2.0
431vllm:request_queue_time_seconds_bucket{engine="0",le="40.0",model_name="gpt-oss-safeguard-20b"} 2.0
432vllm:request_queue_time_seconds_bucket{engine="0",le="50.0",model_name="gpt-oss-safeguard-20b"} 2.0
433vllm:request_queue_time_seconds_bucket{engine="0",le="60.0",model_name="gpt-oss-safeguard-20b"} 2.0
434vllm:request_queue_time_seconds_bucket{engine="0",le="120.0",model_name="gpt-oss-safeguard-20b"} 2.0
435vllm:request_queue_time_seconds_bucket{engine="0",le="240.0",model_name="gpt-oss-safeguard-20b"} 2.0
436vllm:request_queue_time_seconds_bucket{engine="0",le="480.0",model_name="gpt-oss-safeguard-20b"} 2.0
437vllm:request_queue_time_seconds_bucket{engine="0",le="960.0",model_name="gpt-oss-safeguard-20b"} 2.0
438vllm:request_queue_time_seconds_bucket{engine="0",le="1920.0",model_name="gpt-oss-safeguard-20b"} 2.0
439vllm:request_queue_time_seconds_bucket{engine="0",le="7680.0",model_name="gpt-oss-safeguard-20b"} 2.0
440vllm:request_queue_time_seconds_bucket{engine="0",le="+Inf",model_name="gpt-oss-safeguard-20b"} 2.0
441vllm:request_queue_time_seconds_count{engine="0",model_name="gpt-oss-safeguard-20b"} 2.0
442vllm:request_queue_time_seconds_sum{engine="0",model_name="gpt-oss-safeguard-20b"} 4.593934863805771e-05
443# HELP vllm:request_queue_time_seconds_created Histogram of time spent in WAITING phase for request.
444# TYPE vllm:request_queue_time_seconds_created gauge
445vllm:request_queue_time_seconds_created{engine="0",model_name="gpt-oss-safeguard-20b"} 1.7834197655146308e+09
446# HELP vllm:request_inference_time_seconds Histogram of time spent in RUNNING phase for request.
447# TYPE vllm:request_inference_time_seconds histogram
448vllm:request_inference_time_seconds_bucket{engine="0",le="0.3",model_name="gpt-oss-safeguard-20b"} 0.0
449vllm:request_inference_time_seconds_bucket{engine="0",le="0.5",model_name="gpt-oss-safeguard-20b"} 0.0
450vllm:request_inference_time_seconds_bucket{engine="0",le="0.8",model_name="gpt-oss-safeguard-20b"} 1.0
451vllm:request_inference_time_seconds_bucket{engine="0",le="1.0",model_name="gpt-oss-safeguard-20b"} 1.0
452vllm:request_inference_time_seconds_bucket{engine="0",le="1.5",model_name="gpt-oss-safeguard-20b"} 2.0
453vllm:request_inference_time_seconds_bucket{engine="0",le="2.0",model_name="gpt-oss-safeguard-20b"} 2.0
454vllm:request_inference_time_seconds_bucket{engine="0",le="2.5",model_name="gpt-oss-safeguard-20b"} 2.0
455vllm:request_inference_time_seconds_bucket{engine="0",le="5.0",model_name="gpt-oss-safeguard-20b"} 2.0
456vllm:request_inference_time_seconds_bucket{engine="0",le="10.0",model_name="gpt-oss-safeguard-20b"} 2.0
457vllm:request_inference_time_seconds_bucket{engine="0",le="15.0",model_name="gpt-oss-safeguard-20b"} 2.0
458vllm:request_inference_time_seconds_bucket{engine="0",le="20.0",model_name="gpt-oss-safeguard-20b"} 2.0
459vllm:request_inference_time_seconds_bucket{engine="0",le="30.0",model_name="gpt-oss-safeguard-20b"} 2.0
460vllm:request_inference_time_seconds_bucket{engine="0",le="40.0",model_name="gpt-oss-safeguard-20b"} 2.0
461vllm:request_inference_time_seconds_bucket{engine="0",le="50.0",model_name="gpt-oss-safeguard-20b"} 2.0
462vllm:request_inference_time_seconds_bucket{engine="0",le="60.0",model_name="gpt-oss-safeguard-20b"} 2.0
463vllm:request_inference_time_seconds_bucket{engine="0",le="120.0",model_name="gpt-oss-safeguard-20b"} 2.0
464vllm:request_inference_time_seconds_bucket{engine="0",le="240.0",model_name="gpt-oss-safeguard-20b"} 2.0
465vllm:request_inference_time_seconds_bucket{engine="0",le="480.0",model_name="gpt-oss-safeguard-20b"} 2.0
466vllm:request_inference_time_seconds_bucket{engine="0",le="960.0",model_name="gpt-oss-safeguard-20b"} 2.0
467vllm:request_inference_time_seconds_bucket{engine="0",le="1920.0",model_name="gpt-oss-safeguard-20b"} 2.0
468vllm:request_inference_time_seconds_bucket{engine="0",le="7680.0",model_name="gpt-oss-safeguard-20b"} 2.0
469vllm:request_inference_time_seconds_bucket{engine="0",le="+Inf",model_name="gpt-oss-safeguard-20b"} 2.0
470vllm:request_inference_time_seconds_count{engine="0",model_name="gpt-oss-safeguard-20b"} 2.0
471vllm:request_inference_time_seconds_sum{engine="0",model_name="gpt-oss-safeguard-20b"} 1.60428455658257
472# HELP vllm:request_inference_time_seconds_created Histogram of time spent in RUNNING phase for request.
473# TYPE vllm:request_inference_time_seconds_created gauge
474vllm:request_inference_time_seconds_created{engine="0",model_name="gpt-oss-safeguard-20b"} 1.7834197655146635e+09
475# HELP vllm:request_prefill_time_seconds Histogram of time spent in PREFILL phase for request.
476# TYPE vllm:request_prefill_time_seconds histogram
477vllm:request_prefill_time_seconds_bucket{engine="0",le="0.3",model_name="gpt-oss-safeguard-20b"} 1.0
478vllm:request_prefill_time_seconds_bucket{engine="0",le="0.5",model_name="gpt-oss-safeguard-20b"} 1.0
479vllm:request_prefill_time_seconds_bucket{engine="0",le="0.8",model_name="gpt-oss-safeguard-20b"} 2.0
480vllm:request_prefill_time_seconds_bucket{engine="0",le="1.0",model_name="gpt-oss-safeguard-20b"} 2.0
481vllm:request_prefill_time_seconds_bucket{engine="0",le="1.5",model_name="gpt-oss-safeguard-20b"} 2.0
482vllm:request_prefill_time_seconds_bucket{engine="0",le="2.0",model_name="gpt-oss-safeguard-20b"} 2.0
483vllm:request_prefill_time_seconds_bucket{engine="0",le="2.5",model_name="gpt-oss-safeguard-20b"} 2.0
484vllm:request_prefill_time_seconds_bucket{engine="0",le="5.0",model_name="gpt-oss-safeguard-20b"} 2.0
485vllm:request_prefill_time_seconds_bucket{engine="0",le="10.0",model_name="gpt-oss-safeguard-20b"} 2.0
486vllm:request_prefill_time_seconds_bucket{engine="0",le="15.0",model_name="gpt-oss-safeguard-20b"} 2.0
487vllm:request_prefill_time_seconds_bucket{engine="0",le="20.0",model_name="gpt-oss-safeguard-20b"} 2.0
488vllm:request_prefill_time_seconds_bucket{engine="0",le="30.0",model_name="gpt-oss-safeguard-20b"} 2.0
489vllm:request_prefill_time_seconds_bucket{engine="0",le="40.0",model_name="gpt-oss-safeguard-20b"} 2.0
490vllm:request_prefill_time_seconds_bucket{engine="0",le="50.0",model_name="gpt-oss-safeguard-20b"} 2.0
491vllm:request_prefill_time_seconds_bucket{engine="0",le="60.0",model_name="gpt-oss-safeguard-20b"} 2.0
492vllm:request_prefill_time_seconds_bucket{engine="0",le="120.0",model_name="gpt-oss-safeguard-20b"} 2.0
493vllm:request_prefill_time_seconds_bucket{engine="0",le="240.0",model_name="gpt-oss-safeguard-20b"} 2.0
494vllm:request_prefill_time_seconds_bucket{engine="0",le="480.0",model_name="gpt-oss-safeguard-20b"} 2.0
495vllm:request_prefill_time_seconds_bucket{engine="0",le="960.0",model_name="gpt-oss-safeguard-20b"} 2.0
496vllm:request_prefill_time_seconds_bucket{engine="0",le="1920.0",model_name="gpt-oss-safeguard-20b"} 2.0
497vllm:request_prefill_time_seconds_bucket{engine="0",le="7680.0",model_name="gpt-oss-safeguard-20b"} 2.0
498vllm:request_prefill_time_seconds_bucket{engine="0",le="+Inf",model_name="gpt-oss-safeguard-20b"} 2.0
499vllm:request_prefill_time_seconds_count{engine="0",model_name="gpt-oss-safeguard-20b"} 2.0
500vllm:request_prefill_time_seconds_sum{engine="0",model_name="gpt-oss-safeguard-20b"} 0.6783164599910378
501# HELP vllm:request_prefill_time_seconds_created Histogram of time spent in PREFILL phase for request.
502# TYPE vllm:request_prefill_time_seconds_created gauge
503vllm:request_prefill_time_seconds_created{engine="0",model_name="gpt-oss-safeguard-20b"} 1.7834197655146992e+09
504# HELP vllm:request_decode_time_seconds Histogram of time spent in DECODE phase for request.
505# TYPE vllm:request_decode_time_seconds histogram
506vllm:request_decode_time_seconds_bucket{engine="0",le="0.3",model_name="gpt-oss-safeguard-20b"} 0.0
507vllm:request_decode_time_seconds_bucket{engine="0",le="0.5",model_name="gpt-oss-safeguard-20b"} 2.0
508vllm:request_decode_time_seconds_bucket{engine="0",le="0.8",model_name="gpt-oss-safeguard-20b"} 2.0
509vllm:request_decode_time_seconds_bucket{engine="0",le="1.0",model_name="gpt-oss-safeguard-20b"} 2.0
510vllm:request_decode_time_seconds_bucket{engine="0",le="1.5",model_name="gpt-oss-safeguard-20b"} 2.0
511vllm:request_decode_time_seconds_bucket{engine="0",le="2.0",model_name="gpt-oss-safeguard-20b"} 2.0
512vllm:request_decode_time_seconds_bucket{engine="0",le="2.5",model_name="gpt-oss-safeguard-20b"} 2.0
513vllm:request_decode_time_seconds_bucket{engine="0",le="5.0",model_name="gpt-oss-safeguard-20b"} 2.0
514vllm:request_decode_time_seconds_bucket{engine="0",le="10.0",model_name="gpt-oss-safeguard-20b"} 2.0
515vllm:request_decode_time_seconds_bucket{engine="0",le="15.0",model_name="gpt-oss-safeguard-20b"} 2.0
516vllm:request_decode_time_seconds_bucket{engine="0",le="20.0",model_name="gpt-oss-safeguard-20b"} 2.0
517vllm:request_decode_time_seconds_bucket{engine="0",le="30.0",model_name="gpt-oss-safeguard-20b"} 2.0
518vllm:request_decode_time_seconds_bucket{engine="0",le="40.0",model_name="gpt-oss-safeguard-20b"} 2.0
519vllm:request_decode_time_seconds_bucket{engine="0",le="50.0",model_name="gpt-oss-safeguard-20b"} 2.0
520vllm:request_decode_time_seconds_bucket{engine="0",le="60.0",model_name="gpt-oss-safeguard-20b"} 2.0
521vllm:request_decode_time_seconds_bucket{engine="0",le="120.0",model_name="gpt-oss-safeguard-20b"} 2.0
522vllm:request_decode_time_seconds_bucket{engine="0",le="240.0",model_name="gpt-oss-safeguard-20b"} 2.0
523vllm:request_decode_time_seconds_bucket{engine="0",le="480.0",model_name="gpt-oss-safeguard-20b"} 2.0
524vllm:request_decode_time_seconds_bucket{engine="0",le="960.0",model_name="gpt-oss-safeguard-20b"} 2.0
525vllm:request_decode_time_seconds_bucket{engine="0",le="1920.0",model_name="gpt-oss-safeguard-20b"} 2.0
526vllm:request_decode_time_seconds_bucket{engine="0",le="7680.0",model_name="gpt-oss-safeguard-20b"} 2.0
527vllm:request_decode_time_seconds_bucket{engine="0",le="+Inf",model_name="gpt-oss-safeguard-20b"} 2.0
528vllm:request_decode_time_seconds_count{engine="0",model_name="gpt-oss-safeguard-20b"} 2.0
529vllm:request_decode_time_seconds_sum{engine="0",model_name="gpt-oss-safeguard-20b"} 0.9259680965915322
530# HELP vllm:request_decode_time_seconds_created Histogram of time spent in DECODE phase for request.
531# TYPE vllm:request_decode_time_seconds_created gauge
532vllm:request_decode_time_seconds_created{engine="0",model_name="gpt-oss-safeguard-20b"} 1.7834197655147297e+09
533# HELP vllm:request_prefill_kv_computed_tokens Histogram of new KV tokens computed during prefill (excluding cached tokens).
534# TYPE vllm:request_prefill_kv_computed_tokens histogram
535vllm:request_prefill_kv_computed_tokens_bucket{engine="0",le="1.0",model_name="gpt-oss-safeguard-20b"} 0.0
536vllm:request_prefill_kv_computed_tokens_bucket{engine="0",le="2.0",model_name="gpt-oss-safeguard-20b"} 0.0
537vllm:request_prefill_kv_computed_tokens_bucket{engine="0",le="5.0",model_name="gpt-oss-safeguard-20b"} 0.0
538vllm:request_prefill_kv_computed_tokens_bucket{engine="0",le="10.0",model_name="gpt-oss-safeguard-20b"} 0.0
539vllm:request_prefill_kv_computed_tokens_bucket{engine="0",le="20.0",model_name="gpt-oss-safeguard-20b"} 0.0
540vllm:request_prefill_kv_computed_tokens_bucket{engine="0",le="50.0",model_name="gpt-oss-safeguard-20b"} 0.0
541vllm:request_prefill_kv_computed_tokens_bucket{engine="0",le="100.0",model_name="gpt-oss-safeguard-20b"} 1.0
542vllm:request_prefill_kv_computed_tokens_bucket{engine="0",le="200.0",model_name="gpt-oss-safeguard-20b"} 2.0
543vllm:request_prefill_kv_computed_tokens_bucket{engine="0",le="500.0",model_name="gpt-oss-safeguard-20b"} 2.0
544vllm:request_prefill_kv_computed_tokens_bucket{engine="0",le="1000.0",model_name="gpt-oss-safeguard-20b"} 2.0
545vllm:request_prefill_kv_computed_tokens_bucket{engine="0",le="2000.0",model_name="gpt-oss-safeguard-20b"} 2.0
546vllm:request_prefill_kv_computed_tokens_bucket{engine="0",le="5000.0",model_name="gpt-oss-safeguard-20b"} 2.0
547vllm:request_prefill_kv_computed_tokens_bucket{engine="0",le="10000.0",model_name="gpt-oss-safeguard-20b"} 2.0
548vllm:request_prefill_kv_computed_tokens_bucket{engine="0",le="+Inf",model_name="gpt-oss-safeguard-20b"} 2.0
549vllm:request_prefill_kv_computed_tokens_count{engine="0",model_name="gpt-oss-safeguard-20b"} 2.0
550vllm:request_prefill_kv_computed_tokens_sum{engine="0",model_name="gpt-oss-safeguard-20b"} 256.0
551# HELP vllm:request_prefill_kv_computed_tokens_created Histogram of new KV tokens computed during prefill (excluding cached tokens).
552# TYPE vllm:request_prefill_kv_computed_tokens_created gauge
553vllm:request_prefill_kv_computed_tokens_created{engine="0",model_name="gpt-oss-safeguard-20b"} 1.7834197655147634e+09
554# HELP vllm:cache_config_info Information of the LLMEngine CacheConfig
555# TYPE vllm:cache_config_info gauge
556vllm:cache_config_info{block_size="64",cache_dtype="auto",calculate_kv_scales="False",cpu_kvcache_space_bytes="None",cpu_offload_gb="0.0",enable_prefix_caching="True",engine="0",gpu_memory_utilization="0.95",is_attention_free="False",kv_cache_memory_bytes="None",kv_offloading_backend="native",kv_offloading_size="None",kv_sharing_fast_prefill="False",mamba_block_size="None",mamba_cache_dtype="auto",mamba_cache_mode="none",mamba_page_size_padded="None",mamba_ssm_cache_dtype="auto",num_cpu_blocks="None",num_gpu_blocks="233515",num_gpu_blocks_override="None",prefix_caching_hash_algo="sha256",sliding_window="None",swap_space="4.0"} 1.0

Change #1308051 merged by jenkins-bot:

[operations/deployment-charts@master] ml-services: deploy embeddings isvc image that emits vLLM metrics

https://gerrit.wikimedia.org/r/1308051

The embeddings isvc now emits vLLM metrics as shown below:

1$ curl "https://inference.svc.eqiad.wmnet:30443/metrics" -H "Host: embeddings.llm.wikimedia.org"
2# HELP python_gc_objects_collected_total Objects collected during gc
3# TYPE python_gc_objects_collected_total counter
4python_gc_objects_collected_total{generation="0"} 17351.0
5python_gc_objects_collected_total{generation="1"} 2810.0
6python_gc_objects_collected_total{generation="2"} 123.0
7# HELP python_gc_objects_uncollectable_total Uncollectable objects found during GC
8# TYPE python_gc_objects_uncollectable_total counter
9python_gc_objects_uncollectable_total{generation="0"} 0.0
10python_gc_objects_uncollectable_total{generation="1"} 0.0
11python_gc_objects_uncollectable_total{generation="2"} 0.0
12# HELP python_gc_collections_total Number of times this generation was collected
13# TYPE python_gc_collections_total counter
14python_gc_collections_total{generation="0"} 1883.0
15python_gc_collections_total{generation="1"} 171.0
16python_gc_collections_total{generation="2"} 10.0
17# HELP python_info Python platform information
18# TYPE python_info gauge
19python_info{implementation="CPython",major="3",minor="11",patchlevel="2",version="3.11.2"} 1.0
20# HELP process_virtual_memory_bytes Virtual memory size in bytes.
21# TYPE process_virtual_memory_bytes gauge
22process_virtual_memory_bytes 1.4548189184e+010
23# HELP process_resident_memory_bytes Resident memory size in bytes.
24# TYPE process_resident_memory_bytes gauge
25process_resident_memory_bytes 1.654079488e+09
26# HELP process_start_time_seconds Start time of the process since unix epoch in seconds.
27# TYPE process_start_time_seconds gauge
28process_start_time_seconds 1.7834205618e+09
29# HELP process_cpu_seconds_total Total user and system CPU time spent in seconds.
30# TYPE process_cpu_seconds_total counter
31process_cpu_seconds_total 12.69
32# HELP process_open_fds Number of open file descriptors.
33# TYPE process_open_fds gauge
34process_open_fds 36.0
35# HELP process_max_fds Maximum number of open file descriptors.
36# TYPE process_max_fds gauge
37process_max_fds 1.073741816e+09
38# HELP request_preprocess_seconds pre-process request latency
39# TYPE request_preprocess_seconds histogram
40request_preprocess_seconds_bucket{le="0.005",model_name="qwen3-embedding"} 2.0
41request_preprocess_seconds_bucket{le="0.01",model_name="qwen3-embedding"} 2.0
42request_preprocess_seconds_bucket{le="0.025",model_name="qwen3-embedding"} 2.0
43request_preprocess_seconds_bucket{le="0.05",model_name="qwen3-embedding"} 2.0
44request_preprocess_seconds_bucket{le="0.075",model_name="qwen3-embedding"} 2.0
45request_preprocess_seconds_bucket{le="0.1",model_name="qwen3-embedding"} 2.0
46request_preprocess_seconds_bucket{le="0.25",model_name="qwen3-embedding"} 2.0
47request_preprocess_seconds_bucket{le="0.5",model_name="qwen3-embedding"} 2.0
48request_preprocess_seconds_bucket{le="0.75",model_name="qwen3-embedding"} 2.0
49request_preprocess_seconds_bucket{le="1.0",model_name="qwen3-embedding"} 2.0
50request_preprocess_seconds_bucket{le="2.5",model_name="qwen3-embedding"} 2.0
51request_preprocess_seconds_bucket{le="5.0",model_name="qwen3-embedding"} 2.0
52request_preprocess_seconds_bucket{le="7.5",model_name="qwen3-embedding"} 2.0
53request_preprocess_seconds_bucket{le="10.0",model_name="qwen3-embedding"} 2.0
54request_preprocess_seconds_bucket{le="+Inf",model_name="qwen3-embedding"} 2.0
55request_preprocess_seconds_count{model_name="qwen3-embedding"} 2.0
56request_preprocess_seconds_sum{model_name="qwen3-embedding"} 1.6706064343452454e-05
57# HELP request_preprocess_seconds_created pre-process request latency
58# TYPE request_preprocess_seconds_created gauge
59request_preprocess_seconds_created{model_name="qwen3-embedding"} 1.7834207395563562e+09
60# HELP request_postprocess_seconds post-process request latency
61# TYPE request_postprocess_seconds histogram
62request_postprocess_seconds_bucket{le="0.005",model_name="qwen3-embedding"} 2.0
63request_postprocess_seconds_bucket{le="0.01",model_name="qwen3-embedding"} 2.0
64request_postprocess_seconds_bucket{le="0.025",model_name="qwen3-embedding"} 2.0
65request_postprocess_seconds_bucket{le="0.05",model_name="qwen3-embedding"} 2.0
66request_postprocess_seconds_bucket{le="0.075",model_name="qwen3-embedding"} 2.0
67request_postprocess_seconds_bucket{le="0.1",model_name="qwen3-embedding"} 2.0
68request_postprocess_seconds_bucket{le="0.25",model_name="qwen3-embedding"} 2.0
69request_postprocess_seconds_bucket{le="0.5",model_name="qwen3-embedding"} 2.0
70request_postprocess_seconds_bucket{le="0.75",model_name="qwen3-embedding"} 2.0
71request_postprocess_seconds_bucket{le="1.0",model_name="qwen3-embedding"} 2.0
72request_postprocess_seconds_bucket{le="2.5",model_name="qwen3-embedding"} 2.0
73request_postprocess_seconds_bucket{le="5.0",model_name="qwen3-embedding"} 2.0
74request_postprocess_seconds_bucket{le="7.5",model_name="qwen3-embedding"} 2.0
75request_postprocess_seconds_bucket{le="10.0",model_name="qwen3-embedding"} 2.0
76request_postprocess_seconds_bucket{le="+Inf",model_name="qwen3-embedding"} 2.0
77request_postprocess_seconds_count{model_name="qwen3-embedding"} 2.0
78request_postprocess_seconds_sum{model_name="qwen3-embedding"} 2.2963620722293854e-05
79# HELP request_postprocess_seconds_created post-process request latency
80# TYPE request_postprocess_seconds_created gauge
81request_postprocess_seconds_created{model_name="qwen3-embedding"} 1.7834207399139264e+09
82# HELP request_predict_seconds predict request latency
83# TYPE request_predict_seconds histogram
84request_predict_seconds_bucket{le="0.005",model_name="qwen3-embedding"} 0.0
85request_predict_seconds_bucket{le="0.01",model_name="qwen3-embedding"} 0.0
86request_predict_seconds_bucket{le="0.025",model_name="qwen3-embedding"} 0.0
87request_predict_seconds_bucket{le="0.05",model_name="qwen3-embedding"} 0.0
88request_predict_seconds_bucket{le="0.075",model_name="qwen3-embedding"} 0.0
89request_predict_seconds_bucket{le="0.1",model_name="qwen3-embedding"} 0.0
90request_predict_seconds_bucket{le="0.25",model_name="qwen3-embedding"} 0.0
91request_predict_seconds_bucket{le="0.5",model_name="qwen3-embedding"} 2.0
92request_predict_seconds_bucket{le="0.75",model_name="qwen3-embedding"} 2.0
93request_predict_seconds_bucket{le="1.0",model_name="qwen3-embedding"} 2.0
94request_predict_seconds_bucket{le="2.5",model_name="qwen3-embedding"} 2.0
95request_predict_seconds_bucket{le="5.0",model_name="qwen3-embedding"} 2.0
96request_predict_seconds_bucket{le="7.5",model_name="qwen3-embedding"} 2.0
97request_predict_seconds_bucket{le="10.0",model_name="qwen3-embedding"} 2.0
98request_predict_seconds_bucket{le="+Inf",model_name="qwen3-embedding"} 2.0
99request_predict_seconds_count{model_name="qwen3-embedding"} 2.0
100request_predict_seconds_sum{model_name="qwen3-embedding"} 0.6459501725621521
101# HELP request_predict_seconds_created predict request latency
102# TYPE request_predict_seconds_created gauge
103request_predict_seconds_created{model_name="qwen3-embedding"} 1.7834207395564275e+09
104# HELP request_explain_seconds explain request latency
105# TYPE request_explain_seconds histogram
106# HELP vllm:num_requests_running Number of requests in model execution batches.
107# TYPE vllm:num_requests_running gauge
108vllm:num_requests_running{engine="0",model_name="qwen3-embedding"} 0.0
109# HELP vllm:num_requests_waiting Number of requests waiting to be processed.
110# TYPE vllm:num_requests_waiting gauge
111vllm:num_requests_waiting{engine="0",model_name="qwen3-embedding"} 0.0
112# HELP vllm:engine_sleep_state Engine sleep state; awake = 0 means engine is sleeping; awake = 1 means engine is awake; weights_offloaded = 1 means sleep level 1; discard_all = 1 means sleep level 2.
113# TYPE vllm:engine_sleep_state gauge
114vllm:engine_sleep_state{engine="0",model_name="qwen3-embedding",sleep_state="awake"} 1.0
115vllm:engine_sleep_state{engine="0",model_name="qwen3-embedding",sleep_state="weights_offloaded"} 0.0
116vllm:engine_sleep_state{engine="0",model_name="qwen3-embedding",sleep_state="discard_all"} 0.0
117# HELP vllm:kv_cache_usage_perc KV-cache usage. 1 means 100 percent usage.
118# TYPE vllm:kv_cache_usage_perc gauge
119vllm:kv_cache_usage_perc{engine="0",model_name="qwen3-embedding"} 0.0
120# HELP vllm:prefix_cache_queries_total Prefix cache queries, in terms of number of queried tokens.
121# TYPE vllm:prefix_cache_queries_total counter
122vllm:prefix_cache_queries_total{engine="0",model_name="qwen3-embedding"} 0.0
123# HELP vllm:prefix_cache_queries_created Prefix cache queries, in terms of number of queried tokens.
124# TYPE vllm:prefix_cache_queries_created gauge
125vllm:prefix_cache_queries_created{engine="0",model_name="qwen3-embedding"} 1.783420629586751e+09
126# HELP vllm:prefix_cache_hits_total Prefix cache hits, in terms of number of cached tokens.
127# TYPE vllm:prefix_cache_hits_total counter
128vllm:prefix_cache_hits_total{engine="0",model_name="qwen3-embedding"} 0.0
129# HELP vllm:prefix_cache_hits_created Prefix cache hits, in terms of number of cached tokens.
130# TYPE vllm:prefix_cache_hits_created gauge
131vllm:prefix_cache_hits_created{engine="0",model_name="qwen3-embedding"} 1.7834206295867686e+09
132# HELP vllm:external_prefix_cache_queries_total External prefix cache queries from KV connector cross-instance cache sharing, in terms of number of queried tokens.
133# TYPE vllm:external_prefix_cache_queries_total counter
134vllm:external_prefix_cache_queries_total{engine="0",model_name="qwen3-embedding"} 0.0
135# HELP vllm:external_prefix_cache_queries_created External prefix cache queries from KV connector cross-instance cache sharing, in terms of number of queried tokens.
136# TYPE vllm:external_prefix_cache_queries_created gauge
137vllm:external_prefix_cache_queries_created{engine="0",model_name="qwen3-embedding"} 1.7834206295867817e+09
138# HELP vllm:external_prefix_cache_hits_total External prefix cache hits from KV connector cross-instance cache sharing, in terms of number of cached tokens.
139# TYPE vllm:external_prefix_cache_hits_total counter
140vllm:external_prefix_cache_hits_total{engine="0",model_name="qwen3-embedding"} 0.0
141# HELP vllm:external_prefix_cache_hits_created External prefix cache hits from KV connector cross-instance cache sharing, in terms of number of cached tokens.
142# TYPE vllm:external_prefix_cache_hits_created gauge
143vllm:external_prefix_cache_hits_created{engine="0",model_name="qwen3-embedding"} 1.783420629586797e+09
144# HELP vllm:mm_cache_queries_total Multi-modal cache queries, in terms of number of queried items.
145# TYPE vllm:mm_cache_queries_total counter
146vllm:mm_cache_queries_total{engine="0",model_name="qwen3-embedding"} 0.0
147# HELP vllm:mm_cache_queries_created Multi-modal cache queries, in terms of number of queried items.
148# TYPE vllm:mm_cache_queries_created gauge
149vllm:mm_cache_queries_created{engine="0",model_name="qwen3-embedding"} 1.7834206295868104e+09
150# HELP vllm:mm_cache_hits_total Multi-modal cache hits, in terms of number of cached items.
151# TYPE vllm:mm_cache_hits_total counter
152vllm:mm_cache_hits_total{engine="0",model_name="qwen3-embedding"} 0.0
153# HELP vllm:mm_cache_hits_created Multi-modal cache hits, in terms of number of cached items.
154# TYPE vllm:mm_cache_hits_created gauge
155vllm:mm_cache_hits_created{engine="0",model_name="qwen3-embedding"} 1.783420629586821e+09
156# HELP vllm:num_preemptions_total Cumulative number of preemption from the engine.
157# TYPE vllm:num_preemptions_total counter
158vllm:num_preemptions_total{engine="0",model_name="qwen3-embedding"} 0.0
159# HELP vllm:num_preemptions_created Cumulative number of preemption from the engine.
160# TYPE vllm:num_preemptions_created gauge
161vllm:num_preemptions_created{engine="0",model_name="qwen3-embedding"} 1.7834206295868342e+09
162# HELP vllm:prompt_tokens_total Number of prefill tokens processed.
163# TYPE vllm:prompt_tokens_total counter
164vllm:prompt_tokens_total{engine="0",model_name="qwen3-embedding"} 12.0
165# HELP vllm:prompt_tokens_created Number of prefill tokens processed.
166# TYPE vllm:prompt_tokens_created gauge
167vllm:prompt_tokens_created{engine="0",model_name="qwen3-embedding"} 1.7834206295868502e+09
168# HELP vllm:generation_tokens_total Number of generation tokens processed.
169# TYPE vllm:generation_tokens_total counter
170vllm:generation_tokens_total{engine="0",model_name="qwen3-embedding"} 0.0
171# HELP vllm:generation_tokens_created Number of generation tokens processed.
172# TYPE vllm:generation_tokens_created gauge
173vllm:generation_tokens_created{engine="0",model_name="qwen3-embedding"} 1.7834206295868626e+09
174# HELP vllm:request_success_total Count of successfully processed requests.
175# TYPE vllm:request_success_total counter
176vllm:request_success_total{engine="0",finished_reason="stop",model_name="qwen3-embedding"} 4.0
177vllm:request_success_total{engine="0",finished_reason="length",model_name="qwen3-embedding"} 0.0
178vllm:request_success_total{engine="0",finished_reason="abort",model_name="qwen3-embedding"} 0.0
179vllm:request_success_total{engine="0",finished_reason="error",model_name="qwen3-embedding"} 0.0
180# HELP vllm:request_success_created Count of successfully processed requests.
181# TYPE vllm:request_success_created gauge
182vllm:request_success_created{engine="0",finished_reason="stop",model_name="qwen3-embedding"} 1.7834206295868852e+09
183vllm:request_success_created{engine="0",finished_reason="length",model_name="qwen3-embedding"} 1.783420629586892e+09
184vllm:request_success_created{engine="0",finished_reason="abort",model_name="qwen3-embedding"} 1.7834206295868979e+09
185vllm:request_success_created{engine="0",finished_reason="error",model_name="qwen3-embedding"} 1.783420629586904e+09
186# HELP vllm:request_prompt_tokens Number of prefill tokens processed.
187# TYPE vllm:request_prompt_tokens histogram
188vllm:request_prompt_tokens_bucket{engine="0",le="1.0",model_name="qwen3-embedding"} 0.0
189vllm:request_prompt_tokens_bucket{engine="0",le="2.0",model_name="qwen3-embedding"} 0.0
190vllm:request_prompt_tokens_bucket{engine="0",le="5.0",model_name="qwen3-embedding"} 4.0
191vllm:request_prompt_tokens_bucket{engine="0",le="10.0",model_name="qwen3-embedding"} 4.0
192vllm:request_prompt_tokens_bucket{engine="0",le="20.0",model_name="qwen3-embedding"} 4.0
193vllm:request_prompt_tokens_bucket{engine="0",le="50.0",model_name="qwen3-embedding"} 4.0
194vllm:request_prompt_tokens_bucket{engine="0",le="100.0",model_name="qwen3-embedding"} 4.0
195vllm:request_prompt_tokens_bucket{engine="0",le="200.0",model_name="qwen3-embedding"} 4.0
196vllm:request_prompt_tokens_bucket{engine="0",le="500.0",model_name="qwen3-embedding"} 4.0
197vllm:request_prompt_tokens_bucket{engine="0",le="1000.0",model_name="qwen3-embedding"} 4.0
198vllm:request_prompt_tokens_bucket{engine="0",le="2000.0",model_name="qwen3-embedding"} 4.0
199vllm:request_prompt_tokens_bucket{engine="0",le="5000.0",model_name="qwen3-embedding"} 4.0
200vllm:request_prompt_tokens_bucket{engine="0",le="+Inf",model_name="qwen3-embedding"} 4.0
201vllm:request_prompt_tokens_count{engine="0",model_name="qwen3-embedding"} 4.0
202vllm:request_prompt_tokens_sum{engine="0",model_name="qwen3-embedding"} 12.0
203# HELP vllm:request_prompt_tokens_created Number of prefill tokens processed.
204# TYPE vllm:request_prompt_tokens_created gauge
205vllm:request_prompt_tokens_created{engine="0",model_name="qwen3-embedding"} 1.7834206295869386e+09
206# HELP vllm:request_generation_tokens Number of generation tokens processed.
207# TYPE vllm:request_generation_tokens histogram
208vllm:request_generation_tokens_bucket{engine="0",le="1.0",model_name="qwen3-embedding"} 4.0
209vllm:request_generation_tokens_bucket{engine="0",le="2.0",model_name="qwen3-embedding"} 4.0
210vllm:request_generation_tokens_bucket{engine="0",le="5.0",model_name="qwen3-embedding"} 4.0
211vllm:request_generation_tokens_bucket{engine="0",le="10.0",model_name="qwen3-embedding"} 4.0
212vllm:request_generation_tokens_bucket{engine="0",le="20.0",model_name="qwen3-embedding"} 4.0
213vllm:request_generation_tokens_bucket{engine="0",le="50.0",model_name="qwen3-embedding"} 4.0
214vllm:request_generation_tokens_bucket{engine="0",le="100.0",model_name="qwen3-embedding"} 4.0
215vllm:request_generation_tokens_bucket{engine="0",le="200.0",model_name="qwen3-embedding"} 4.0
216vllm:request_generation_tokens_bucket{engine="0",le="500.0",model_name="qwen3-embedding"} 4.0
217vllm:request_generation_tokens_bucket{engine="0",le="1000.0",model_name="qwen3-embedding"} 4.0
218vllm:request_generation_tokens_bucket{engine="0",le="2000.0",model_name="qwen3-embedding"} 4.0
219vllm:request_generation_tokens_bucket{engine="0",le="5000.0",model_name="qwen3-embedding"} 4.0
220vllm:request_generation_tokens_bucket{engine="0",le="+Inf",model_name="qwen3-embedding"} 4.0
221vllm:request_generation_tokens_count{engine="0",model_name="qwen3-embedding"} 4.0
222vllm:request_generation_tokens_sum{engine="0",model_name="qwen3-embedding"} 0.0
223# HELP vllm:request_generation_tokens_created Number of generation tokens processed.
224# TYPE vllm:request_generation_tokens_created gauge
225vllm:request_generation_tokens_created{engine="0",model_name="qwen3-embedding"} 1.7834206295869856e+09
226# HELP vllm:iteration_tokens_total Histogram of number of tokens per engine_step.
227# TYPE vllm:iteration_tokens_total histogram
228vllm:iteration_tokens_total_bucket{engine="0",le="1.0",model_name="qwen3-embedding"} 2.0
229vllm:iteration_tokens_total_bucket{engine="0",le="8.0",model_name="qwen3-embedding"} 5.0
230vllm:iteration_tokens_total_bucket{engine="0",le="16.0",model_name="qwen3-embedding"} 5.0
231vllm:iteration_tokens_total_bucket{engine="0",le="32.0",model_name="qwen3-embedding"} 5.0
232vllm:iteration_tokens_total_bucket{engine="0",le="64.0",model_name="qwen3-embedding"} 5.0
233vllm:iteration_tokens_total_bucket{engine="0",le="128.0",model_name="qwen3-embedding"} 5.0
234vllm:iteration_tokens_total_bucket{engine="0",le="256.0",model_name="qwen3-embedding"} 5.0
235vllm:iteration_tokens_total_bucket{engine="0",le="512.0",model_name="qwen3-embedding"} 5.0
236vllm:iteration_tokens_total_bucket{engine="0",le="1024.0",model_name="qwen3-embedding"} 5.0
237vllm:iteration_tokens_total_bucket{engine="0",le="2048.0",model_name="qwen3-embedding"} 5.0
238vllm:iteration_tokens_total_bucket{engine="0",le="4096.0",model_name="qwen3-embedding"} 5.0
239vllm:iteration_tokens_total_bucket{engine="0",le="8192.0",model_name="qwen3-embedding"} 5.0
240vllm:iteration_tokens_total_bucket{engine="0",le="16384.0",model_name="qwen3-embedding"} 5.0
241vllm:iteration_tokens_total_bucket{engine="0",le="+Inf",model_name="qwen3-embedding"} 5.0
242vllm:iteration_tokens_total_count{engine="0",model_name="qwen3-embedding"} 5.0
243vllm:iteration_tokens_total_sum{engine="0",model_name="qwen3-embedding"} 12.0
244# HELP vllm:iteration_tokens_total_created Histogram of number of tokens per engine_step.
245# TYPE vllm:iteration_tokens_total_created gauge
246vllm:iteration_tokens_total_created{engine="0",model_name="qwen3-embedding"} 1.7834206295870194e+09
247# HELP vllm:request_max_num_generation_tokens Histogram of maximum number of requested generation tokens.
248# TYPE vllm:request_max_num_generation_tokens histogram
249vllm:request_max_num_generation_tokens_bucket{engine="0",le="1.0",model_name="qwen3-embedding"} 4.0
250vllm:request_max_num_generation_tokens_bucket{engine="0",le="2.0",model_name="qwen3-embedding"} 4.0
251vllm:request_max_num_generation_tokens_bucket{engine="0",le="5.0",model_name="qwen3-embedding"} 4.0
252vllm:request_max_num_generation_tokens_bucket{engine="0",le="10.0",model_name="qwen3-embedding"} 4.0
253vllm:request_max_num_generation_tokens_bucket{engine="0",le="20.0",model_name="qwen3-embedding"} 4.0
254vllm:request_max_num_generation_tokens_bucket{engine="0",le="50.0",model_name="qwen3-embedding"} 4.0
255vllm:request_max_num_generation_tokens_bucket{engine="0",le="100.0",model_name="qwen3-embedding"} 4.0
256vllm:request_max_num_generation_tokens_bucket{engine="0",le="200.0",model_name="qwen3-embedding"} 4.0
257vllm:request_max_num_generation_tokens_bucket{engine="0",le="500.0",model_name="qwen3-embedding"} 4.0
258vllm:request_max_num_generation_tokens_bucket{engine="0",le="1000.0",model_name="qwen3-embedding"} 4.0
259vllm:request_max_num_generation_tokens_bucket{engine="0",le="2000.0",model_name="qwen3-embedding"} 4.0
260vllm:request_max_num_generation_tokens_bucket{engine="0",le="5000.0",model_name="qwen3-embedding"} 4.0
261vllm:request_max_num_generation_tokens_bucket{engine="0",le="+Inf",model_name="qwen3-embedding"} 4.0
262vllm:request_max_num_generation_tokens_count{engine="0",model_name="qwen3-embedding"} 4.0
263vllm:request_max_num_generation_tokens_sum{engine="0",model_name="qwen3-embedding"} 0.0
264# HELP vllm:request_max_num_generation_tokens_created Histogram of maximum number of requested generation tokens.
265# TYPE vllm:request_max_num_generation_tokens_created gauge
266vllm:request_max_num_generation_tokens_created{engine="0",model_name="qwen3-embedding"} 1.7834206295870492e+09
267# HELP vllm:request_params_n Histogram of the n request parameter.
268# TYPE vllm:request_params_n histogram
269vllm:request_params_n_bucket{engine="0",le="1.0",model_name="qwen3-embedding"} 4.0
270vllm:request_params_n_bucket{engine="0",le="2.0",model_name="qwen3-embedding"} 4.0
271vllm:request_params_n_bucket{engine="0",le="5.0",model_name="qwen3-embedding"} 4.0
272vllm:request_params_n_bucket{engine="0",le="10.0",model_name="qwen3-embedding"} 4.0
273vllm:request_params_n_bucket{engine="0",le="20.0",model_name="qwen3-embedding"} 4.0
274vllm:request_params_n_bucket{engine="0",le="+Inf",model_name="qwen3-embedding"} 4.0
275vllm:request_params_n_count{engine="0",model_name="qwen3-embedding"} 4.0
276vllm:request_params_n_sum{engine="0",model_name="qwen3-embedding"} 4.0
277# HELP vllm:request_params_n_created Histogram of the n request parameter.
278# TYPE vllm:request_params_n_created gauge
279vllm:request_params_n_created{engine="0",model_name="qwen3-embedding"} 1.7834206295870776e+09
280# HELP vllm:request_params_max_tokens Histogram of the max_tokens request parameter.
281# TYPE vllm:request_params_max_tokens histogram
282vllm:request_params_max_tokens_bucket{engine="0",le="1.0",model_name="qwen3-embedding"} 0.0
283vllm:request_params_max_tokens_bucket{engine="0",le="2.0",model_name="qwen3-embedding"} 0.0
284vllm:request_params_max_tokens_bucket{engine="0",le="5.0",model_name="qwen3-embedding"} 0.0
285vllm:request_params_max_tokens_bucket{engine="0",le="10.0",model_name="qwen3-embedding"} 0.0
286vllm:request_params_max_tokens_bucket{engine="0",le="20.0",model_name="qwen3-embedding"} 0.0
287vllm:request_params_max_tokens_bucket{engine="0",le="50.0",model_name="qwen3-embedding"} 0.0
288vllm:request_params_max_tokens_bucket{engine="0",le="100.0",model_name="qwen3-embedding"} 0.0
289vllm:request_params_max_tokens_bucket{engine="0",le="200.0",model_name="qwen3-embedding"} 0.0
290vllm:request_params_max_tokens_bucket{engine="0",le="500.0",model_name="qwen3-embedding"} 0.0
291vllm:request_params_max_tokens_bucket{engine="0",le="1000.0",model_name="qwen3-embedding"} 0.0
292vllm:request_params_max_tokens_bucket{engine="0",le="2000.0",model_name="qwen3-embedding"} 0.0
293vllm:request_params_max_tokens_bucket{engine="0",le="5000.0",model_name="qwen3-embedding"} 0.0
294vllm:request_params_max_tokens_bucket{engine="0",le="+Inf",model_name="qwen3-embedding"} 0.0
295vllm:request_params_max_tokens_count{engine="0",model_name="qwen3-embedding"} 0.0
296vllm:request_params_max_tokens_sum{engine="0",model_name="qwen3-embedding"} 0.0
297# HELP vllm:request_params_max_tokens_created Histogram of the max_tokens request parameter.
298# TYPE vllm:request_params_max_tokens_created gauge
299vllm:request_params_max_tokens_created{engine="0",model_name="qwen3-embedding"} 1.7834206295871005e+09
300# HELP vllm:time_to_first_token_seconds Histogram of time to first token in seconds.
301# TYPE vllm:time_to_first_token_seconds histogram
302vllm:time_to_first_token_seconds_bucket{engine="0",le="0.001",model_name="qwen3-embedding"} 0.0
303vllm:time_to_first_token_seconds_bucket{engine="0",le="0.005",model_name="qwen3-embedding"} 0.0
304vllm:time_to_first_token_seconds_bucket{engine="0",le="0.01",model_name="qwen3-embedding"} 0.0
305vllm:time_to_first_token_seconds_bucket{engine="0",le="0.02",model_name="qwen3-embedding"} 0.0
306vllm:time_to_first_token_seconds_bucket{engine="0",le="0.04",model_name="qwen3-embedding"} 0.0
307vllm:time_to_first_token_seconds_bucket{engine="0",le="0.06",model_name="qwen3-embedding"} 0.0
308vllm:time_to_first_token_seconds_bucket{engine="0",le="0.08",model_name="qwen3-embedding"} 0.0
309vllm:time_to_first_token_seconds_bucket{engine="0",le="0.1",model_name="qwen3-embedding"} 0.0
310vllm:time_to_first_token_seconds_bucket{engine="0",le="0.25",model_name="qwen3-embedding"} 0.0
311vllm:time_to_first_token_seconds_bucket{engine="0",le="0.5",model_name="qwen3-embedding"} 4.0
312vllm:time_to_first_token_seconds_bucket{engine="0",le="0.75",model_name="qwen3-embedding"} 4.0
313vllm:time_to_first_token_seconds_bucket{engine="0",le="1.0",model_name="qwen3-embedding"} 4.0
314vllm:time_to_first_token_seconds_bucket{engine="0",le="2.5",model_name="qwen3-embedding"} 4.0
315vllm:time_to_first_token_seconds_bucket{engine="0",le="5.0",model_name="qwen3-embedding"} 4.0
316vllm:time_to_first_token_seconds_bucket{engine="0",le="7.5",model_name="qwen3-embedding"} 4.0
317vllm:time_to_first_token_seconds_bucket{engine="0",le="10.0",model_name="qwen3-embedding"} 4.0
318vllm:time_to_first_token_seconds_bucket{engine="0",le="20.0",model_name="qwen3-embedding"} 4.0
319vllm:time_to_first_token_seconds_bucket{engine="0",le="40.0",model_name="qwen3-embedding"} 4.0
320vllm:time_to_first_token_seconds_bucket{engine="0",le="80.0",model_name="qwen3-embedding"} 4.0
321vllm:time_to_first_token_seconds_bucket{engine="0",le="160.0",model_name="qwen3-embedding"} 4.0
322vllm:time_to_first_token_seconds_bucket{engine="0",le="640.0",model_name="qwen3-embedding"} 4.0
323vllm:time_to_first_token_seconds_bucket{engine="0",le="2560.0",model_name="qwen3-embedding"} 4.0
324vllm:time_to_first_token_seconds_bucket{engine="0",le="+Inf",model_name="qwen3-embedding"} 4.0
325vllm:time_to_first_token_seconds_count{engine="0",model_name="qwen3-embedding"} 4.0
326vllm:time_to_first_token_seconds_sum{engine="0",model_name="qwen3-embedding"} 1.2689967155456543
327# HELP vllm:time_to_first_token_seconds_created Histogram of time to first token in seconds.
328# TYPE vllm:time_to_first_token_seconds_created gauge
329vllm:time_to_first_token_seconds_created{engine="0",model_name="qwen3-embedding"} 1.783420629587128e+09
330# HELP vllm:inter_token_latency_seconds Histogram of inter-token latency in seconds.
331# TYPE vllm:inter_token_latency_seconds histogram
332vllm:inter_token_latency_seconds_bucket{engine="0",le="0.01",model_name="qwen3-embedding"} 0.0
333vllm:inter_token_latency_seconds_bucket{engine="0",le="0.025",model_name="qwen3-embedding"} 0.0
334vllm:inter_token_latency_seconds_bucket{engine="0",le="0.05",model_name="qwen3-embedding"} 0.0
335vllm:inter_token_latency_seconds_bucket{engine="0",le="0.075",model_name="qwen3-embedding"} 0.0
336vllm:inter_token_latency_seconds_bucket{engine="0",le="0.1",model_name="qwen3-embedding"} 0.0
337vllm:inter_token_latency_seconds_bucket{engine="0",le="0.15",model_name="qwen3-embedding"} 0.0
338vllm:inter_token_latency_seconds_bucket{engine="0",le="0.2",model_name="qwen3-embedding"} 0.0
339vllm:inter_token_latency_seconds_bucket{engine="0",le="0.3",model_name="qwen3-embedding"} 0.0
340vllm:inter_token_latency_seconds_bucket{engine="0",le="0.4",model_name="qwen3-embedding"} 0.0
341vllm:inter_token_latency_seconds_bucket{engine="0",le="0.5",model_name="qwen3-embedding"} 0.0
342vllm:inter_token_latency_seconds_bucket{engine="0",le="0.75",model_name="qwen3-embedding"} 0.0
343vllm:inter_token_latency_seconds_bucket{engine="0",le="1.0",model_name="qwen3-embedding"} 0.0
344vllm:inter_token_latency_seconds_bucket{engine="0",le="2.5",model_name="qwen3-embedding"} 0.0
345vllm:inter_token_latency_seconds_bucket{engine="0",le="5.0",model_name="qwen3-embedding"} 0.0
346vllm:inter_token_latency_seconds_bucket{engine="0",le="7.5",model_name="qwen3-embedding"} 0.0
347vllm:inter_token_latency_seconds_bucket{engine="0",le="10.0",model_name="qwen3-embedding"} 0.0
348vllm:inter_token_latency_seconds_bucket{engine="0",le="20.0",model_name="qwen3-embedding"} 0.0
349vllm:inter_token_latency_seconds_bucket{engine="0",le="40.0",model_name="qwen3-embedding"} 0.0
350vllm:inter_token_latency_seconds_bucket{engine="0",le="80.0",model_name="qwen3-embedding"} 0.0
351vllm:inter_token_latency_seconds_bucket{engine="0",le="+Inf",model_name="qwen3-embedding"} 0.0
352vllm:inter_token_latency_seconds_count{engine="0",model_name="qwen3-embedding"} 0.0
353vllm:inter_token_latency_seconds_sum{engine="0",model_name="qwen3-embedding"} 0.0
354# HELP vllm:inter_token_latency_seconds_created Histogram of inter-token latency in seconds.
355# TYPE vllm:inter_token_latency_seconds_created gauge
356vllm:inter_token_latency_seconds_created{engine="0",model_name="qwen3-embedding"} 1.7834206295871673e+09
357# HELP vllm:request_time_per_output_token_seconds Histogram of time_per_output_token_seconds per request.
358# TYPE vllm:request_time_per_output_token_seconds histogram
359vllm:request_time_per_output_token_seconds_bucket{engine="0",le="0.01",model_name="qwen3-embedding"} 4.0
360vllm:request_time_per_output_token_seconds_bucket{engine="0",le="0.025",model_name="qwen3-embedding"} 4.0
361vllm:request_time_per_output_token_seconds_bucket{engine="0",le="0.05",model_name="qwen3-embedding"} 4.0
362vllm:request_time_per_output_token_seconds_bucket{engine="0",le="0.075",model_name="qwen3-embedding"} 4.0
363vllm:request_time_per_output_token_seconds_bucket{engine="0",le="0.1",model_name="qwen3-embedding"} 4.0
364vllm:request_time_per_output_token_seconds_bucket{engine="0",le="0.15",model_name="qwen3-embedding"} 4.0
365vllm:request_time_per_output_token_seconds_bucket{engine="0",le="0.2",model_name="qwen3-embedding"} 4.0
366vllm:request_time_per_output_token_seconds_bucket{engine="0",le="0.3",model_name="qwen3-embedding"} 4.0
367vllm:request_time_per_output_token_seconds_bucket{engine="0",le="0.4",model_name="qwen3-embedding"} 4.0
368vllm:request_time_per_output_token_seconds_bucket{engine="0",le="0.5",model_name="qwen3-embedding"} 4.0
369vllm:request_time_per_output_token_seconds_bucket{engine="0",le="0.75",model_name="qwen3-embedding"} 4.0
370vllm:request_time_per_output_token_seconds_bucket{engine="0",le="1.0",model_name="qwen3-embedding"} 4.0
371vllm:request_time_per_output_token_seconds_bucket{engine="0",le="2.5",model_name="qwen3-embedding"} 4.0
372vllm:request_time_per_output_token_seconds_bucket{engine="0",le="5.0",model_name="qwen3-embedding"} 4.0
373vllm:request_time_per_output_token_seconds_bucket{engine="0",le="7.5",model_name="qwen3-embedding"} 4.0
374vllm:request_time_per_output_token_seconds_bucket{engine="0",le="10.0",model_name="qwen3-embedding"} 4.0
375vllm:request_time_per_output_token_seconds_bucket{engine="0",le="20.0",model_name="qwen3-embedding"} 4.0
376vllm:request_time_per_output_token_seconds_bucket{engine="0",le="40.0",model_name="qwen3-embedding"} 4.0
377vllm:request_time_per_output_token_seconds_bucket{engine="0",le="80.0",model_name="qwen3-embedding"} 4.0
378vllm:request_time_per_output_token_seconds_bucket{engine="0",le="+Inf",model_name="qwen3-embedding"} 4.0
379vllm:request_time_per_output_token_seconds_count{engine="0",model_name="qwen3-embedding"} 4.0
380vllm:request_time_per_output_token_seconds_sum{engine="0",model_name="qwen3-embedding"} 0.0
381# HELP vllm:request_time_per_output_token_seconds_created Histogram of time_per_output_token_seconds per request.
382# TYPE vllm:request_time_per_output_token_seconds_created gauge
383vllm:request_time_per_output_token_seconds_created{engine="0",model_name="qwen3-embedding"} 1.7834206295872009e+09
384# HELP vllm:e2e_request_latency_seconds Histogram of e2e request latency in seconds.
385# TYPE vllm:e2e_request_latency_seconds histogram
386vllm:e2e_request_latency_seconds_bucket{engine="0",le="0.3",model_name="qwen3-embedding"} 2.0
387vllm:e2e_request_latency_seconds_bucket{engine="0",le="0.5",model_name="qwen3-embedding"} 4.0
388vllm:e2e_request_latency_seconds_bucket{engine="0",le="0.8",model_name="qwen3-embedding"} 4.0
389vllm:e2e_request_latency_seconds_bucket{engine="0",le="1.0",model_name="qwen3-embedding"} 4.0
390vllm:e2e_request_latency_seconds_bucket{engine="0",le="1.5",model_name="qwen3-embedding"} 4.0
391vllm:e2e_request_latency_seconds_bucket{engine="0",le="2.0",model_name="qwen3-embedding"} 4.0
392vllm:e2e_request_latency_seconds_bucket{engine="0",le="2.5",model_name="qwen3-embedding"} 4.0
393vllm:e2e_request_latency_seconds_bucket{engine="0",le="5.0",model_name="qwen3-embedding"} 4.0
394vllm:e2e_request_latency_seconds_bucket{engine="0",le="10.0",model_name="qwen3-embedding"} 4.0
395vllm:e2e_request_latency_seconds_bucket{engine="0",le="15.0",model_name="qwen3-embedding"} 4.0
396vllm:e2e_request_latency_seconds_bucket{engine="0",le="20.0",model_name="qwen3-embedding"} 4.0
397vllm:e2e_request_latency_seconds_bucket{engine="0",le="30.0",model_name="qwen3-embedding"} 4.0
398vllm:e2e_request_latency_seconds_bucket{engine="0",le="40.0",model_name="qwen3-embedding"} 4.0
399vllm:e2e_request_latency_seconds_bucket{engine="0",le="50.0",model_name="qwen3-embedding"} 4.0
400vllm:e2e_request_latency_seconds_bucket{engine="0",le="60.0",model_name="qwen3-embedding"} 4.0
401vllm:e2e_request_latency_seconds_bucket{engine="0",le="120.0",model_name="qwen3-embedding"} 4.0
402vllm:e2e_request_latency_seconds_bucket{engine="0",le="240.0",model_name="qwen3-embedding"} 4.0
403vllm:e2e_request_latency_seconds_bucket{engine="0",le="480.0",model_name="qwen3-embedding"} 4.0
404vllm:e2e_request_latency_seconds_bucket{engine="0",le="960.0",model_name="qwen3-embedding"} 4.0
405vllm:e2e_request_latency_seconds_bucket{engine="0",le="1920.0",model_name="qwen3-embedding"} 4.0
406vllm:e2e_request_latency_seconds_bucket{engine="0",le="7680.0",model_name="qwen3-embedding"} 4.0
407vllm:e2e_request_latency_seconds_bucket{engine="0",le="+Inf",model_name="qwen3-embedding"} 4.0
408vllm:e2e_request_latency_seconds_count{engine="0",model_name="qwen3-embedding"} 4.0
409vllm:e2e_request_latency_seconds_sum{engine="0",model_name="qwen3-embedding"} 1.2689967155456543
410# HELP vllm:e2e_request_latency_seconds_created Histogram of e2e request latency in seconds.
411# TYPE vllm:e2e_request_latency_seconds_created gauge
412vllm:e2e_request_latency_seconds_created{engine="0",model_name="qwen3-embedding"} 1.7834206295872338e+09
413# HELP vllm:request_queue_time_seconds Histogram of time spent in WAITING phase for request.
414# TYPE vllm:request_queue_time_seconds histogram
415vllm:request_queue_time_seconds_bucket{engine="0",le="0.3",model_name="qwen3-embedding"} 4.0
416vllm:request_queue_time_seconds_bucket{engine="0",le="0.5",model_name="qwen3-embedding"} 4.0
417vllm:request_queue_time_seconds_bucket{engine="0",le="0.8",model_name="qwen3-embedding"} 4.0
418vllm:request_queue_time_seconds_bucket{engine="0",le="1.0",model_name="qwen3-embedding"} 4.0
419vllm:request_queue_time_seconds_bucket{engine="0",le="1.5",model_name="qwen3-embedding"} 4.0
420vllm:request_queue_time_seconds_bucket{engine="0",le="2.0",model_name="qwen3-embedding"} 4.0
421vllm:request_queue_time_seconds_bucket{engine="0",le="2.5",model_name="qwen3-embedding"} 4.0
422vllm:request_queue_time_seconds_bucket{engine="0",le="5.0",model_name="qwen3-embedding"} 4.0
423vllm:request_queue_time_seconds_bucket{engine="0",le="10.0",model_name="qwen3-embedding"} 4.0
424vllm:request_queue_time_seconds_bucket{engine="0",le="15.0",model_name="qwen3-embedding"} 4.0
425vllm:request_queue_time_seconds_bucket{engine="0",le="20.0",model_name="qwen3-embedding"} 4.0
426vllm:request_queue_time_seconds_bucket{engine="0",le="30.0",model_name="qwen3-embedding"} 4.0
427vllm:request_queue_time_seconds_bucket{engine="0",le="40.0",model_name="qwen3-embedding"} 4.0
428vllm:request_queue_time_seconds_bucket{engine="0",le="50.0",model_name="qwen3-embedding"} 4.0
429vllm:request_queue_time_seconds_bucket{engine="0",le="60.0",model_name="qwen3-embedding"} 4.0
430vllm:request_queue_time_seconds_bucket{engine="0",le="120.0",model_name="qwen3-embedding"} 4.0
431vllm:request_queue_time_seconds_bucket{engine="0",le="240.0",model_name="qwen3-embedding"} 4.0
432vllm:request_queue_time_seconds_bucket{engine="0",le="480.0",model_name="qwen3-embedding"} 4.0
433vllm:request_queue_time_seconds_bucket{engine="0",le="960.0",model_name="qwen3-embedding"} 4.0
434vllm:request_queue_time_seconds_bucket{engine="0",le="1920.0",model_name="qwen3-embedding"} 4.0
435vllm:request_queue_time_seconds_bucket{engine="0",le="7680.0",model_name="qwen3-embedding"} 4.0
436vllm:request_queue_time_seconds_bucket{engine="0",le="+Inf",model_name="qwen3-embedding"} 4.0
437vllm:request_queue_time_seconds_count{engine="0",model_name="qwen3-embedding"} 4.0
438vllm:request_queue_time_seconds_sum{engine="0",model_name="qwen3-embedding"} 5.937740206718445e-05
439# HELP vllm:request_queue_time_seconds_created Histogram of time spent in WAITING phase for request.
440# TYPE vllm:request_queue_time_seconds_created gauge
441vllm:request_queue_time_seconds_created{engine="0",model_name="qwen3-embedding"} 1.783420629587268e+09
442# HELP vllm:request_inference_time_seconds Histogram of time spent in RUNNING phase for request.
443# TYPE vllm:request_inference_time_seconds histogram
444vllm:request_inference_time_seconds_bucket{engine="0",le="0.3",model_name="qwen3-embedding"} 3.0
445vllm:request_inference_time_seconds_bucket{engine="0",le="0.5",model_name="qwen3-embedding"} 4.0
446vllm:request_inference_time_seconds_bucket{engine="0",le="0.8",model_name="qwen3-embedding"} 4.0
447vllm:request_inference_time_seconds_bucket{engine="0",le="1.0",model_name="qwen3-embedding"} 4.0
448vllm:request_inference_time_seconds_bucket{engine="0",le="1.5",model_name="qwen3-embedding"} 4.0
449vllm:request_inference_time_seconds_bucket{engine="0",le="2.0",model_name="qwen3-embedding"} 4.0
450vllm:request_inference_time_seconds_bucket{engine="0",le="2.5",model_name="qwen3-embedding"} 4.0
451vllm:request_inference_time_seconds_bucket{engine="0",le="5.0",model_name="qwen3-embedding"} 4.0
452vllm:request_inference_time_seconds_bucket{engine="0",le="10.0",model_name="qwen3-embedding"} 4.0
453vllm:request_inference_time_seconds_bucket{engine="0",le="15.0",model_name="qwen3-embedding"} 4.0
454vllm:request_inference_time_seconds_bucket{engine="0",le="20.0",model_name="qwen3-embedding"} 4.0
455vllm:request_inference_time_seconds_bucket{engine="0",le="30.0",model_name="qwen3-embedding"} 4.0
456vllm:request_inference_time_seconds_bucket{engine="0",le="40.0",model_name="qwen3-embedding"} 4.0
457vllm:request_inference_time_seconds_bucket{engine="0",le="50.0",model_name="qwen3-embedding"} 4.0
458vllm:request_inference_time_seconds_bucket{engine="0",le="60.0",model_name="qwen3-embedding"} 4.0
459vllm:request_inference_time_seconds_bucket{engine="0",le="120.0",model_name="qwen3-embedding"} 4.0
460vllm:request_inference_time_seconds_bucket{engine="0",le="240.0",model_name="qwen3-embedding"} 4.0
461vllm:request_inference_time_seconds_bucket{engine="0",le="480.0",model_name="qwen3-embedding"} 4.0
462vllm:request_inference_time_seconds_bucket{engine="0",le="960.0",model_name="qwen3-embedding"} 4.0
463vllm:request_inference_time_seconds_bucket{engine="0",le="1920.0",model_name="qwen3-embedding"} 4.0
464vllm:request_inference_time_seconds_bucket{engine="0",le="7680.0",model_name="qwen3-embedding"} 4.0
465vllm:request_inference_time_seconds_bucket{engine="0",le="+Inf",model_name="qwen3-embedding"} 4.0
466vllm:request_inference_time_seconds_count{engine="0",model_name="qwen3-embedding"} 4.0
467vllm:request_inference_time_seconds_sum{engine="0",model_name="qwen3-embedding"} 0.9257321138866246
468# HELP vllm:request_inference_time_seconds_created Histogram of time spent in RUNNING phase for request.
469# TYPE vllm:request_inference_time_seconds_created gauge
470vllm:request_inference_time_seconds_created{engine="0",model_name="qwen3-embedding"} 1.7834206295873404e+09
471# HELP vllm:request_prefill_time_seconds Histogram of time spent in PREFILL phase for request.
472# TYPE vllm:request_prefill_time_seconds histogram
473vllm:request_prefill_time_seconds_bucket{engine="0",le="0.3",model_name="qwen3-embedding"} 3.0
474vllm:request_prefill_time_seconds_bucket{engine="0",le="0.5",model_name="qwen3-embedding"} 4.0
475vllm:request_prefill_time_seconds_bucket{engine="0",le="0.8",model_name="qwen3-embedding"} 4.0
476vllm:request_prefill_time_seconds_bucket{engine="0",le="1.0",model_name="qwen3-embedding"} 4.0
477vllm:request_prefill_time_seconds_bucket{engine="0",le="1.5",model_name="qwen3-embedding"} 4.0
478vllm:request_prefill_time_seconds_bucket{engine="0",le="2.0",model_name="qwen3-embedding"} 4.0
479vllm:request_prefill_time_seconds_bucket{engine="0",le="2.5",model_name="qwen3-embedding"} 4.0
480vllm:request_prefill_time_seconds_bucket{engine="0",le="5.0",model_name="qwen3-embedding"} 4.0
481vllm:request_prefill_time_seconds_bucket{engine="0",le="10.0",model_name="qwen3-embedding"} 4.0
482vllm:request_prefill_time_seconds_bucket{engine="0",le="15.0",model_name="qwen3-embedding"} 4.0
483vllm:request_prefill_time_seconds_bucket{engine="0",le="20.0",model_name="qwen3-embedding"} 4.0
484vllm:request_prefill_time_seconds_bucket{engine="0",le="30.0",model_name="qwen3-embedding"} 4.0
485vllm:request_prefill_time_seconds_bucket{engine="0",le="40.0",model_name="qwen3-embedding"} 4.0
486vllm:request_prefill_time_seconds_bucket{engine="0",le="50.0",model_name="qwen3-embedding"} 4.0
487vllm:request_prefill_time_seconds_bucket{engine="0",le="60.0",model_name="qwen3-embedding"} 4.0
488vllm:request_prefill_time_seconds_bucket{engine="0",le="120.0",model_name="qwen3-embedding"} 4.0
489vllm:request_prefill_time_seconds_bucket{engine="0",le="240.0",model_name="qwen3-embedding"} 4.0
490vllm:request_prefill_time_seconds_bucket{engine="0",le="480.0",model_name="qwen3-embedding"} 4.0
491vllm:request_prefill_time_seconds_bucket{engine="0",le="960.0",model_name="qwen3-embedding"} 4.0
492vllm:request_prefill_time_seconds_bucket{engine="0",le="1920.0",model_name="qwen3-embedding"} 4.0
493vllm:request_prefill_time_seconds_bucket{engine="0",le="7680.0",model_name="qwen3-embedding"} 4.0
494vllm:request_prefill_time_seconds_bucket{engine="0",le="+Inf",model_name="qwen3-embedding"} 4.0
495vllm:request_prefill_time_seconds_count{engine="0",model_name="qwen3-embedding"} 4.0
496vllm:request_prefill_time_seconds_sum{engine="0",model_name="qwen3-embedding"} 0.9257321138866246
497# HELP vllm:request_prefill_time_seconds_created Histogram of time spent in PREFILL phase for request.
498# TYPE vllm:request_prefill_time_seconds_created gauge
499vllm:request_prefill_time_seconds_created{engine="0",model_name="qwen3-embedding"} 1.783420629587382e+09
500# HELP vllm:request_decode_time_seconds Histogram of time spent in DECODE phase for request.
501# TYPE vllm:request_decode_time_seconds histogram
502vllm:request_decode_time_seconds_bucket{engine="0",le="0.3",model_name="qwen3-embedding"} 4.0
503vllm:request_decode_time_seconds_bucket{engine="0",le="0.5",model_name="qwen3-embedding"} 4.0
504vllm:request_decode_time_seconds_bucket{engine="0",le="0.8",model_name="qwen3-embedding"} 4.0
505vllm:request_decode_time_seconds_bucket{engine="0",le="1.0",model_name="qwen3-embedding"} 4.0
506vllm:request_decode_time_seconds_bucket{engine="0",le="1.5",model_name="qwen3-embedding"} 4.0
507vllm:request_decode_time_seconds_bucket{engine="0",le="2.0",model_name="qwen3-embedding"} 4.0
508vllm:request_decode_time_seconds_bucket{engine="0",le="2.5",model_name="qwen3-embedding"} 4.0
509vllm:request_decode_time_seconds_bucket{engine="0",le="5.0",model_name="qwen3-embedding"} 4.0
510vllm:request_decode_time_seconds_bucket{engine="0",le="10.0",model_name="qwen3-embedding"} 4.0
511vllm:request_decode_time_seconds_bucket{engine="0",le="15.0",model_name="qwen3-embedding"} 4.0
512vllm:request_decode_time_seconds_bucket{engine="0",le="20.0",model_name="qwen3-embedding"} 4.0
513vllm:request_decode_time_seconds_bucket{engine="0",le="30.0",model_name="qwen3-embedding"} 4.0
514vllm:request_decode_time_seconds_bucket{engine="0",le="40.0",model_name="qwen3-embedding"} 4.0
515vllm:request_decode_time_seconds_bucket{engine="0",le="50.0",model_name="qwen3-embedding"} 4.0
516vllm:request_decode_time_seconds_bucket{engine="0",le="60.0",model_name="qwen3-embedding"} 4.0
517vllm:request_decode_time_seconds_bucket{engine="0",le="120.0",model_name="qwen3-embedding"} 4.0
518vllm:request_decode_time_seconds_bucket{engine="0",le="240.0",model_name="qwen3-embedding"} 4.0
519vllm:request_decode_time_seconds_bucket{engine="0",le="480.0",model_name="qwen3-embedding"} 4.0
520vllm:request_decode_time_seconds_bucket{engine="0",le="960.0",model_name="qwen3-embedding"} 4.0
521vllm:request_decode_time_seconds_bucket{engine="0",le="1920.0",model_name="qwen3-embedding"} 4.0
522vllm:request_decode_time_seconds_bucket{engine="0",le="7680.0",model_name="qwen3-embedding"} 4.0
523vllm:request_decode_time_seconds_bucket{engine="0",le="+Inf",model_name="qwen3-embedding"} 4.0
524vllm:request_decode_time_seconds_count{engine="0",model_name="qwen3-embedding"} 4.0
525vllm:request_decode_time_seconds_sum{engine="0",model_name="qwen3-embedding"} 0.0
526# HELP vllm:request_decode_time_seconds_created Histogram of time spent in DECODE phase for request.
527# TYPE vllm:request_decode_time_seconds_created gauge
528vllm:request_decode_time_seconds_created{engine="0",model_name="qwen3-embedding"} 1.7834206295874138e+09
529# HELP vllm:request_prefill_kv_computed_tokens Histogram of new KV tokens computed during prefill (excluding cached tokens).
530# TYPE vllm:request_prefill_kv_computed_tokens histogram
531vllm:request_prefill_kv_computed_tokens_bucket{engine="0",le="1.0",model_name="qwen3-embedding"} 0.0
532vllm:request_prefill_kv_computed_tokens_bucket{engine="0",le="2.0",model_name="qwen3-embedding"} 0.0
533vllm:request_prefill_kv_computed_tokens_bucket{engine="0",le="5.0",model_name="qwen3-embedding"} 4.0
534vllm:request_prefill_kv_computed_tokens_bucket{engine="0",le="10.0",model_name="qwen3-embedding"} 4.0
535vllm:request_prefill_kv_computed_tokens_bucket{engine="0",le="20.0",model_name="qwen3-embedding"} 4.0
536vllm:request_prefill_kv_computed_tokens_bucket{engine="0",le="50.0",model_name="qwen3-embedding"} 4.0
537vllm:request_prefill_kv_computed_tokens_bucket{engine="0",le="100.0",model_name="qwen3-embedding"} 4.0
538vllm:request_prefill_kv_computed_tokens_bucket{engine="0",le="200.0",model_name="qwen3-embedding"} 4.0
539vllm:request_prefill_kv_computed_tokens_bucket{engine="0",le="500.0",model_name="qwen3-embedding"} 4.0
540vllm:request_prefill_kv_computed_tokens_bucket{engine="0",le="1000.0",model_name="qwen3-embedding"} 4.0
541vllm:request_prefill_kv_computed_tokens_bucket{engine="0",le="2000.0",model_name="qwen3-embedding"} 4.0
542vllm:request_prefill_kv_computed_tokens_bucket{engine="0",le="5000.0",model_name="qwen3-embedding"} 4.0
543vllm:request_prefill_kv_computed_tokens_bucket{engine="0",le="+Inf",model_name="qwen3-embedding"} 4.0
544vllm:request_prefill_kv_computed_tokens_count{engine="0",model_name="qwen3-embedding"} 4.0
545vllm:request_prefill_kv_computed_tokens_sum{engine="0",model_name="qwen3-embedding"} 12.0
546# HELP vllm:request_prefill_kv_computed_tokens_created Histogram of new KV tokens computed during prefill (excluding cached tokens).
547# TYPE vllm:request_prefill_kv_computed_tokens_created gauge
548vllm:request_prefill_kv_computed_tokens_created{engine="0",model_name="qwen3-embedding"} 1.783420629587453e+09
549# HELP vllm:cache_config_info Information of the LLMEngine CacheConfig
550# TYPE vllm:cache_config_info gauge
551vllm:cache_config_info{block_size="16",cache_dtype="auto",calculate_kv_scales="False",cpu_kvcache_space_bytes="None",cpu_offload_gb="0.0",enable_prefix_caching="False",engine="0",gpu_memory_utilization="0.9",is_attention_free="False",kv_cache_memory_bytes="None",kv_offloading_backend="native",kv_offloading_size="None",kv_sharing_fast_prefill="False",mamba_block_size="None",mamba_cache_dtype="auto",mamba_cache_mode="none",mamba_page_size_padded="None",mamba_ssm_cache_dtype="auto",num_cpu_blocks="None",num_gpu_blocks="11477",num_gpu_blocks_override="None",prefix_caching_hash_algo="sha256",sliding_window="None",swap_space="4.0"} 1.0

Change #1308052 merged by jenkins-bot:

[operations/deployment-charts@master] ml-services: deploy qwen36 isvc image that emits vLLM metrics

https://gerrit.wikimedia.org/r/1308052

The llm-qwen3-14b isvc now emits vLLM metrics as shown below:

1$ curl "https://inference.svc.eqiad.wmnet:30443/metrics" -H "Host: llm-qwen3-14b-predictor.llm.wikimedia.org"
2# HELP python_gc_objects_collected_total Objects collected during gc
3# TYPE python_gc_objects_collected_total counter
4python_gc_objects_collected_total{generation="0"} 19606.0
5python_gc_objects_collected_total{generation="1"} 2539.0
6python_gc_objects_collected_total{generation="2"} 358.0
7# HELP python_gc_objects_uncollectable_total Uncollectable objects found during GC
8# TYPE python_gc_objects_uncollectable_total counter
9python_gc_objects_uncollectable_total{generation="0"} 0.0
10python_gc_objects_uncollectable_total{generation="1"} 0.0
11python_gc_objects_uncollectable_total{generation="2"} 0.0
12# HELP python_gc_collections_total Number of times this generation was collected
13# TYPE python_gc_collections_total counter
14python_gc_collections_total{generation="0"} 2213.0
15python_gc_collections_total{generation="1"} 201.0
16python_gc_collections_total{generation="2"} 11.0
17# HELP python_info Python platform information
18# TYPE python_info gauge
19python_info{implementation="CPython",major="3",minor="12",patchlevel="13",version="3.12.13"} 1.0
20# HELP process_virtual_memory_bytes Virtual memory size in bytes.
21# TYPE process_virtual_memory_bytes gauge
22process_virtual_memory_bytes 4.8395821056e+010
23# HELP process_resident_memory_bytes Resident memory size in bytes.
24# TYPE process_resident_memory_bytes gauge
25process_resident_memory_bytes 1.929314304e+09
26# HELP process_start_time_seconds Start time of the process since unix epoch in seconds.
27# TYPE process_start_time_seconds gauge
28process_start_time_seconds 1.78342147591e+09
29# HELP process_cpu_seconds_total Total user and system CPU time spent in seconds.
30# TYPE process_cpu_seconds_total counter
31process_cpu_seconds_total 24.669999999999998
32# HELP process_open_fds Number of open file descriptors.
33# TYPE process_open_fds gauge
34process_open_fds 39.0
35# HELP process_max_fds Maximum number of open file descriptors.
36# TYPE process_max_fds gauge
37process_max_fds 1.073741816e+09
38# HELP request_preprocess_seconds pre-process request latency
39# TYPE request_preprocess_seconds histogram
40# HELP request_postprocess_seconds post-process request latency
41# TYPE request_postprocess_seconds histogram
42# HELP request_predict_seconds predict request latency
43# TYPE request_predict_seconds histogram
44# HELP request_explain_seconds explain request latency
45# TYPE request_explain_seconds histogram
46# HELP vllm:estimated_flops_per_gpu_total Estimated number of floating point operations per GPU (for Model Flops Utilization calculations).
47# TYPE vllm:estimated_flops_per_gpu_total counter
48vllm:estimated_flops_per_gpu_total{engine="0",model_name="llm-qwen3-14b"} 0.0
49# HELP vllm:estimated_flops_per_gpu_created Estimated number of floating point operations per GPU (for Model Flops Utilization calculations).
50# TYPE vllm:estimated_flops_per_gpu_created gauge
51vllm:estimated_flops_per_gpu_created{engine="0",model_name="llm-qwen3-14b"} 1.7834215670775456e+09
52# HELP vllm:estimated_read_bytes_per_gpu_total Estimated number of bytes read from memory per GPU (for Model Flops Utilization calculations).
53# TYPE vllm:estimated_read_bytes_per_gpu_total counter
54vllm:estimated_read_bytes_per_gpu_total{engine="0",model_name="llm-qwen3-14b"} 0.0
55# HELP vllm:estimated_read_bytes_per_gpu_created Estimated number of bytes read from memory per GPU (for Model Flops Utilization calculations).
56# TYPE vllm:estimated_read_bytes_per_gpu_created gauge
57vllm:estimated_read_bytes_per_gpu_created{engine="0",model_name="llm-qwen3-14b"} 1.7834215670775683e+09
58# HELP vllm:estimated_write_bytes_per_gpu_total Estimated number of bytes written to memory per GPU (for Model Flops Utilization calculations).
59# TYPE vllm:estimated_write_bytes_per_gpu_total counter
60vllm:estimated_write_bytes_per_gpu_total{engine="0",model_name="llm-qwen3-14b"} 0.0
61# HELP vllm:estimated_write_bytes_per_gpu_created Estimated number of bytes written to memory per GPU (for Model Flops Utilization calculations).
62# TYPE vllm:estimated_write_bytes_per_gpu_created gauge
63vllm:estimated_write_bytes_per_gpu_created{engine="0",model_name="llm-qwen3-14b"} 1.7834215670775826e+09
64# HELP vllm:num_requests_running Number of requests in model execution batches.
65# TYPE vllm:num_requests_running gauge
66vllm:num_requests_running{engine="0",model_name="llm-qwen3-14b"} 0.0
67# HELP vllm:num_requests_waiting Number of requests waiting to be processed.
68# TYPE vllm:num_requests_waiting gauge
69vllm:num_requests_waiting{engine="0",model_name="llm-qwen3-14b"} 0.0
70# HELP vllm:num_requests_waiting_by_reason Number of waiting requests by reason. Reason labels: 'capacity' = waiting for scheduling capacity; 'deferred' = deferred by transient constraints (LoRA budget, KV transfer, blocked status). Sum of all reasons equals vllm:num_requests_waiting.
71# TYPE vllm:num_requests_waiting_by_reason gauge
72vllm:num_requests_waiting_by_reason{engine="0",model_name="llm-qwen3-14b",reason="capacity"} 0.0
73vllm:num_requests_waiting_by_reason{engine="0",model_name="llm-qwen3-14b",reason="deferred"} 0.0
74# HELP vllm:engine_sleep_state Engine sleep state; awake = 0 means engine is sleeping; awake = 1 means engine is awake; weights_offloaded = 1 means sleep level 1; discard_all = 1 means sleep level 2.
75# TYPE vllm:engine_sleep_state gauge
76vllm:engine_sleep_state{engine="0",model_name="llm-qwen3-14b",sleep_state="awake"} 1.0
77vllm:engine_sleep_state{engine="0",model_name="llm-qwen3-14b",sleep_state="weights_offloaded"} 0.0
78vllm:engine_sleep_state{engine="0",model_name="llm-qwen3-14b",sleep_state="discard_all"} 0.0
79# HELP vllm:kv_cache_usage_perc KV-cache usage. 1 means 100 percent usage.
80# TYPE vllm:kv_cache_usage_perc gauge
81vllm:kv_cache_usage_perc{engine="0",model_name="llm-qwen3-14b"} 0.0
82# HELP vllm:prefix_cache_queries_total Prefix cache queries, in terms of number of queried tokens.
83# TYPE vllm:prefix_cache_queries_total counter
84vllm:prefix_cache_queries_total{engine="0",model_name="llm-qwen3-14b"} 38.0
85# HELP vllm:prefix_cache_queries_created Prefix cache queries, in terms of number of queried tokens.
86# TYPE vllm:prefix_cache_queries_created gauge
87vllm:prefix_cache_queries_created{engine="0",model_name="llm-qwen3-14b"} 1.783421567077721e+09
88# HELP vllm:prefix_cache_hits_total Prefix cache hits, in terms of number of cached tokens.
89# TYPE vllm:prefix_cache_hits_total counter
90vllm:prefix_cache_hits_total{engine="0",model_name="llm-qwen3-14b"} 0.0
91# HELP vllm:prefix_cache_hits_created Prefix cache hits, in terms of number of cached tokens.
92# TYPE vllm:prefix_cache_hits_created gauge
93vllm:prefix_cache_hits_created{engine="0",model_name="llm-qwen3-14b"} 1.7834215670777328e+09
94# HELP vllm:external_prefix_cache_queries_total External prefix cache queries from KV connector cross-instance cache sharing, in terms of number of queried tokens.
95# TYPE vllm:external_prefix_cache_queries_total counter
96vllm:external_prefix_cache_queries_total{engine="0",model_name="llm-qwen3-14b"} 0.0
97# HELP vllm:external_prefix_cache_queries_created External prefix cache queries from KV connector cross-instance cache sharing, in terms of number of queried tokens.
98# TYPE vllm:external_prefix_cache_queries_created gauge
99vllm:external_prefix_cache_queries_created{engine="0",model_name="llm-qwen3-14b"} 1.7834215670777435e+09
100# HELP vllm:external_prefix_cache_hits_total External prefix cache hits from KV connector cross-instance cache sharing, in terms of number of cached tokens.
101# TYPE vllm:external_prefix_cache_hits_total counter
102vllm:external_prefix_cache_hits_total{engine="0",model_name="llm-qwen3-14b"} 0.0
103# HELP vllm:external_prefix_cache_hits_created External prefix cache hits from KV connector cross-instance cache sharing, in terms of number of cached tokens.
104# TYPE vllm:external_prefix_cache_hits_created gauge
105vllm:external_prefix_cache_hits_created{engine="0",model_name="llm-qwen3-14b"} 1.7834215670777576e+09
106# HELP vllm:mm_cache_queries_total Multi-modal cache queries, in terms of number of queried items.
107# TYPE vllm:mm_cache_queries_total counter
108vllm:mm_cache_queries_total{engine="0",model_name="llm-qwen3-14b"} 0.0
109# HELP vllm:mm_cache_queries_created Multi-modal cache queries, in terms of number of queried items.
110# TYPE vllm:mm_cache_queries_created gauge
111vllm:mm_cache_queries_created{engine="0",model_name="llm-qwen3-14b"} 1.7834215670777688e+09
112# HELP vllm:mm_cache_hits_total Multi-modal cache hits, in terms of number of cached items.
113# TYPE vllm:mm_cache_hits_total counter
114vllm:mm_cache_hits_total{engine="0",model_name="llm-qwen3-14b"} 0.0
115# HELP vllm:mm_cache_hits_created Multi-modal cache hits, in terms of number of cached items.
116# TYPE vllm:mm_cache_hits_created gauge
117vllm:mm_cache_hits_created{engine="0",model_name="llm-qwen3-14b"} 1.783421567077779e+09
118# HELP vllm:num_preemptions_total Cumulative number of preemption from the engine.
119# TYPE vllm:num_preemptions_total counter
120vllm:num_preemptions_total{engine="0",model_name="llm-qwen3-14b"} 0.0
121# HELP vllm:num_preemptions_created Cumulative number of preemption from the engine.
122# TYPE vllm:num_preemptions_created gauge
123vllm:num_preemptions_created{engine="0",model_name="llm-qwen3-14b"} 1.783421567077789e+09
124# HELP vllm:prompt_tokens_total Number of prefill tokens processed.
125# TYPE vllm:prompt_tokens_total counter
126vllm:prompt_tokens_total{engine="0",model_name="llm-qwen3-14b"} 38.0
127# HELP vllm:prompt_tokens_created Number of prefill tokens processed.
128# TYPE vllm:prompt_tokens_created gauge
129vllm:prompt_tokens_created{engine="0",model_name="llm-qwen3-14b"} 1.7834215670777988e+09
130# HELP vllm:prompt_tokens_by_source_total Number of prompt tokens by source.
131# TYPE vllm:prompt_tokens_by_source_total counter
132vllm:prompt_tokens_by_source_total{engine="0",model_name="llm-qwen3-14b",source="local_compute"} 38.0
133vllm:prompt_tokens_by_source_total{engine="0",model_name="llm-qwen3-14b",source="local_cache_hit"} 0.0
134vllm:prompt_tokens_by_source_total{engine="0",model_name="llm-qwen3-14b",source="external_kv_transfer"} 0.0
135# HELP vllm:prompt_tokens_by_source_created Number of prompt tokens by source.
136# TYPE vllm:prompt_tokens_by_source_created gauge
137vllm:prompt_tokens_by_source_created{engine="0",model_name="llm-qwen3-14b",source="local_compute"} 1.7834215670778117e+09
138vllm:prompt_tokens_by_source_created{engine="0",model_name="llm-qwen3-14b",source="local_cache_hit"} 1.7834215670786095e+09
139vllm:prompt_tokens_by_source_created{engine="0",model_name="llm-qwen3-14b",source="external_kv_transfer"} 1.7834215670786161e+09
140# HELP vllm:prompt_tokens_cached_total Number of cached prompt tokens (local + external).
141# TYPE vllm:prompt_tokens_cached_total counter
142vllm:prompt_tokens_cached_total{engine="0",model_name="llm-qwen3-14b"} 0.0
143# HELP vllm:prompt_tokens_cached_created Number of cached prompt tokens (local + external).
144# TYPE vllm:prompt_tokens_cached_created gauge
145vllm:prompt_tokens_cached_created{engine="0",model_name="llm-qwen3-14b"} 1.78342156707863e+09
146# HELP vllm:generation_tokens_total Number of generation tokens processed.
147# TYPE vllm:generation_tokens_total counter
148vllm:generation_tokens_total{engine="0",model_name="llm-qwen3-14b"} 18.0
149# HELP vllm:generation_tokens_created Number of generation tokens processed.
150# TYPE vllm:generation_tokens_created gauge
151vllm:generation_tokens_created{engine="0",model_name="llm-qwen3-14b"} 1.783421567078641e+09
152# HELP vllm:request_success_total Count of successfully processed requests.
153# TYPE vllm:request_success_total counter
154vllm:request_success_total{engine="0",finished_reason="stop",model_name="llm-qwen3-14b"} 2.0
155vllm:request_success_total{engine="0",finished_reason="length",model_name="llm-qwen3-14b"} 0.0
156vllm:request_success_total{engine="0",finished_reason="abort",model_name="llm-qwen3-14b"} 0.0
157vllm:request_success_total{engine="0",finished_reason="error",model_name="llm-qwen3-14b"} 0.0
158vllm:request_success_total{engine="0",finished_reason="repetition",model_name="llm-qwen3-14b"} 0.0
159# HELP vllm:request_success_created Count of successfully processed requests.
160# TYPE vllm:request_success_created gauge
161vllm:request_success_created{engine="0",finished_reason="stop",model_name="llm-qwen3-14b"} 1.7834215670786648e+09
162vllm:request_success_created{engine="0",finished_reason="length",model_name="llm-qwen3-14b"} 1.7834215670786707e+09
163vllm:request_success_created{engine="0",finished_reason="abort",model_name="llm-qwen3-14b"} 1.7834215670786765e+09
164vllm:request_success_created{engine="0",finished_reason="error",model_name="llm-qwen3-14b"} 1.7834215670786831e+09
165vllm:request_success_created{engine="0",finished_reason="repetition",model_name="llm-qwen3-14b"} 1.7834215670786877e+09
166# HELP vllm:request_prompt_tokens Number of prefill tokens processed.
167# TYPE vllm:request_prompt_tokens histogram
168vllm:request_prompt_tokens_bucket{engine="0",le="1.0",model_name="llm-qwen3-14b"} 0.0
169vllm:request_prompt_tokens_bucket{engine="0",le="2.0",model_name="llm-qwen3-14b"} 0.0
170vllm:request_prompt_tokens_bucket{engine="0",le="5.0",model_name="llm-qwen3-14b"} 0.0
171vllm:request_prompt_tokens_bucket{engine="0",le="10.0",model_name="llm-qwen3-14b"} 0.0
172vllm:request_prompt_tokens_bucket{engine="0",le="20.0",model_name="llm-qwen3-14b"} 2.0
173vllm:request_prompt_tokens_bucket{engine="0",le="50.0",model_name="llm-qwen3-14b"} 2.0
174vllm:request_prompt_tokens_bucket{engine="0",le="100.0",model_name="llm-qwen3-14b"} 2.0
175vllm:request_prompt_tokens_bucket{engine="0",le="200.0",model_name="llm-qwen3-14b"} 2.0
176vllm:request_prompt_tokens_bucket{engine="0",le="500.0",model_name="llm-qwen3-14b"} 2.0
177vllm:request_prompt_tokens_bucket{engine="0",le="1000.0",model_name="llm-qwen3-14b"} 2.0
178vllm:request_prompt_tokens_bucket{engine="0",le="2000.0",model_name="llm-qwen3-14b"} 2.0
179vllm:request_prompt_tokens_bucket{engine="0",le="5000.0",model_name="llm-qwen3-14b"} 2.0
180vllm:request_prompt_tokens_bucket{engine="0",le="10000.0",model_name="llm-qwen3-14b"} 2.0
181vllm:request_prompt_tokens_bucket{engine="0",le="20000.0",model_name="llm-qwen3-14b"} 2.0
182vllm:request_prompt_tokens_bucket{engine="0",le="+Inf",model_name="llm-qwen3-14b"} 2.0
183vllm:request_prompt_tokens_count{engine="0",model_name="llm-qwen3-14b"} 2.0
184vllm:request_prompt_tokens_sum{engine="0",model_name="llm-qwen3-14b"} 38.0
185# HELP vllm:request_prompt_tokens_created Number of prefill tokens processed.
186# TYPE vllm:request_prompt_tokens_created gauge
187vllm:request_prompt_tokens_created{engine="0",model_name="llm-qwen3-14b"} 1.7834215670787218e+09
188# HELP vllm:request_generation_tokens Number of generation tokens processed.
189# TYPE vllm:request_generation_tokens histogram
190vllm:request_generation_tokens_bucket{engine="0",le="1.0",model_name="llm-qwen3-14b"} 0.0
191vllm:request_generation_tokens_bucket{engine="0",le="2.0",model_name="llm-qwen3-14b"} 0.0
192vllm:request_generation_tokens_bucket{engine="0",le="5.0",model_name="llm-qwen3-14b"} 0.0
193vllm:request_generation_tokens_bucket{engine="0",le="10.0",model_name="llm-qwen3-14b"} 2.0
194vllm:request_generation_tokens_bucket{engine="0",le="20.0",model_name="llm-qwen3-14b"} 2.0
195vllm:request_generation_tokens_bucket{engine="0",le="50.0",model_name="llm-qwen3-14b"} 2.0
196vllm:request_generation_tokens_bucket{engine="0",le="100.0",model_name="llm-qwen3-14b"} 2.0
197vllm:request_generation_tokens_bucket{engine="0",le="200.0",model_name="llm-qwen3-14b"} 2.0
198vllm:request_generation_tokens_bucket{engine="0",le="500.0",model_name="llm-qwen3-14b"} 2.0
199vllm:request_generation_tokens_bucket{engine="0",le="1000.0",model_name="llm-qwen3-14b"} 2.0
200vllm:request_generation_tokens_bucket{engine="0",le="2000.0",model_name="llm-qwen3-14b"} 2.0
201vllm:request_generation_tokens_bucket{engine="0",le="5000.0",model_name="llm-qwen3-14b"} 2.0
202vllm:request_generation_tokens_bucket{engine="0",le="10000.0",model_name="llm-qwen3-14b"} 2.0
203vllm:request_generation_tokens_bucket{engine="0",le="20000.0",model_name="llm-qwen3-14b"} 2.0
204vllm:request_generation_tokens_bucket{engine="0",le="+Inf",model_name="llm-qwen3-14b"} 2.0
205vllm:request_generation_tokens_count{engine="0",model_name="llm-qwen3-14b"} 2.0
206vllm:request_generation_tokens_sum{engine="0",model_name="llm-qwen3-14b"} 18.0
207# HELP vllm:request_generation_tokens_created Number of generation tokens processed.
208# TYPE vllm:request_generation_tokens_created gauge
209vllm:request_generation_tokens_created{engine="0",model_name="llm-qwen3-14b"} 1.783421567078772e+09
210# HELP vllm:iteration_tokens_total Histogram of number of tokens per engine_step.
211# TYPE vllm:iteration_tokens_total histogram
212vllm:iteration_tokens_total_bucket{engine="0",le="1.0",model_name="llm-qwen3-14b"} 16.0
213vllm:iteration_tokens_total_bucket{engine="0",le="8.0",model_name="llm-qwen3-14b"} 16.0
214vllm:iteration_tokens_total_bucket{engine="0",le="16.0",model_name="llm-qwen3-14b"} 16.0
215vllm:iteration_tokens_total_bucket{engine="0",le="32.0",model_name="llm-qwen3-14b"} 18.0
216vllm:iteration_tokens_total_bucket{engine="0",le="64.0",model_name="llm-qwen3-14b"} 18.0
217vllm:iteration_tokens_total_bucket{engine="0",le="128.0",model_name="llm-qwen3-14b"} 18.0
218vllm:iteration_tokens_total_bucket{engine="0",le="256.0",model_name="llm-qwen3-14b"} 18.0
219vllm:iteration_tokens_total_bucket{engine="0",le="512.0",model_name="llm-qwen3-14b"} 18.0
220vllm:iteration_tokens_total_bucket{engine="0",le="1024.0",model_name="llm-qwen3-14b"} 18.0
221vllm:iteration_tokens_total_bucket{engine="0",le="2048.0",model_name="llm-qwen3-14b"} 18.0
222vllm:iteration_tokens_total_bucket{engine="0",le="4096.0",model_name="llm-qwen3-14b"} 18.0
223vllm:iteration_tokens_total_bucket{engine="0",le="8192.0",model_name="llm-qwen3-14b"} 18.0
224vllm:iteration_tokens_total_bucket{engine="0",le="16384.0",model_name="llm-qwen3-14b"} 18.0
225vllm:iteration_tokens_total_bucket{engine="0",le="+Inf",model_name="llm-qwen3-14b"} 18.0
226vllm:iteration_tokens_total_count{engine="0",model_name="llm-qwen3-14b"} 18.0
227vllm:iteration_tokens_total_sum{engine="0",model_name="llm-qwen3-14b"} 56.0
228# HELP vllm:iteration_tokens_total_created Histogram of number of tokens per engine_step.
229# TYPE vllm:iteration_tokens_total_created gauge
230vllm:iteration_tokens_total_created{engine="0",model_name="llm-qwen3-14b"} 1.7834215670788004e+09
231# HELP vllm:request_max_num_generation_tokens Histogram of maximum number of requested generation tokens.
232# TYPE vllm:request_max_num_generation_tokens histogram
233vllm:request_max_num_generation_tokens_bucket{engine="0",le="1.0",model_name="llm-qwen3-14b"} 0.0
234vllm:request_max_num_generation_tokens_bucket{engine="0",le="2.0",model_name="llm-qwen3-14b"} 0.0
235vllm:request_max_num_generation_tokens_bucket{engine="0",le="5.0",model_name="llm-qwen3-14b"} 0.0
236vllm:request_max_num_generation_tokens_bucket{engine="0",le="10.0",model_name="llm-qwen3-14b"} 2.0
237vllm:request_max_num_generation_tokens_bucket{engine="0",le="20.0",model_name="llm-qwen3-14b"} 2.0
238vllm:request_max_num_generation_tokens_bucket{engine="0",le="50.0",model_name="llm-qwen3-14b"} 2.0
239vllm:request_max_num_generation_tokens_bucket{engine="0",le="100.0",model_name="llm-qwen3-14b"} 2.0
240vllm:request_max_num_generation_tokens_bucket{engine="0",le="200.0",model_name="llm-qwen3-14b"} 2.0
241vllm:request_max_num_generation_tokens_bucket{engine="0",le="500.0",model_name="llm-qwen3-14b"} 2.0
242vllm:request_max_num_generation_tokens_bucket{engine="0",le="1000.0",model_name="llm-qwen3-14b"} 2.0
243vllm:request_max_num_generation_tokens_bucket{engine="0",le="2000.0",model_name="llm-qwen3-14b"} 2.0
244vllm:request_max_num_generation_tokens_bucket{engine="0",le="5000.0",model_name="llm-qwen3-14b"} 2.0
245vllm:request_max_num_generation_tokens_bucket{engine="0",le="10000.0",model_name="llm-qwen3-14b"} 2.0
246vllm:request_max_num_generation_tokens_bucket{engine="0",le="20000.0",model_name="llm-qwen3-14b"} 2.0
247vllm:request_max_num_generation_tokens_bucket{engine="0",le="+Inf",model_name="llm-qwen3-14b"} 2.0
248vllm:request_max_num_generation_tokens_count{engine="0",model_name="llm-qwen3-14b"} 2.0
249vllm:request_max_num_generation_tokens_sum{engine="0",model_name="llm-qwen3-14b"} 18.0
250# HELP vllm:request_max_num_generation_tokens_created Histogram of maximum number of requested generation tokens.
251# TYPE vllm:request_max_num_generation_tokens_created gauge
252vllm:request_max_num_generation_tokens_created{engine="0",model_name="llm-qwen3-14b"} 1.783421567078826e+09
253# HELP vllm:request_params_n Histogram of the n request parameter.
254# TYPE vllm:request_params_n histogram
255vllm:request_params_n_bucket{engine="0",le="1.0",model_name="llm-qwen3-14b"} 2.0
256vllm:request_params_n_bucket{engine="0",le="2.0",model_name="llm-qwen3-14b"} 2.0
257vllm:request_params_n_bucket{engine="0",le="5.0",model_name="llm-qwen3-14b"} 2.0
258vllm:request_params_n_bucket{engine="0",le="10.0",model_name="llm-qwen3-14b"} 2.0
259vllm:request_params_n_bucket{engine="0",le="20.0",model_name="llm-qwen3-14b"} 2.0
260vllm:request_params_n_bucket{engine="0",le="+Inf",model_name="llm-qwen3-14b"} 2.0
261vllm:request_params_n_count{engine="0",model_name="llm-qwen3-14b"} 2.0
262vllm:request_params_n_sum{engine="0",model_name="llm-qwen3-14b"} 2.0
263# HELP vllm:request_params_n_created Histogram of the n request parameter.
264# TYPE vllm:request_params_n_created gauge
265vllm:request_params_n_created{engine="0",model_name="llm-qwen3-14b"} 1.78342156707885e+09
266# HELP vllm:request_params_max_tokens Histogram of the max_tokens request parameter.
267# TYPE vllm:request_params_max_tokens histogram
268vllm:request_params_max_tokens_bucket{engine="0",le="1.0",model_name="llm-qwen3-14b"} 0.0
269vllm:request_params_max_tokens_bucket{engine="0",le="2.0",model_name="llm-qwen3-14b"} 0.0
270vllm:request_params_max_tokens_bucket{engine="0",le="5.0",model_name="llm-qwen3-14b"} 0.0
271vllm:request_params_max_tokens_bucket{engine="0",le="10.0",model_name="llm-qwen3-14b"} 0.0
272vllm:request_params_max_tokens_bucket{engine="0",le="20.0",model_name="llm-qwen3-14b"} 0.0
273vllm:request_params_max_tokens_bucket{engine="0",le="50.0",model_name="llm-qwen3-14b"} 2.0
274vllm:request_params_max_tokens_bucket{engine="0",le="100.0",model_name="llm-qwen3-14b"} 2.0
275vllm:request_params_max_tokens_bucket{engine="0",le="200.0",model_name="llm-qwen3-14b"} 2.0
276vllm:request_params_max_tokens_bucket{engine="0",le="500.0",model_name="llm-qwen3-14b"} 2.0
277vllm:request_params_max_tokens_bucket{engine="0",le="1000.0",model_name="llm-qwen3-14b"} 2.0
278vllm:request_params_max_tokens_bucket{engine="0",le="2000.0",model_name="llm-qwen3-14b"} 2.0
279vllm:request_params_max_tokens_bucket{engine="0",le="5000.0",model_name="llm-qwen3-14b"} 2.0
280vllm:request_params_max_tokens_bucket{engine="0",le="10000.0",model_name="llm-qwen3-14b"} 2.0
281vllm:request_params_max_tokens_bucket{engine="0",le="20000.0",model_name="llm-qwen3-14b"} 2.0
282vllm:request_params_max_tokens_bucket{engine="0",le="+Inf",model_name="llm-qwen3-14b"} 2.0
283vllm:request_params_max_tokens_count{engine="0",model_name="llm-qwen3-14b"} 2.0
284vllm:request_params_max_tokens_sum{engine="0",model_name="llm-qwen3-14b"} 100.0
285# HELP vllm:request_params_max_tokens_created Histogram of the max_tokens request parameter.
286# TYPE vllm:request_params_max_tokens_created gauge
287vllm:request_params_max_tokens_created{engine="0",model_name="llm-qwen3-14b"} 1.783421567078874e+09
288# HELP vllm:time_to_first_token_seconds Histogram of time to first token in seconds.
289# TYPE vllm:time_to_first_token_seconds histogram
290vllm:time_to_first_token_seconds_bucket{engine="0",le="0.001",model_name="llm-qwen3-14b"} 0.0
291vllm:time_to_first_token_seconds_bucket{engine="0",le="0.005",model_name="llm-qwen3-14b"} 0.0
292vllm:time_to_first_token_seconds_bucket{engine="0",le="0.01",model_name="llm-qwen3-14b"} 0.0
293vllm:time_to_first_token_seconds_bucket{engine="0",le="0.02",model_name="llm-qwen3-14b"} 1.0
294vllm:time_to_first_token_seconds_bucket{engine="0",le="0.04",model_name="llm-qwen3-14b"} 1.0
295vllm:time_to_first_token_seconds_bucket{engine="0",le="0.06",model_name="llm-qwen3-14b"} 1.0
296vllm:time_to_first_token_seconds_bucket{engine="0",le="0.08",model_name="llm-qwen3-14b"} 1.0
297vllm:time_to_first_token_seconds_bucket{engine="0",le="0.1",model_name="llm-qwen3-14b"} 1.0
298vllm:time_to_first_token_seconds_bucket{engine="0",le="0.25",model_name="llm-qwen3-14b"} 1.0
299vllm:time_to_first_token_seconds_bucket{engine="0",le="0.5",model_name="llm-qwen3-14b"} 1.0
300vllm:time_to_first_token_seconds_bucket{engine="0",le="0.75",model_name="llm-qwen3-14b"} 1.0
301vllm:time_to_first_token_seconds_bucket{engine="0",le="1.0",model_name="llm-qwen3-14b"} 1.0
302vllm:time_to_first_token_seconds_bucket{engine="0",le="2.5",model_name="llm-qwen3-14b"} 2.0
303vllm:time_to_first_token_seconds_bucket{engine="0",le="5.0",model_name="llm-qwen3-14b"} 2.0
304vllm:time_to_first_token_seconds_bucket{engine="0",le="7.5",model_name="llm-qwen3-14b"} 2.0
305vllm:time_to_first_token_seconds_bucket{engine="0",le="10.0",model_name="llm-qwen3-14b"} 2.0
306vllm:time_to_first_token_seconds_bucket{engine="0",le="20.0",model_name="llm-qwen3-14b"} 2.0
307vllm:time_to_first_token_seconds_bucket{engine="0",le="40.0",model_name="llm-qwen3-14b"} 2.0
308vllm:time_to_first_token_seconds_bucket{engine="0",le="80.0",model_name="llm-qwen3-14b"} 2.0
309vllm:time_to_first_token_seconds_bucket{engine="0",le="160.0",model_name="llm-qwen3-14b"} 2.0
310vllm:time_to_first_token_seconds_bucket{engine="0",le="640.0",model_name="llm-qwen3-14b"} 2.0
311vllm:time_to_first_token_seconds_bucket{engine="0",le="2560.0",model_name="llm-qwen3-14b"} 2.0
312vllm:time_to_first_token_seconds_bucket{engine="0",le="+Inf",model_name="llm-qwen3-14b"} 2.0
313vllm:time_to_first_token_seconds_count{engine="0",model_name="llm-qwen3-14b"} 2.0
314vllm:time_to_first_token_seconds_sum{engine="0",model_name="llm-qwen3-14b"} 1.4195771217346191
315# HELP vllm:time_to_first_token_seconds_created Histogram of time to first token in seconds.
316# TYPE vllm:time_to_first_token_seconds_created gauge
317vllm:time_to_first_token_seconds_created{engine="0",model_name="llm-qwen3-14b"} 1.783421567078899e+09
318# HELP vllm:inter_token_latency_seconds Histogram of inter-token latency in seconds.
319# TYPE vllm:inter_token_latency_seconds histogram
320vllm:inter_token_latency_seconds_bucket{engine="0",le="0.01",model_name="llm-qwen3-14b"} 16.0
321vllm:inter_token_latency_seconds_bucket{engine="0",le="0.025",model_name="llm-qwen3-14b"} 16.0
322vllm:inter_token_latency_seconds_bucket{engine="0",le="0.05",model_name="llm-qwen3-14b"} 16.0
323vllm:inter_token_latency_seconds_bucket{engine="0",le="0.075",model_name="llm-qwen3-14b"} 16.0
324vllm:inter_token_latency_seconds_bucket{engine="0",le="0.1",model_name="llm-qwen3-14b"} 16.0
325vllm:inter_token_latency_seconds_bucket{engine="0",le="0.15",model_name="llm-qwen3-14b"} 16.0
326vllm:inter_token_latency_seconds_bucket{engine="0",le="0.2",model_name="llm-qwen3-14b"} 16.0
327vllm:inter_token_latency_seconds_bucket{engine="0",le="0.3",model_name="llm-qwen3-14b"} 16.0
328vllm:inter_token_latency_seconds_bucket{engine="0",le="0.4",model_name="llm-qwen3-14b"} 16.0
329vllm:inter_token_latency_seconds_bucket{engine="0",le="0.5",model_name="llm-qwen3-14b"} 16.0
330vllm:inter_token_latency_seconds_bucket{engine="0",le="0.75",model_name="llm-qwen3-14b"} 16.0
331vllm:inter_token_latency_seconds_bucket{engine="0",le="1.0",model_name="llm-qwen3-14b"} 16.0
332vllm:inter_token_latency_seconds_bucket{engine="0",le="2.5",model_name="llm-qwen3-14b"} 16.0
333vllm:inter_token_latency_seconds_bucket{engine="0",le="5.0",model_name="llm-qwen3-14b"} 16.0
334vllm:inter_token_latency_seconds_bucket{engine="0",le="7.5",model_name="llm-qwen3-14b"} 16.0
335vllm:inter_token_latency_seconds_bucket{engine="0",le="10.0",model_name="llm-qwen3-14b"} 16.0
336vllm:inter_token_latency_seconds_bucket{engine="0",le="20.0",model_name="llm-qwen3-14b"} 16.0
337vllm:inter_token_latency_seconds_bucket{engine="0",le="40.0",model_name="llm-qwen3-14b"} 16.0
338vllm:inter_token_latency_seconds_bucket{engine="0",le="80.0",model_name="llm-qwen3-14b"} 16.0
339vllm:inter_token_latency_seconds_bucket{engine="0",le="+Inf",model_name="llm-qwen3-14b"} 16.0
340vllm:inter_token_latency_seconds_count{engine="0",model_name="llm-qwen3-14b"} 16.0
341vllm:inter_token_latency_seconds_sum{engine="0",model_name="llm-qwen3-14b"} 0.13465193193405867
342# HELP vllm:inter_token_latency_seconds_created Histogram of inter-token latency in seconds.
343# TYPE vllm:inter_token_latency_seconds_created gauge
344vllm:inter_token_latency_seconds_created{engine="0",model_name="llm-qwen3-14b"} 1.7834215670789344e+09
345# HELP vllm:request_time_per_output_token_seconds Histogram of time_per_output_token_seconds per request.
346# TYPE vllm:request_time_per_output_token_seconds histogram
347vllm:request_time_per_output_token_seconds_bucket{engine="0",le="0.01",model_name="llm-qwen3-14b"} 2.0
348vllm:request_time_per_output_token_seconds_bucket{engine="0",le="0.025",model_name="llm-qwen3-14b"} 2.0
349vllm:request_time_per_output_token_seconds_bucket{engine="0",le="0.05",model_name="llm-qwen3-14b"} 2.0
350vllm:request_time_per_output_token_seconds_bucket{engine="0",le="0.075",model_name="llm-qwen3-14b"} 2.0
351vllm:request_time_per_output_token_seconds_bucket{engine="0",le="0.1",model_name="llm-qwen3-14b"} 2.0
352vllm:request_time_per_output_token_seconds_bucket{engine="0",le="0.15",model_name="llm-qwen3-14b"} 2.0
353vllm:request_time_per_output_token_seconds_bucket{engine="0",le="0.2",model_name="llm-qwen3-14b"} 2.0
354vllm:request_time_per_output_token_seconds_bucket{engine="0",le="0.3",model_name="llm-qwen3-14b"} 2.0
355vllm:request_time_per_output_token_seconds_bucket{engine="0",le="0.4",model_name="llm-qwen3-14b"} 2.0
356vllm:request_time_per_output_token_seconds_bucket{engine="0",le="0.5",model_name="llm-qwen3-14b"} 2.0
357vllm:request_time_per_output_token_seconds_bucket{engine="0",le="0.75",model_name="llm-qwen3-14b"} 2.0
358vllm:request_time_per_output_token_seconds_bucket{engine="0",le="1.0",model_name="llm-qwen3-14b"} 2.0
359vllm:request_time_per_output_token_seconds_bucket{engine="0",le="2.5",model_name="llm-qwen3-14b"} 2.0
360vllm:request_time_per_output_token_seconds_bucket{engine="0",le="5.0",model_name="llm-qwen3-14b"} 2.0
361vllm:request_time_per_output_token_seconds_bucket{engine="0",le="7.5",model_name="llm-qwen3-14b"} 2.0
362vllm:request_time_per_output_token_seconds_bucket{engine="0",le="10.0",model_name="llm-qwen3-14b"} 2.0
363vllm:request_time_per_output_token_seconds_bucket{engine="0",le="20.0",model_name="llm-qwen3-14b"} 2.0
364vllm:request_time_per_output_token_seconds_bucket{engine="0",le="40.0",model_name="llm-qwen3-14b"} 2.0
365vllm:request_time_per_output_token_seconds_bucket{engine="0",le="80.0",model_name="llm-qwen3-14b"} 2.0
366vllm:request_time_per_output_token_seconds_bucket{engine="0",le="+Inf",model_name="llm-qwen3-14b"} 2.0
367vllm:request_time_per_output_token_seconds_count{engine="0",model_name="llm-qwen3-14b"} 2.0
368vllm:request_time_per_output_token_seconds_sum{engine="0",model_name="llm-qwen3-14b"} 0.016831491491757333
369# HELP vllm:request_time_per_output_token_seconds_created Histogram of time_per_output_token_seconds per request.
370# TYPE vllm:request_time_per_output_token_seconds_created gauge
371vllm:request_time_per_output_token_seconds_created{engine="0",model_name="llm-qwen3-14b"} 1.7834215670789697e+09
372# HELP vllm:e2e_request_latency_seconds Histogram of e2e request latency in seconds.
373# TYPE vllm:e2e_request_latency_seconds histogram
374vllm:e2e_request_latency_seconds_bucket{engine="0",le="0.3",model_name="llm-qwen3-14b"} 1.0
375vllm:e2e_request_latency_seconds_bucket{engine="0",le="0.5",model_name="llm-qwen3-14b"} 1.0
376vllm:e2e_request_latency_seconds_bucket{engine="0",le="0.8",model_name="llm-qwen3-14b"} 1.0
377vllm:e2e_request_latency_seconds_bucket{engine="0",le="1.0",model_name="llm-qwen3-14b"} 1.0
378vllm:e2e_request_latency_seconds_bucket{engine="0",le="1.5",model_name="llm-qwen3-14b"} 2.0
379vllm:e2e_request_latency_seconds_bucket{engine="0",le="2.0",model_name="llm-qwen3-14b"} 2.0
380vllm:e2e_request_latency_seconds_bucket{engine="0",le="2.5",model_name="llm-qwen3-14b"} 2.0
381vllm:e2e_request_latency_seconds_bucket{engine="0",le="5.0",model_name="llm-qwen3-14b"} 2.0
382vllm:e2e_request_latency_seconds_bucket{engine="0",le="10.0",model_name="llm-qwen3-14b"} 2.0
383vllm:e2e_request_latency_seconds_bucket{engine="0",le="15.0",model_name="llm-qwen3-14b"} 2.0
384vllm:e2e_request_latency_seconds_bucket{engine="0",le="20.0",model_name="llm-qwen3-14b"} 2.0
385vllm:e2e_request_latency_seconds_bucket{engine="0",le="30.0",model_name="llm-qwen3-14b"} 2.0
386vllm:e2e_request_latency_seconds_bucket{engine="0",le="40.0",model_name="llm-qwen3-14b"} 2.0
387vllm:e2e_request_latency_seconds_bucket{engine="0",le="50.0",model_name="llm-qwen3-14b"} 2.0
388vllm:e2e_request_latency_seconds_bucket{engine="0",le="60.0",model_name="llm-qwen3-14b"} 2.0
389vllm:e2e_request_latency_seconds_bucket{engine="0",le="120.0",model_name="llm-qwen3-14b"} 2.0
390vllm:e2e_request_latency_seconds_bucket{engine="0",le="240.0",model_name="llm-qwen3-14b"} 2.0
391vllm:e2e_request_latency_seconds_bucket{engine="0",le="480.0",model_name="llm-qwen3-14b"} 2.0
392vllm:e2e_request_latency_seconds_bucket{engine="0",le="960.0",model_name="llm-qwen3-14b"} 2.0
393vllm:e2e_request_latency_seconds_bucket{engine="0",le="1920.0",model_name="llm-qwen3-14b"} 2.0
394vllm:e2e_request_latency_seconds_bucket{engine="0",le="7680.0",model_name="llm-qwen3-14b"} 2.0
395vllm:e2e_request_latency_seconds_bucket{engine="0",le="+Inf",model_name="llm-qwen3-14b"} 2.0
396vllm:e2e_request_latency_seconds_count{engine="0",model_name="llm-qwen3-14b"} 2.0
397vllm:e2e_request_latency_seconds_sum{engine="0",model_name="llm-qwen3-14b"} 1.553670883178711
398# HELP vllm:e2e_request_latency_seconds_created Histogram of e2e request latency in seconds.
399# TYPE vllm:e2e_request_latency_seconds_created gauge
400vllm:e2e_request_latency_seconds_created{engine="0",model_name="llm-qwen3-14b"} 1.7834215670789983e+09
401# HELP vllm:request_queue_time_seconds Histogram of time spent in WAITING phase for request.
402# TYPE vllm:request_queue_time_seconds histogram
403vllm:request_queue_time_seconds_bucket{engine="0",le="0.3",model_name="llm-qwen3-14b"} 2.0
404vllm:request_queue_time_seconds_bucket{engine="0",le="0.5",model_name="llm-qwen3-14b"} 2.0
405vllm:request_queue_time_seconds_bucket{engine="0",le="0.8",model_name="llm-qwen3-14b"} 2.0
406vllm:request_queue_time_seconds_bucket{engine="0",le="1.0",model_name="llm-qwen3-14b"} 2.0
407vllm:request_queue_time_seconds_bucket{engine="0",le="1.5",model_name="llm-qwen3-14b"} 2.0
408vllm:request_queue_time_seconds_bucket{engine="0",le="2.0",model_name="llm-qwen3-14b"} 2.0
409vllm:request_queue_time_seconds_bucket{engine="0",le="2.5",model_name="llm-qwen3-14b"} 2.0
410vllm:request_queue_time_seconds_bucket{engine="0",le="5.0",model_name="llm-qwen3-14b"} 2.0
411vllm:request_queue_time_seconds_bucket{engine="0",le="10.0",model_name="llm-qwen3-14b"} 2.0
412vllm:request_queue_time_seconds_bucket{engine="0",le="15.0",model_name="llm-qwen3-14b"} 2.0
413vllm:request_queue_time_seconds_bucket{engine="0",le="20.0",model_name="llm-qwen3-14b"} 2.0
414vllm:request_queue_time_seconds_bucket{engine="0",le="30.0",model_name="llm-qwen3-14b"} 2.0
415vllm:request_queue_time_seconds_bucket{engine="0",le="40.0",model_name="llm-qwen3-14b"} 2.0
416vllm:request_queue_time_seconds_bucket{engine="0",le="50.0",model_name="llm-qwen3-14b"} 2.0
417vllm:request_queue_time_seconds_bucket{engine="0",le="60.0",model_name="llm-qwen3-14b"} 2.0
418vllm:request_queue_time_seconds_bucket{engine="0",le="120.0",model_name="llm-qwen3-14b"} 2.0
419vllm:request_queue_time_seconds_bucket{engine="0",le="240.0",model_name="llm-qwen3-14b"} 2.0
420vllm:request_queue_time_seconds_bucket{engine="0",le="480.0",model_name="llm-qwen3-14b"} 2.0
421vllm:request_queue_time_seconds_bucket{engine="0",le="960.0",model_name="llm-qwen3-14b"} 2.0
422vllm:request_queue_time_seconds_bucket{engine="0",le="1920.0",model_name="llm-qwen3-14b"} 2.0
423vllm:request_queue_time_seconds_bucket{engine="0",le="7680.0",model_name="llm-qwen3-14b"} 2.0
424vllm:request_queue_time_seconds_bucket{engine="0",le="+Inf",model_name="llm-qwen3-14b"} 2.0
425vllm:request_queue_time_seconds_count{engine="0",model_name="llm-qwen3-14b"} 2.0
426vllm:request_queue_time_seconds_sum{engine="0",model_name="llm-qwen3-14b"} 2.9673799872398376e-05
427# HELP vllm:request_queue_time_seconds_created Histogram of time spent in WAITING phase for request.
428# TYPE vllm:request_queue_time_seconds_created gauge
429vllm:request_queue_time_seconds_created{engine="0",model_name="llm-qwen3-14b"} 1.7834215670790312e+09
430# HELP vllm:request_inference_time_seconds Histogram of time spent in RUNNING phase for request.
431# TYPE vllm:request_inference_time_seconds histogram
432vllm:request_inference_time_seconds_bucket{engine="0",le="0.3",model_name="llm-qwen3-14b"} 1.0
433vllm:request_inference_time_seconds_bucket{engine="0",le="0.5",model_name="llm-qwen3-14b"} 1.0
434vllm:request_inference_time_seconds_bucket{engine="0",le="0.8",model_name="llm-qwen3-14b"} 1.0
435vllm:request_inference_time_seconds_bucket{engine="0",le="1.0",model_name="llm-qwen3-14b"} 1.0
436vllm:request_inference_time_seconds_bucket{engine="0",le="1.5",model_name="llm-qwen3-14b"} 2.0
437vllm:request_inference_time_seconds_bucket{engine="0",le="2.0",model_name="llm-qwen3-14b"} 2.0
438vllm:request_inference_time_seconds_bucket{engine="0",le="2.5",model_name="llm-qwen3-14b"} 2.0
439vllm:request_inference_time_seconds_bucket{engine="0",le="5.0",model_name="llm-qwen3-14b"} 2.0
440vllm:request_inference_time_seconds_bucket{engine="0",le="10.0",model_name="llm-qwen3-14b"} 2.0
441vllm:request_inference_time_seconds_bucket{engine="0",le="15.0",model_name="llm-qwen3-14b"} 2.0
442vllm:request_inference_time_seconds_bucket{engine="0",le="20.0",model_name="llm-qwen3-14b"} 2.0
443vllm:request_inference_time_seconds_bucket{engine="0",le="30.0",model_name="llm-qwen3-14b"} 2.0
444vllm:request_inference_time_seconds_bucket{engine="0",le="40.0",model_name="llm-qwen3-14b"} 2.0
445vllm:request_inference_time_seconds_bucket{engine="0",le="50.0",model_name="llm-qwen3-14b"} 2.0
446vllm:request_inference_time_seconds_bucket{engine="0",le="60.0",model_name="llm-qwen3-14b"} 2.0
447vllm:request_inference_time_seconds_bucket{engine="0",le="120.0",model_name="llm-qwen3-14b"} 2.0
448vllm:request_inference_time_seconds_bucket{engine="0",le="240.0",model_name="llm-qwen3-14b"} 2.0
449vllm:request_inference_time_seconds_bucket{engine="0",le="480.0",model_name="llm-qwen3-14b"} 2.0
450vllm:request_inference_time_seconds_bucket{engine="0",le="960.0",model_name="llm-qwen3-14b"} 2.0
451vllm:request_inference_time_seconds_bucket{engine="0",le="1920.0",model_name="llm-qwen3-14b"} 2.0
452vllm:request_inference_time_seconds_bucket{engine="0",le="7680.0",model_name="llm-qwen3-14b"} 2.0
453vllm:request_inference_time_seconds_bucket{engine="0",le="+Inf",model_name="llm-qwen3-14b"} 2.0
454vllm:request_inference_time_seconds_count{engine="0",model_name="llm-qwen3-14b"} 2.0
455vllm:request_inference_time_seconds_sum{engine="0",model_name="llm-qwen3-14b"} 1.55032794829458
456# HELP vllm:request_inference_time_seconds_created Histogram of time spent in RUNNING phase for request.
457# TYPE vllm:request_inference_time_seconds_created gauge
458vllm:request_inference_time_seconds_created{engine="0",model_name="llm-qwen3-14b"} 1.7834215670790608e+09
459# HELP vllm:request_prefill_time_seconds Histogram of time spent in PREFILL phase for request.
460# TYPE vllm:request_prefill_time_seconds histogram
461vllm:request_prefill_time_seconds_bucket{engine="0",le="0.3",model_name="llm-qwen3-14b"} 1.0
462vllm:request_prefill_time_seconds_bucket{engine="0",le="0.5",model_name="llm-qwen3-14b"} 1.0
463vllm:request_prefill_time_seconds_bucket{engine="0",le="0.8",model_name="llm-qwen3-14b"} 1.0
464vllm:request_prefill_time_seconds_bucket{engine="0",le="1.0",model_name="llm-qwen3-14b"} 1.0
465vllm:request_prefill_time_seconds_bucket{engine="0",le="1.5",model_name="llm-qwen3-14b"} 2.0
466vllm:request_prefill_time_seconds_bucket{engine="0",le="2.0",model_name="llm-qwen3-14b"} 2.0
467vllm:request_prefill_time_seconds_bucket{engine="0",le="2.5",model_name="llm-qwen3-14b"} 2.0
468vllm:request_prefill_time_seconds_bucket{engine="0",le="5.0",model_name="llm-qwen3-14b"} 2.0
469vllm:request_prefill_time_seconds_bucket{engine="0",le="10.0",model_name="llm-qwen3-14b"} 2.0
470vllm:request_prefill_time_seconds_bucket{engine="0",le="15.0",model_name="llm-qwen3-14b"} 2.0
471vllm:request_prefill_time_seconds_bucket{engine="0",le="20.0",model_name="llm-qwen3-14b"} 2.0
472vllm:request_prefill_time_seconds_bucket{engine="0",le="30.0",model_name="llm-qwen3-14b"} 2.0
473vllm:request_prefill_time_seconds_bucket{engine="0",le="40.0",model_name="llm-qwen3-14b"} 2.0
474vllm:request_prefill_time_seconds_bucket{engine="0",le="50.0",model_name="llm-qwen3-14b"} 2.0
475vllm:request_prefill_time_seconds_bucket{engine="0",le="60.0",model_name="llm-qwen3-14b"} 2.0
476vllm:request_prefill_time_seconds_bucket{engine="0",le="120.0",model_name="llm-qwen3-14b"} 2.0
477vllm:request_prefill_time_seconds_bucket{engine="0",le="240.0",model_name="llm-qwen3-14b"} 2.0
478vllm:request_prefill_time_seconds_bucket{engine="0",le="480.0",model_name="llm-qwen3-14b"} 2.0
479vllm:request_prefill_time_seconds_bucket{engine="0",le="960.0",model_name="llm-qwen3-14b"} 2.0
480vllm:request_prefill_time_seconds_bucket{engine="0",le="1920.0",model_name="llm-qwen3-14b"} 2.0
481vllm:request_prefill_time_seconds_bucket{engine="0",le="7680.0",model_name="llm-qwen3-14b"} 2.0
482vllm:request_prefill_time_seconds_bucket{engine="0",le="+Inf",model_name="llm-qwen3-14b"} 2.0
483vllm:request_prefill_time_seconds_count{engine="0",model_name="llm-qwen3-14b"} 2.0
484vllm:request_prefill_time_seconds_sum{engine="0",model_name="llm-qwen3-14b"} 1.4156760163605213
485# HELP vllm:request_prefill_time_seconds_created Histogram of time spent in PREFILL phase for request.
486# TYPE vllm:request_prefill_time_seconds_created gauge
487vllm:request_prefill_time_seconds_created{engine="0",model_name="llm-qwen3-14b"} 1.7834215670790908e+09
488# HELP vllm:request_decode_time_seconds Histogram of time spent in DECODE phase for request.
489# TYPE vllm:request_decode_time_seconds histogram
490vllm:request_decode_time_seconds_bucket{engine="0",le="0.3",model_name="llm-qwen3-14b"} 2.0
491vllm:request_decode_time_seconds_bucket{engine="0",le="0.5",model_name="llm-qwen3-14b"} 2.0
492vllm:request_decode_time_seconds_bucket{engine="0",le="0.8",model_name="llm-qwen3-14b"} 2.0
493vllm:request_decode_time_seconds_bucket{engine="0",le="1.0",model_name="llm-qwen3-14b"} 2.0
494vllm:request_decode_time_seconds_bucket{engine="0",le="1.5",model_name="llm-qwen3-14b"} 2.0
495vllm:request_decode_time_seconds_bucket{engine="0",le="2.0",model_name="llm-qwen3-14b"} 2.0
496vllm:request_decode_time_seconds_bucket{engine="0",le="2.5",model_name="llm-qwen3-14b"} 2.0
497vllm:request_decode_time_seconds_bucket{engine="0",le="5.0",model_name="llm-qwen3-14b"} 2.0
498vllm:request_decode_time_seconds_bucket{engine="0",le="10.0",model_name="llm-qwen3-14b"} 2.0
499vllm:request_decode_time_seconds_bucket{engine="0",le="15.0",model_name="llm-qwen3-14b"} 2.0
500vllm:request_decode_time_seconds_bucket{engine="0",le="20.0",model_name="llm-qwen3-14b"} 2.0
501vllm:request_decode_time_seconds_bucket{engine="0",le="30.0",model_name="llm-qwen3-14b"} 2.0
502vllm:request_decode_time_seconds_bucket{engine="0",le="40.0",model_name="llm-qwen3-14b"} 2.0
503vllm:request_decode_time_seconds_bucket{engine="0",le="50.0",model_name="llm-qwen3-14b"} 2.0
504vllm:request_decode_time_seconds_bucket{engine="0",le="60.0",model_name="llm-qwen3-14b"} 2.0
505vllm:request_decode_time_seconds_bucket{engine="0",le="120.0",model_name="llm-qwen3-14b"} 2.0
506vllm:request_decode_time_seconds_bucket{engine="0",le="240.0",model_name="llm-qwen3-14b"} 2.0
507vllm:request_decode_time_seconds_bucket{engine="0",le="480.0",model_name="llm-qwen3-14b"} 2.0
508vllm:request_decode_time_seconds_bucket{engine="0",le="960.0",model_name="llm-qwen3-14b"} 2.0
509vllm:request_decode_time_seconds_bucket{engine="0",le="1920.0",model_name="llm-qwen3-14b"} 2.0
510vllm:request_decode_time_seconds_bucket{engine="0",le="7680.0",model_name="llm-qwen3-14b"} 2.0
511vllm:request_decode_time_seconds_bucket{engine="0",le="+Inf",model_name="llm-qwen3-14b"} 2.0
512vllm:request_decode_time_seconds_count{engine="0",model_name="llm-qwen3-14b"} 2.0
513vllm:request_decode_time_seconds_sum{engine="0",model_name="llm-qwen3-14b"} 0.13465193193405867
514# HELP vllm:request_decode_time_seconds_created Histogram of time spent in DECODE phase for request.
515# TYPE vllm:request_decode_time_seconds_created gauge
516vllm:request_decode_time_seconds_created{engine="0",model_name="llm-qwen3-14b"} 1.7834215670791585e+09
517# HELP vllm:request_prefill_kv_computed_tokens Histogram of new KV tokens computed during prefill (excluding cached tokens).
518# TYPE vllm:request_prefill_kv_computed_tokens histogram
519vllm:request_prefill_kv_computed_tokens_bucket{engine="0",le="1.0",model_name="llm-qwen3-14b"} 0.0
520vllm:request_prefill_kv_computed_tokens_bucket{engine="0",le="2.0",model_name="llm-qwen3-14b"} 0.0
521vllm:request_prefill_kv_computed_tokens_bucket{engine="0",le="5.0",model_name="llm-qwen3-14b"} 0.0
522vllm:request_prefill_kv_computed_tokens_bucket{engine="0",le="10.0",model_name="llm-qwen3-14b"} 0.0
523vllm:request_prefill_kv_computed_tokens_bucket{engine="0",le="20.0",model_name="llm-qwen3-14b"} 2.0
524vllm:request_prefill_kv_computed_tokens_bucket{engine="0",le="50.0",model_name="llm-qwen3-14b"} 2.0
525vllm:request_prefill_kv_computed_tokens_bucket{engine="0",le="100.0",model_name="llm-qwen3-14b"} 2.0
526vllm:request_prefill_kv_computed_tokens_bucket{engine="0",le="200.0",model_name="llm-qwen3-14b"} 2.0
527vllm:request_prefill_kv_computed_tokens_bucket{engine="0",le="500.0",model_name="llm-qwen3-14b"} 2.0
528vllm:request_prefill_kv_computed_tokens_bucket{engine="0",le="1000.0",model_name="llm-qwen3-14b"} 2.0
529vllm:request_prefill_kv_computed_tokens_bucket{engine="0",le="2000.0",model_name="llm-qwen3-14b"} 2.0
530vllm:request_prefill_kv_computed_tokens_bucket{engine="0",le="5000.0",model_name="llm-qwen3-14b"} 2.0
531vllm:request_prefill_kv_computed_tokens_bucket{engine="0",le="10000.0",model_name="llm-qwen3-14b"} 2.0
532vllm:request_prefill_kv_computed_tokens_bucket{engine="0",le="20000.0",model_name="llm-qwen3-14b"} 2.0
533vllm:request_prefill_kv_computed_tokens_bucket{engine="0",le="+Inf",model_name="llm-qwen3-14b"} 2.0
534vllm:request_prefill_kv_computed_tokens_count{engine="0",model_name="llm-qwen3-14b"} 2.0
535vllm:request_prefill_kv_computed_tokens_sum{engine="0",model_name="llm-qwen3-14b"} 38.0
536# HELP vllm:request_prefill_kv_computed_tokens_created Histogram of new KV tokens computed during prefill (excluding cached tokens).
537# TYPE vllm:request_prefill_kv_computed_tokens_created gauge
538vllm:request_prefill_kv_computed_tokens_created{engine="0",model_name="llm-qwen3-14b"} 1.783421567079191e+09
539# HELP vllm:cache_config_info Information of the LLMEngine CacheConfig
540# TYPE vllm:cache_config_info gauge
541vllm:cache_config_info{_block_size_resolved="True",block_size="64",cache_dtype="auto",calculate_kv_scales="False",enable_prefix_caching="True",engine="0",gpu_memory_utilization="0.85",hash_block_size="None",is_attention_free="False",kv_cache_dtype_skip_layers="[]",kv_cache_memory_bytes="None",kv_offloading_backend="native",kv_offloading_size="None",kv_sharing_fast_prefill="False",mamba_block_size="None",mamba_cache_dtype="auto",mamba_cache_mode="none",mamba_page_size_padded="None",mamba_ssm_cache_dtype="auto",num_cpu_blocks="None",num_gpu_blocks="13422",num_gpu_blocks_override="None",prefix_caching_hash_algo="sha256",sliding_window="None",user_specified_block_size="True",user_specified_mamba_block_size="False"} 1.0

Demo grafana dashboard showing the vLLM metrics we worked on this task: T429597#12094788