Page MenuHomePhabricator
Paste P93837

cope-b-a4b LLM loaded successfully in prod experimental ns on MI300x GPU
ActivePublic
Actions

Authored by kevinbazira on Jun 4 2026, 6:22 AM.
$ kubectl logs cope-b-a4b-predictor-00002-deployment-869b9cd8fc-b7znb
+ source common_settings.sh
+++ /usr/bin/python3 -c 'from python.resource_utils import get_cpu_count; print(get_cpu_count())'
++ CPU_COUNT=6
++ echo 'CPU count detected from get_cpu_count: 6'
++ export OMP_NUM_THREADS=6
++ OMP_NUM_THREADS=6
++ echo 'OMP_NUM_THREADS set to: 6'
+ MODEL_SERVER_PATH=model.py
+ exec /usr/bin/python3 model.py
CPU count detected from get_cpu_count: 6
OMP_NUM_THREADS set to: 6
INFO:root:Loading tokenizer...
/opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
INFO:root:Loading model...
[transformers] `torch_dtype` is deprecated! Use `dtype` instead!
Loading weights: 100%|██████████| 657/657 [00:21<00:00, 29.90it/s]
INFO:root:Model loaded successfully!
INFO:kserve:Registering model: cope-b-a4b
INFO:kserve:Setting max asyncio worker threads as 32
INFO:kserve:OpenAI endpoints registered
INFO:kserve:Time series endpoints not registered
INFO:kserve:Starting uvicorn with 1 workers
INFO:uvicorn.error:Started server process [1]
INFO:uvicorn.error:Waiting for application startup.
INFO:kserve:Starting gRPC server with 4 workers
INFO:kserve:Starting gRPC server on [::]:8081
INFO:uvicorn.error:Application startup complete.
INFO:uvicorn.error:Uvicorn running on http://0.0.0.0:8080 (Press CTRL+C to quit)
INFO:uvicorn.access:127.0.0.6:50733 - "GET /metrics HTTP/1.1" 200
INFO:kserve.trace:kserve.io.kserve.protocol.rest.server.metrics_handler: 0.0025026798248291016 ['http_status:200', 'http_method:GET', 'time:wall']
INFO:kserve.trace:kserve.io.kserve.protocol.rest.server.metrics_handler: 0.002496999999998195 ['http_status:200', 'http_method:GET', 'time:cpu']
INFO:uvicorn.access:127.0.0.6:48781 - "GET /metrics HTTP/1.1" 200
INFO:kserve.trace:kserve.io.kserve.protocol.rest.server.metrics_handler: 0.0045011043548583984 ['http_status:200', 'http_method:GET', 'time:wall']
INFO:kserve.trace:kserve.io.kserve.protocol.rest.server.metrics_handler: 0.004481999999995878 ['http_status:200', 'http_method:GET', 'time:cpu']
INFO:uvicorn.access:127.0.0.6:55693 - "GET /metrics HTTP/1.1" 200
INFO:kserve.trace:kserve.io.kserve.protocol.rest.server.metrics_handler: 0.0027043819427490234 ['http_status:200', 'http_method:GET', 'time:wall']
INFO:kserve.trace:kserve.io.kserve.protocol.rest.server.metrics_handler: 0.0026839999999950237 ['http_status:200', 'http_method:GET', 'time:cpu']
INFO:uvicorn.access:127.0.0.6:51877 - "GET /metrics HTTP/1.1" 200
INFO:kserve.trace:kserve.io.kserve.protocol.rest.server.metrics_handler: 0.0006699562072753906 ['http_status:200', 'http_method:GET', 'time:wall']
INFO:kserve.trace:kserve.io.kserve.protocol.rest.server.metrics_handler: 0.000656999999996799 ['http_status:200', 'http_method:GET', 'time:cpu']
INFO:uvicorn.access:127.0.0.6:60305 - "GET /metrics HTTP/1.1" 200
INFO:kserve.trace:kserve.io.kserve.protocol.rest.server.metrics_handler: 0.0006456375122070312 ['http_status:200', 'http_method:GET', 'time:wall']
INFO:kserve.trace:kserve.io.kserve.protocol.rest.server.metrics_handler: 0.0006330000000005498 ['http_status:200', 'http_method:GET', 'time:cpu']
INFO:uvicorn.access:127.0.0.6:33783 - "GET /metrics HTTP/1.1" 200
INFO:kserve.trace:kserve.io.kserve.protocol.rest.server.metrics_handler: 0.0007698535919189453 ['http_status:200', 'http_method:GET', 'time:wall']
INFO:kserve.trace:kserve.io.kserve.protocol.rest.server.metrics_handler: 0.0007530000000031123 ['http_status:200', 'http_method:GET', 'time:cpu']
INFO:uvicorn.access:127.0.0.6:58125 - "GET /metrics HTTP/1.1" 200
INFO:kserve.trace:kserve.io.kserve.protocol.rest.server.metrics_handler: 0.0006680488586425781 ['http_status:200', 'http_method:GET', 'time:wall']
INFO:kserve.trace:kserve.io.kserve.protocol.rest.server.metrics_handler: 0.0006570000000039045 ['http_status:200', 'http_method:GET', 'time:cpu']
INFO:uvicorn.access:127.0.0.6:49697 - "GET /metrics HTTP/1.1" 200
INFO:kserve.trace:kserve.io.kserve.protocol.rest.server.metrics_handler: 0.0006761550903320312 ['http_status:200', 'http_method:GET', 'time:wall']
INFO:kserve.trace:kserve.io.kserve.protocol.rest.server.metrics_handler: 0.0006619999999983861 ['http_status:200', 'http_method:GET', 'time:cpu']
INFO:uvicorn.access:127.0.0.6:44703 - "GET /metrics HTTP/1.1" 200
INFO:kserve.trace:kserve.io.kserve.protocol.rest.server.metrics_handler: 0.0009989738464355469 ['http_status:200', 'http_method:GET', 'time:wall']
INFO:kserve.trace:kserve.io.kserve.protocol.rest.server.metrics_handler: 0.0009770000000060008 ['http_status:200', 'http_method:GET', 'time:cpu']
INFO:uvicorn.access:127.0.0.6:45231 - "GET /metrics HTTP/1.1" 200
INFO:kserve.trace:kserve.io.kserve.protocol.rest.server.metrics_handler: 0.0006306171417236328 ['http_status:200', 'http_method:GET', 'time:wall']
INFO:kserve.trace:kserve.io.kserve.protocol.rest.server.metrics_handler: 0.0006169999999983133 ['http_status:200', 'http_method:GET', 'time:cpu']
Content licensed under Creative Commons Attribution-ShareAlike (CC BY-SA) 4.0 unless otherwise noted; code licensed under GNU General Public License (GPL) 2.0 or later and other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL · Credits