Page MenuHomePhabricator
Paste P96467

Issues on deploying the Qwen3.8-27b model on llm namespace
ActivePublic
Actions

Authored by gkyziridis on Thu, Sep 17, 11:25 AM.
Project Tags
None
Referenced Files
F103431988: Issues on deploying the Qwen3.8-27b model on llm namespace
Thu, Sep 17, 11:25 AM
Subscribers
None
$ kubectl get isvc llm-qwen38-27b
NAME URL READY PREV LATEST PREVROLLEDOUTREVISION LATESTREADYREVISION AGE
llm-qwen38-27b False 27m
$ kubectl get ksvc llm-qwen38-27b-predictor
NAME URL LATESTCREATED LATESTREADY READY REASON
llm-qwen38-27b-predictor http://llm-qwen38-27b-predictor.llm.wikimedia.org llm-qwen38-27b-predictor-00001 llm-qwen38-27b-predictor-00001 True
$ kubectl get revision | grep qwen38
llm-qwen38-27b-predictor-00001 llm-qwen38-27b-predictor 1 True 1 1
$ kubectl logs llm-qwen38-27b-predictor-00001-deployment-5d77fdc4b5-lw5vn -c kserve-container --tail=5
INFO:kserve.trace:kserve.io.metrics: 0.004782199859619141 ['http_status:200', 'http_method:GET', 'time:wall']
INFO:kserve.trace:kserve.io.metrics: 0.004768999999999579 ['http_status:200', 'http_method:GET', 'time:cpu']
INFO:uvicorn.access:127.0.0.6:53513 - "GET /metrics HTTP/1.1" 200
INFO:kserve.trace:kserve.io.metrics: 0.004786491394042969 ['http_status:200', 'http_method:GET', 'time:wall']
INFO:kserve.trace:kserve.io.metrics: 0.0047739999999976135 ['http_status:200', 'http_method:GET', 'time:cpu']
$ kubectl describe isvc llm-qwen38-27b | tail -40
Severity: Info
Status: Unknown
Type: PredictorRouteReady
Last Transition Time: 2026-09-17T10:50:55Z
Message: fails to reconcile predictor: fails to reconcile knative service: admission webhook "validation.webhook.serving.knative.dev" denied the request: validation callback failed: dry run failed with pods "dry-run-validationkbx84" is forbidden: exceeded quota: quota-compute-resources, requested: limits.cpu=18, used: limits.cpu=145, limited: limits.cpu=150: spec.template
Reason: ReconcileFailed
Status: False
Type: Ready
Last Transition Time: 2026-09-17T10:29:00Z
Reason: PredictorRouteReady not ready
Severity: Info
Status: Unknown
Type: RoutesReady
Last Transition Time: 2026-09-17T10:29:00Z
Severity: Info
Status: False
Type: Stopped
Deployment Mode: Knative
Model Status:
Copies:
Failed Copies: 0
States:
Active Model State:
Target Model State: Pending
Transition Status: InProgress
Observed Generation: 1
Events:
Type Reason Age From Message
---- ------ ---- ---- -------
Warning InternalError 21m v1beta1Controllers fails to reconcile predictor: fails to reconcile knative service: admission webhook "validation.webhook.serving.knative.dev" denied the request: validation callback failed: dry run failed with pods "dry-run-validationzbsds" is forbidden: exceeded quota: quota-compute-resources, requested: limits.cpu=18, used: limits.cpu=145, limited: limits.cpu=150: spec.template
Warning InternalError 21m v1beta1Controllers fails to reconcile predictor: fails to reconcile knative service: admission webhook "validation.webhook.serving.knative.dev" denied the request: validation callback failed: dry run failed with pods "dry-run-validation8scrg" is forbidden: exceeded quota: quota-compute-resources, requested: limits.cpu=18, used: limits.cpu=145, limited: limits.cpu=150: spec.template
Warning InternalError 21m v1beta1Controllers fails to reconcile predictor: fails to reconcile knative service: admission webhook "validation.webhook.serving.knative.dev" denied the request: validation callback failed: dry run failed with pods "dry-run-validationm5vpt" is forbidden: exceeded quota: quota-compute-resources, requested: limits.cpu=18, used: limits.cpu=145, limited: limits.cpu=150: spec.template
Warning InternalError 21m v1beta1Controllers fails to reconcile predictor: fails to reconcile knative service: admission webhook "validation.webhook.serving.knative.dev" denied the request: validation callback failed: dry run failed with pods "dry-run-validation24qv2" is forbidden: exceeded quota: quota-compute-resources, requested: limits.cpu=18, used: limits.cpu=145, limited: limits.cpu=150: spec.template
Warning InternalError 21m v1beta1Controllers fails to reconcile predictor: fails to reconcile knative service: admission webhook "validation.webhook.serving.knative.dev" denied the request: validation callback failed: dry run failed with pods "dry-run-validationtlsrj" is forbidden: exceeded quota: quota-compute-resources, requested: limits.cpu=18, used: limits.cpu=145, limited: limits.cpu=150: spec.template
Warning InternalError 21m v1beta1Controllers fails to reconcile predictor: fails to reconcile knative service: admission webhook "validation.webhook.serving.knative.dev" denied the request: validation callback failed: dry run failed with pods "dry-run-validationlgjsz" is forbidden: exceeded quota: quota-compute-resources, requested: limits.cpu=18, used: limits.cpu=145, limited: limits.cpu=150: spec.template
Warning InternalError 21m v1beta1Controllers fails to reconcile predictor: fails to reconcile knative service: admission webhook "validation.webhook.serving.knative.dev" denied the request: validation callback failed: dry run failed with pods "dry-run-validationdwzmm" is forbidden: exceeded quota: quota-compute-resources, requested: limits.cpu=18, used: limits.cpu=145, limited: limits.cpu=150: spec.template
Warning InternalError 21m v1beta1Controllers fails to reconcile predictor: fails to reconcile knative service: admission webhook "validation.webhook.serving.knative.dev" denied the request: validation callback failed: dry run failed with pods "dry-run-validationqrp2f" is forbidden: exceeded quota: quota-compute-resources, requested: limits.cpu=18, used: limits.cpu=145, limited: limits.cpu=150: spec.template
Warning InternalError 21m v1beta1Controllers fails to reconcile predictor: fails to reconcile knative service: admission webhook "validation.webhook.serving.knative.dev" denied the request: validation callback failed: dry run failed with pods "dry-run-validation44fqh" is forbidden: exceeded quota: quota-compute-resources, requested: limits.cpu=18, used: limits.cpu=145, limited: limits.cpu=150: spec.template
Warning UpdateFailed 21m (x2 over 21m) v1beta1Controllers Failed to update status for InferenceService "llm-qwen38-27b": Operation cannot be fulfilled on inferenceservices.serving.kserve.io "llm-qwen38-27b": the object has been modified; please apply your changes to the latest version and try again
Warning InternalError 115s (x5947 over 21m) v1beta1Controllers (combined from similar events): fails to reconcile predictor: fails to reconcile knative service: admission webhook "validation.webhook.serving.knative.dev" denied the request: validation callback failed: dry run failed with pods "dry-run-validationgx479" is forbidden: exceeded quota: quota-compute-resources, requested: limits.cpu=18, used: limits.cpu=145, limited: limits.cpu=150: spec.template
Content licensed under Creative Commons Attribution-ShareAlike (CC BY-SA) 4.0 unless otherwise noted; code licensed under GNU General Public License (GPL) 2.0 or later and other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL · Credits