Page MenuHomePhabricator
Paste P79250

reference-need-predictor alert logstash logs between 10:34:03 and 10:34:04
ActivePublic
Actions

Authored by kevinbazira on Jul 16 2025, 3:34 PM.
"","Time","kubernetes.pod\_name","kubernetes.container\_name","message","msg","log\_level","log"
"","Jul 16, 2025 @ 10:34:03.864","reference-need-predictor-00012-deployment-7d54649454-z9bn9","kserve-container","\-","\-","\-","INFO:root:Function run\_in\_process\_pool took 626.94 seconds to execute."
"","Jul 16, 2025 @ 10:34:03.864","reference-need-predictor-00012-deployment-7d54649454-z9bn9","kserve-container","\-","\-","\-","2025-07-16 10:34:03.821 kserve.trace requestId: fdb4c3da-e89a-45e6-b187-afac9dbbbf78, preprocess\_ms: 73.625326157, explain\_ms: 0, predict\_ms: 626940.682649612, postprocess\_ms: 0.042676926"
"","Jul 16, 2025 @ 10:34:03.864","reference-need-predictor-00012-deployment-7d54649454-z9bn9","kserve-container","\-","\-","\-","2025-07-16 10:34:03.822 kserve.trace kserve.io.kserve.protocol.rest.v1\_endpoints.predict: 627.0154151916504"
"","Jul 16, 2025 @ 10:34:03.864","reference-need-predictor-00012-deployment-7d54649454-z9bn9","kserve-container","\-","\-","\-","2025-07-16 10:34:03.822 kserve.trace kserve.io.kserve.protocol.rest.v1\_endpoints.predict: 11.58053300000006"
"","Jul 16, 2025 @ 10:34:03.900","reference-need-predictor-00012-deployment-7d54649454-wwfwm","kserve-container","\-","\-","\-","INFO:root:Received request for revision 146201197 (ru)."
"","Jul 16, 2025 @ 10:34:03.901","reference-need-predictor-00012-deployment-7d54649454-wwfwm","kserve-container","\-","\-","\-","2025-07-16 10:34:03.850 kserve.trace kserve.io.kserve.protocol.rest.v1\_endpoints.predict: 388.14414405822754"
"","Jul 16, 2025 @ 10:34:03.901","reference-need-predictor-00012-deployment-7d54649454-wwfwm","kserve-container","\-","\-","\-","INFO:root:Function run\_in\_process\_pool took 388.08 seconds to execute."
"","Jul 16, 2025 @ 10:34:03.901","reference-need-predictor-00012-deployment-7d54649454-wwfwm","kserve-container","\-","\-","\-","2025-07-16 10:34:03.850 kserve.trace requestId: 6ec210fd-ae37-4b8b-8df3-ebfaa0e3647b, preprocess\_ms: 58.975934982, explain\_ms: 0, predict\_ms: 388084.676742554, postprocess\_ms: 0.00834465"
"","Jul 16, 2025 @ 10:34:03.901","reference-need-predictor-00012-deployment-7d54649454-wwfwm","kserve-container","\-","\-","\-","2025-07-16 10:34:03.850 kserve.trace kserve.io.kserve.protocol.rest.v1\_endpoints.predict: 4.374400999999999"
"","Jul 16, 2025 @ 10:34:03.947","reference-need-predictor-00012-deployment-7d54649454-tj5z5","kserve-container","\-","\-","\-","INFO:root:Received request for revision 146180898 (ru)."
"","Jul 16, 2025 @ 10:34:03.994","reference-need-predictor-00012-deployment-7d54649454-z9bn9","kserve-container","\-","\-","\-","INFO:root:Received request for revision 146193976 (ru)."
"","Jul 16, 2025 @ 10:34:04.011","reference-need-predictor-00012-deployment-7d54649454-tj5z5","kserve-container","\-","\-","\-","INFO:root:Received request for revision 146180765 (ru)."
"","Jul 16, 2025 @ 10:34:04.049","reference-need-predictor-00012-deployment-7d54649454-z9bn9","kserve-container","\-","\-","\-","INFO:root:Received request for revision 146193512 (ru)."
"","Jul 16, 2025 @ 10:34:04.085","reference-need-predictor-00012-deployment-7d54649454-tj5z5","kserve-container","\-","\-","\-","INFO:root:Received request for revision 146180861 (ru)."
"","Jul 16, 2025 @ 10:34:04.098","reference-need-predictor-00012-deployment-7d54649454-z9bn9","kserve-container","\-","\-","\-","INFO:root:Received request for revision 146193331 (ru)."
"","Jul 16, 2025 @ 10:34:04.210","reference-need-predictor-00012-deployment-7d54649454-2kdnd","kserve-container","\-","\-","\-","INFO:root:Received request for revision 146170512 (ru)."
"","Jul 16, 2025 @ 10:34:04.326","reference-need-predictor-00012-deployment-7d54649454-jrbmc","kserve-container","\-","\-","\-","Worker initialized with model version: 0"
"","Jul 16, 2025 @ 10:34:04.405","reference-need-predictor-00012-deployment-7d54649454-jrbmc","kserve-container","\-","\-","\-","2025-07-16 10:34:04.362 1 kserve ERROR \[errors.py:inference\_error\_handler():104\] Exception:"
"","Jul 16, 2025 @ 10:34:04.405","reference-need-predictor-00012-deployment-7d54649454-jrbmc","kserve-container","\-","\-","\-","Traceback (most recent call last):"
"","Jul 16, 2025 @ 10:34:04.405","reference-need-predictor-00012-deployment-7d54649454-jrbmc","kserve-container","\-","\-","\-","File "/srv/ref\_quality/model\_server/model.py", line 60, in async\_classify"
"","Jul 16, 2025 @ 10:34:04.405","reference-need-predictor-00012-deployment-7d54649454-jrbmc","kserve-container","\-","\-","\-","result, model\_version = await process\_utils.run\_in\_process\_pool("
"","Jul 16, 2025 @ 10:34:04.405","reference-need-predictor-00012-deployment-7d54649454-jrbmc","kserve-container","\-","\-","\-","^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^"
"","Jul 16, 2025 @ 10:34:04.405","reference-need-predictor-00012-deployment-7d54649454-jrbmc","kserve-container","\-","\-","\-","INFO:root:Create a process pool of 2 workers to support model scoring blocking code."
"","Jul 16, 2025 @ 10:34:04.405","reference-need-predictor-00012-deployment-7d54649454-jrbmc","kserve-container","\-","\-","\-","File "/srv/ref\_quality/python/decorators.py", line 26, in elapsed\_time\_wrapper"
"","Jul 16, 2025 @ 10:34:04.405","reference-need-predictor-00012-deployment-7d54649454-jrbmc","kserve-container","\-","\-","\-","result = await func(\*args, \*\*kwargs)"
"","Jul 16, 2025 @ 10:34:04.405","reference-need-predictor-00012-deployment-7d54649454-jrbmc","kserve-container","\-","\-","\-","^^^^^^^^^^^^^^^^^^^^^^^^^^^"
"","Jul 16, 2025 @ 10:34:04.405","reference-need-predictor-00012-deployment-7d54649454-jrbmc","kserve-container","\-","\-","\-","File "/srv/ref\_quality/python/process\_utils.py", line 75, in run\_in\_process\_pool"
"","Jul 16, 2025 @ 10:34:04.405","reference-need-predictor-00012-deployment-7d54649454-jrbmc","kserve-container","\-","\-","\-","return await asyncio.get\_event\_loop().run\_in\_executor("
"","Jul 16, 2025 @ 10:34:04.406","reference-need-predictor-00012-deployment-7d54649454-jrbmc","kserve-container","\-","\-","\-","File "/opt/lib/venv/lib/python3.11/site-packages/starlette/routing.py", line 74, in app"
"","Jul 16, 2025 @ 10:34:04.406","reference-need-predictor-00012-deployment-7d54649454-jrbmc","kserve-container","\-","\-","\-","^^^^^^^^^^^^^^^^^^^^^^^^^^^^"
"","Jul 16, 2025 @ 10:34:04.406","reference-need-predictor-00012-deployment-7d54649454-jrbmc","kserve-container","\-","\-","\-","File "/opt/lib/venv/lib/python3.11/site-packages/fastapi/routing.py", line 191, in run\_endpoint\_function"
"","Jul 16, 2025 @ 10:34:04.406","reference-need-predictor-00012-deployment-7d54649454-jrbmc","kserve-container","\-","\-","\-","^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^"
"","Jul 16, 2025 @ 10:34:04.406","reference-need-predictor-00012-deployment-7d54649454-jrbmc","kserve-container","\-","\-","\-","concurrent.futures.process.BrokenProcessPool: A process in the process pool was terminated abruptly while the future was running or pending."
"","Jul 16, 2025 @ 10:34:04.406","reference-need-predictor-00012-deployment-7d54649454-jrbmc","kserve-container","\-","\-","\-","During handling of the above exception, another exception occurred:"
"","Jul 16, 2025 @ 10:34:04.406","reference-need-predictor-00012-deployment-7d54649454-jrbmc","kserve-container","\-","\-","\-","response = await func(request)"
"","Jul 16, 2025 @ 10:34:04.406","reference-need-predictor-00012-deployment-7d54649454-jrbmc","kserve-container","\-","\-","\-","File "/opt/lib/venv/lib/python3.11/site-packages/fastapi/routing.py", line 299, in app"
"","Jul 16, 2025 @ 10:34:04.406","reference-need-predictor-00012-deployment-7d54649454-jrbmc","kserve-container","\-","\-","\-",""
"","Jul 16, 2025 @ 10:34:04.406","reference-need-predictor-00012-deployment-7d54649454-jrbmc","kserve-container","\-","\-","\-",""
"","Jul 16, 2025 @ 10:34:04.406","reference-need-predictor-00012-deployment-7d54649454-jrbmc","kserve-container","\-","\-","\-","^^^^^^^^^^^^^^^^^^^"
"","Jul 16, 2025 @ 10:34:04.406","reference-need-predictor-00012-deployment-7d54649454-jrbmc","kserve-container","\-","\-","\-","raise e"
"","Jul 16, 2025 @ 10:34:04.406","reference-need-predictor-00012-deployment-7d54649454-jrbmc","kserve-container","\-","\-","\-","return await dependant.call(\*\*values)"
"","Jul 16, 2025 @ 10:34:04.406","reference-need-predictor-00012-deployment-7d54649454-jrbmc","kserve-container","\-","\-","\-","Traceback (most recent call last):"
"","Jul 16, 2025 @ 10:34:04.406","reference-need-predictor-00012-deployment-7d54649454-jrbmc","kserve-container","\-","\-","\-","File "/opt/lib/venv/lib/python3.11/site-packages/starlette/\_exception\_handler.py", line 53, in wrapped\_app"
"","Jul 16, 2025 @ 10:34:04.406","reference-need-predictor-00012-deployment-7d54649454-jrbmc","kserve-container","\-","\-","\-","await app(scope, receive, sender)"
"","Jul 16, 2025 @ 10:34:04.406","reference-need-predictor-00012-deployment-7d54649454-jrbmc","kserve-container","\-","\-","\-","File "/opt/lib/venv/lib/python3.11/site-packages/fastapi/routing.py", line 294, in app"
"","Jul 16, 2025 @ 10:34:04.406","reference-need-predictor-00012-deployment-7d54649454-jrbmc","kserve-container","\-","\-","\-","raw\_response = await run\_endpoint\_function("
"","Jul 16, 2025 @ 10:34:04.407","reference-need-predictor-00012-deployment-7d54649454-jrbmc","kserve-container","\-","\-","\-","^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^"
"","Jul 16, 2025 @ 10:34:04.407","reference-need-predictor-00012-deployment-7d54649454-jrbmc","kserve-container","\-","\-","\-","File "/opt/lib/venv/lib/python3.11/site-packages/kserve/protocol/dataplane.py", line 343, in infer"
"","Jul 16, 2025 @ 10:34:04.407","reference-need-predictor-00012-deployment-7d54649454-jrbmc","kserve-container","\-","\-","\-","response = await model(request, headers=headers)"
"","Jul 16, 2025 @ 10:34:04.407","reference-need-predictor-00012-deployment-7d54649454-jrbmc","kserve-container","\-","\-","\-","File "/opt/lib/venv/lib/python3.11/site-packages/kserve/model.py", line 201, in \_\_call\_\_""","Jul 16, 2025 @ 10:34:04.407","reference-need-predictor-00012-deployment-7d54649454-jrbmc","kserve-container","\-","\-","\-","raise InferenceError("
"","Jul 16, 2025 @ 10:34:04.407","reference-need-predictor-00012-deployment-7d54649454-jrbmc","kserve-container","\-","\-","\-","^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^"
"","Jul 16, 2025 @ 10:34:04.407","reference-need-predictor-00012-deployment-7d54649454-jrbmc","kserve-container","\-","\-","\-","(await self.predict(payload, headers))"
"","Jul 16, 2025 @ 10:34:04.407","reference-need-predictor-00012-deployment-7d54649454-jrbmc","kserve-container","\-","\-","\-","^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^"
"","Jul 16, 2025 @ 10:34:04.407","reference-need-predictor-00012-deployment-7d54649454-jrbmc","kserve-container","\-","\-","\-","result, model\_version = await self.async\_classifier\_pool.async\_classify("
"","Jul 16, 2025 @ 10:34:04.407","reference-need-predictor-00012-deployment-7d54649454-jrbmc","kserve-container","\-","\-","\-","kserve.errors.InferenceError: An error happened while scoring the revision-id, please contact the ML-Team if the issue persists."
"","Jul 16, 2025 @ 10:34:04.407","reference-need-predictor-00012-deployment-7d54649454-jrbmc","kserve-container","\-","\-","\-","^^^^^^^^^^^^^^^^^^^^^^^^^^^"
"","Jul 16, 2025 @ 10:34:04.407","reference-need-predictor-00012-deployment-7d54649454-jrbmc","kserve-container","\-","\-","\-","^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^"
"","Jul 16, 2025 @ 10:34:04.407","reference-need-predictor-00012-deployment-7d54649454-jrbmc","kserve-container","\-","\-","\-","File "/opt/lib/venv/lib/python3.11/site-packages/kserve/protocol/rest/v1\_endpoints.py", line 81, in predict"
"","Jul 16, 2025 @ 10:34:04.407","reference-need-predictor-00012-deployment-7d54649454-jrbmc","kserve-container","\-","\-","\-","File "/srv/ref\_quality/model\_server/model.py", line 174, in predict"
"","Jul 16, 2025 @ 10:34:04.407","reference-need-predictor-00012-deployment-7d54649454-jrbmc","kserve-container","\-","\-","\-","response, response\_headers = await self.dataplane.infer("
"","Jul 16, 2025 @ 10:34:04.407","reference-need-predictor-00012-deployment-7d54649454-jrbmc","kserve-container","\-","\-","\-","File "/srv/ref\_quality/model\_server/model.py", line 72, in async\_classify"
"","Jul 16, 2025 @ 10:34:04.407","reference-need-predictor-00012-deployment-7d54649454-jrbmc","kserve-container","\-","\-","\-","2025-07-16 10:34:04.364 kserve.trace kserve.io.kserve.protocol.rest.v1\_endpoints.predict: 909.2709107398987"
"","Jul 16, 2025 @ 10:34:04.408","reference-need-predictor-00012-deployment-7d54649454-jrbmc","kserve-container","\-","\-","\-","ERROR:root:Re-creation of a newer process pool before proceeding."
"","Jul 16, 2025 @ 10:34:04.408","reference-need-predictor-00012-deployment-7d54649454-jrbmc","kserve-container","\-","\-","\-","return await asyncio.get\_event\_loop().run\_in\_executor("
"","Jul 16, 2025 @ 10:34:04.408","reference-need-predictor-00012-deployment-7d54649454-jrbmc","kserve-container","\-","\-","\-","File "/srv/ref\_quality/python/process\_utils.py", line 75, in run\_in\_process\_pool"
"","Jul 16, 2025 @ 10:34:04.408","reference-need-predictor-00012-deployment-7d54649454-jrbmc","kserve-container","\-","\-","\-","File "/srv/ref\_quality/python/decorators.py", line 26, in elapsed\_time\_wrapper"
"","Jul 16, 2025 @ 10:34:04.408","reference-need-predictor-00012-deployment-7d54649454-jrbmc","kserve-container","\-","\-","\-","result = await func(\*args, \*\*kwargs)"
"","Jul 16, 2025 @ 10:34:04.408","reference-need-predictor-00012-deployment-7d54649454-jrbmc","kserve-container","\-","\-","\-","^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^"
"","Jul 16, 2025 @ 10:34:04.408","reference-need-predictor-00012-deployment-7d54649454-jrbmc","kserve-container","\-","\-","\-","result, model\_version = await process\_utils.run\_in\_process\_pool("
"","Jul 16, 2025 @ 10:34:04.408","reference-need-predictor-00012-deployment-7d54649454-jrbmc","kserve-container","\-","\-","\-","concurrent.futures.process.BrokenProcessPool: A process in the process pool was terminated abruptly while the future was running or pending."
"","Jul 16, 2025 @ 10:34:04.408","reference-need-predictor-00012-deployment-7d54649454-jrbmc","kserve-container","\-","\-","\-","2025-07-16 10:34:04.364 kserve.trace kserve.io.kserve.protocol.rest.v1\_endpoints.predict: 11.031468000000018"
"","Jul 16, 2025 @ 10:34:04.408","reference-need-predictor-00012-deployment-7d54649454-jrbmc","kserve-container","\-","\-","\-","Traceback (most recent call last):"
"","Jul 16, 2025 @ 10:34:04.408","reference-need-predictor-00012-deployment-7d54649454-jrbmc","kserve-container","\-","\-","\-","File "/srv/ref\_quality/model\_server/model.py", line 60, in async\_classify"
"","Jul 16, 2025 @ 10:34:04.408","reference-need-predictor-00012-deployment-7d54649454-jrbmc","kserve-container","\-","\-","\-","^^^^^^^^^^^^^^^^^^^^^^^^^^^"
"","Jul 16, 2025 @ 10:34:04.408","reference-need-predictor-00012-deployment-7d54649454-jrbmc","kserve-container","\-","\-","\-","^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^"
"","Jul 16, 2025 @ 10:34:04.411","reference-need-predictor-00012-deployment-7d54649454-jrbmc","kserve-container","\-","\-","\-","result = await func(\*args, \*\*kwargs)"
"","Jul 16, 2025 @ 10:34:04.411","reference-need-predictor-00012-deployment-7d54649454-jrbmc","kserve-container","\-","\-","\-","^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^"
"","Jul 16, 2025 @ 10:34:04.411","reference-need-predictor-00012-deployment-7d54649454-jrbmc","kserve-container","\-","\-","\-","File "/srv/ref\_quality/python/decorators.py", line 26, in elapsed\_time\_wrapper"
"","Jul 16, 2025 @ 10:34:04.411","reference-need-predictor-00012-deployment-7d54649454-jrbmc","kserve-container","\-","\-","\-","INFO:root:Create a process pool of 2 workers to support model scoring blocking code."
"","Jul 16, 2025 @ 10:34:04.411","reference-need-predictor-00012-deployment-7d54649454-jrbmc","kserve-container","\-","\-","\-","2025-07-16 10:34:04.364 1 kserve ERROR \[errors.py:inference\_error\_handler():104\] Exception:"
"","Jul 16, 2025 @ 10:34:04.411","reference-need-predictor-00012-deployment-7d54649454-jrbmc","kserve-container","\-","\-","\-","Traceback (most recent call last):"
"","Jul 16, 2025 @ 10:34:04.411","reference-need-predictor-00012-deployment-7d54649454-jrbmc","kserve-container","\-","\-","\-","File "/srv/ref\_quality/model\_server/model.py", line 60, in async\_classify"
"","Jul 16, 2025 @ 10:34:04.411","reference-need-predictor-00012-deployment-7d54649454-jrbmc","kserve-container","\-","\-","\-","result, model\_version = await process\_utils.run\_in\_process\_pool("
"","Jul 16, 2025 @ 10:34:04.412","reference-need-predictor-00012-deployment-7d54649454-jrbmc","kserve-container","\-","\-","\-","^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^"
"","Jul 16, 2025 @ 10:34:04.412","reference-need-predictor-00012-deployment-7d54649454-jrbmc","kserve-container","\-","\-","\-","^^^^^^^^^^^^^^^^^^^^^^^^^^^"
"","Jul 16, 2025 @ 10:34:04.412","reference-need-predictor-00012-deployment-7d54649454-jrbmc","kserve-container","\-","\-","\-","File "/srv/ref\_quality/python/process\_utils.py", line 75, in run\_in\_process\_pool"
"","Jul 16, 2025 @ 10:34:04.412","reference-need-predictor-00012-deployment-7d54649454-jrbmc","kserve-container","\-","\-","\-","return await asyncio.get\_event\_loop().run\_in\_executor("
"","Jul 16, 2025 @ 10:34:04.412","reference-need-predictor-00012-deployment-7d54649454-jrbmc","kserve-container","\-","\-","\-","concurrent.futures.process.BrokenProcessPool: A process in the process pool was terminated abruptly while the future was running or pending."
"","Jul 16, 2025 @ 10:34:04.412","reference-need-predictor-00012-deployment-7d54649454-jrbmc","kserve-container","\-","\-","\-","During handling of the above exception, another exception occurred:"
"","Jul 16, 2025 @ 10:34:04.412","reference-need-predictor-00012-deployment-7d54649454-jrbmc","kserve-container","\-","\-","\-","File "/opt/lib/venv/lib/python3.11/site-packages/starlette/\_exception\_handler.py", line 53, in wrapped\_app"
"","Jul 16, 2025 @ 10:34:04.412","reference-need-predictor-00012-deployment-7d54649454-jrbmc","kserve-container","\-","\-","\-","raise e"
"","Jul 16, 2025 @ 10:34:04.412","reference-need-predictor-00012-deployment-7d54649454-jrbmc","kserve-container","\-","\-","\-","File "/opt/lib/venv/lib/python3.11/site-packages/fastapi/routing.py", line 294, in app"
"","Jul 16, 2025 @ 10:34:04.412","reference-need-predictor-00012-deployment-7d54649454-jrbmc","kserve-container","\-","\-","\-",""
"","Jul 16, 2025 @ 10:34:04.412","reference-need-predictor-00012-deployment-7d54649454-jrbmc","kserve-container","\-","\-","\-",""
"","Jul 16, 2025 @ 10:34:04.412","reference-need-predictor-00012-deployment-7d54649454-jrbmc","kserve-container","\-","\-","\-","Traceback (most recent call last):"
"","Jul 16, 2025 @ 10:34:04.412","reference-need-predictor-00012-deployment-7d54649454-jrbmc","kserve-container","\-","\-","\-","await app(scope, receive, sender)"
"","Jul 16, 2025 @ 10:34:04.412","reference-need-predictor-00012-deployment-7d54649454-jrbmc","kserve-container","\-","\-","\-","File "/opt/lib/venv/lib/python3.11/site-packages/starlette/routing.py", line 74, in app"
"","Jul 16, 2025 @ 10:34:04.412","reference-need-predictor-00012-deployment-7d54649454-jrbmc","kserve-container","\-","\-","\-","response = await func(request)"
"","Jul 16, 2025 @ 10:34:04.412","reference-need-predictor-00012-deployment-7d54649454-jrbmc","kserve-container","\-","\-","\-","^^^^^^^^^^^^^^^^^^^""","Jul 16, 2025 @ 10:34:04.412","reference-need-predictor-00012-deployment-7d54649454-jrbmc","kserve-container","\-","\-","\-","File "/opt/lib/venv/lib/python3.11/site-packages/fastapi/routing.py", line 299, in app"
"","Jul 16, 2025 @ 10:34:04.413","reference-need-predictor-00012-deployment-7d54649454-jrbmc","kserve-container","\-","\-","\-","^^^^^^^^^^^^^^^^^^^^^^^^^^^"
"","Jul 16, 2025 @ 10:34:04.413","reference-need-predictor-00012-deployment-7d54649454-jrbmc","kserve-container","\-","\-","\-","response, response\_headers = await self.dataplane.infer("
"","Jul 16, 2025 @ 10:34:04.413","reference-need-predictor-00012-deployment-7d54649454-jrbmc","kserve-container","\-","\-","\-","^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^"
"","Jul 16, 2025 @ 10:34:04.413","reference-need-predictor-00012-deployment-7d54649454-jrbmc","kserve-container","\-","\-","\-","File "/opt/lib/venv/lib/python3.11/site-packages/kserve/protocol/rest/v1\_endpoints.py", line 81, in predict"
"","Jul 16, 2025 @ 10:34:04.413","reference-need-predictor-00012-deployment-7d54649454-jrbmc","kserve-container","\-","\-","\-","raw\_response = await run\_endpoint\_function("
"","Jul 16, 2025 @ 10:34:04.413","reference-need-predictor-00012-deployment-7d54649454-jrbmc","kserve-container","\-","\-","\-","^^^^^^^^^^^^^^^^^^^^^^^^^^^^"
"","Jul 16, 2025 @ 10:34:04.413","reference-need-predictor-00012-deployment-7d54649454-jrbmc","kserve-container","\-","\-","\-","File "/opt/lib/venv/lib/python3.11/site-packages/fastapi/routing.py", line 191, in run\_endpoint\_function"
"","Jul 16, 2025 @ 10:34:04.413","reference-need-predictor-00012-deployment-7d54649454-jrbmc","kserve-container","\-","\-","\-","return await dependant.call(\*\*values)"
"","Jul 16, 2025 @ 10:34:04.414","reference-need-predictor-00012-deployment-7d54649454-jrbmc","kserve-container","\-","\-","\-","File "/opt/lib/venv/lib/python3.11/site-packages/kserve/protocol/dataplane.py", line 343, in infer"
"","Jul 16, 2025 @ 10:34:04.414","reference-need-predictor-00012-deployment-7d54649454-jrbmc","kserve-container","\-","\-","\-","response = await model(request, headers=headers)"
"","Jul 16, 2025 @ 10:34:04.414","reference-need-predictor-00012-deployment-7d54649454-jrbmc","kserve-container","\-","\-","\-","result, model\_version = await self.async\_classifier\_pool.async\_classify("
"","Jul 16, 2025 @ 10:34:04.414","reference-need-predictor-00012-deployment-7d54649454-jrbmc","kserve-container","\-","\-","\-","File "/opt/lib/venv/lib/python3.11/site-packages/kserve/model.py", line 201, in \_\_call\_\_"
"","Jul 16, 2025 @ 10:34:04.414","reference-need-predictor-00012-deployment-7d54649454-jrbmc","kserve-container","\-","\-","\-","(await self.predict(payload, headers))"
"","Jul 16, 2025 @ 10:34:04.414","reference-need-predictor-00012-deployment-7d54649454-jrbmc","kserve-container","\-","\-","\-","^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^"
"","Jul 16, 2025 @ 10:34:04.414","reference-need-predictor-00012-deployment-7d54649454-jrbmc","kserve-container","\-","\-","\-","File "/srv/ref\_quality/model\_server/model.py", line 174, in predict"
"","Jul 16, 2025 @ 10:34:04.414","reference-need-predictor-00012-deployment-7d54649454-jrbmc","kserve-container","\-","\-","\-","^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^"
"","Jul 16, 2025 @ 10:34:04.415","reference-need-predictor-00012-deployment-7d54649454-jrbmc","kserve-container","\-","\-","\-","^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^"
"","Jul 16, 2025 @ 10:34:04.415","reference-need-predictor-00012-deployment-7d54649454-jrbmc","kserve-container","\-","\-","\-","File "/srv/ref\_quality/model\_server/model.py", line 72, in async\_classify"
"","Jul 16, 2025 @ 10:34:04.415","reference-need-predictor-00012-deployment-7d54649454-jrbmc","kserve-container","\-","\-","\-","raise InferenceError("
"","Jul 16, 2025 @ 10:34:04.415","reference-need-predictor-00012-deployment-7d54649454-jrbmc","kserve-container","\-","\-","\-","kserve.errors.InferenceError: An error happened while scoring the revision-id, please contact the ML-Team if the issue persists."
"","Jul 16, 2025 @ 10:34:04.415","reference-need-predictor-00012-deployment-7d54649454-jrbmc","kserve-container","\-","\-","\-","2025-07-16 10:34:04.365 kserve.trace kserve.io.kserve.protocol.rest.v1\_endpoints.predict: 906.4698672294617"
"","Jul 16, 2025 @ 10:34:04.415","reference-need-predictor-00012-deployment-7d54649454-jrbmc","kserve-container","\-","\-","\-","2025-07-16 10:34:04.365 kserve.trace kserve.io.kserve.protocol.rest.v1\_endpoints.predict: 11.014702999999969"
"","Jul 16, 2025 @ 10:34:04.416","reference-need-predictor-00012-deployment-7d54649454-jrbmc","kserve-container","\-","\-","\-","Traceback (most recent call last):"
"","Jul 16, 2025 @ 10:34:04.416","reference-need-predictor-00012-deployment-7d54649454-jrbmc","kserve-container","\-","\-","\-","File "/srv/ref\_quality/model\_server/model.py", line 60, in async\_classify"
"","Jul 16, 2025 @ 10:34:04.416","reference-need-predictor-00012-deployment-7d54649454-jrbmc","kserve-container","\-","\-","\-","result, model\_version = await process\_utils.run\_in\_process\_pool("
"","Jul 16, 2025 @ 10:34:04.416","reference-need-predictor-00012-deployment-7d54649454-jrbmc","kserve-container","\-","\-","\-","ERROR:root:Re-creation of a newer process pool before proceeding."
"","Jul 16, 2025 @ 10:34:04.417","reference-need-predictor-00012-deployment-7d54649454-jrbmc","kserve-container","\-","\-","\-","^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^"
Content licensed under Creative Commons Attribution-ShareAlike (CC BY-SA) 4.0 unless otherwise noted; code licensed under GNU General Public License (GPL) 2.0 or later and other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL · Credits