Change Details

An alert for the reference-need-predictor service was triggered on 18/07/2025. The initial alert information was: ``` reference-need-predictor-00012-deployment revision-models k8s-mlserve warning eqiad prometheus ``` I looked at the reference-need-predictor logs in logstash: https://logstash.wikimedia.org/goto/3235c894dbdbbcd08d64e3e78f82570f and after 23:07 UTC several queue-proxy errors with: ``` error reverse proxying request; sockstat: sockets: used 1078 TCP: inuse 1022 orphan 2 tw 121 alloc 11745 mem 0 UDP: inuse 0 mem 7616 UDPLITE: inuse 0 RAW: inuse 0 FRAG: inuse 0 memory 0 ``` after 23:44 UTC several istio-proxy errors with: ``` Request to probe app failed: Get "http://10.67.16.134:8012/": context deadline exceeded (Client.Timeout exceeded while awaiting headers), original URL path = /app-health/queue-proxy/readyz app URL path = / ``` after 00:11 UTC several queue-proxy errors with: ``` aggressive probe error (failed 73 times): dial tcp 127.0.0.1:8080: i/o timeout ``` Surprisingly, before the first queue-proxy error shown in the next column (on the right), the kserve-container seemed to be working fine: ``` INFO:root:Function run_in_process_pool took 0.27 seconds to execute. 2025-07-17 23:07:36.085 kserve.trace kserve.io.kserve.protocol.rest.v1_endpoints.predict: 0.0034160000000156288 2025-07-17 23:07:36.085 kserve.trace kserve.io.kserve.protocol.rest.v1_endpoints.predict: 0.33701109886169434 2025-07-17 23:07:36.084 kserve.trace requestId: 1364c0a1-4b6f-4eb4-b2a7-ad1530e4bc2a, preprocess_ms: 62.111616135, explain_ms: 0, predict_ms: 274.287462234, postprocess_ms: 0.009298325 2025-07-17 23:07:36.085 uvicorn.access INFO: 127.0.0.6:0 1 - "POST /v1/models/reference-need%3Apredict HTTP/1.1" 200 OK ``` This alert resolved itself at ~1 UTC. The root cause of this issue is yet to be identified but it seems to be similar to the one identified in {https://phabricator.wikimedia.org/T346445} where the solution ended up being a fix of a memory leak.

An alert for the reference-need-predictor service was triggered on 18/07/2025. The initial alert information was: ``` reference-need-predictor-00012-deployment revision-models k8s-mlserve warning eqiad prometheus ``` I looked at the reference-need-predictor logs in logstash: https://logstash.wikimedia.org/goto/3235c894dbdbbcd08d64e3e78f82570f and after 23:07 UTC several queue-proxy errors with: ``` error reverse proxying request; sockstat: sockets: used 1078 TCP: inuse 1022 orphan 2 tw 121 alloc 11745 mem 0 UDP: inuse 0 mem 7616 UDPLITE: inuse 0 RAW: inuse 0 FRAG: inuse 0 memory 0 ``` after 23:44 UTC several istio-proxy errors with: ``` Request to probe app failed: Get "http://10.67.16.134:8012/": context deadline exceeded (Client.Timeout exceeded while awaiting headers), original URL path = /app-health/queue-proxy/readyz app URL path = / ``` after 00:11 UTC several queue-proxy errors with: ``` aggressive probe error (failed 73 times): dial tcp 127.0.0.1:8080: i/o timeout ``` Surprisingly, before the first queue-proxy error shown above, the kserve-container seemed to be working fine: ``` INFO:root:Function run_in_process_pool took 0.27 seconds to execute. 2025-07-17 23:07:36.085 kserve.trace kserve.io.kserve.protocol.rest.v1_endpoints.predict: 0.0034160000000156288 2025-07-17 23:07:36.085 kserve.trace kserve.io.kserve.protocol.rest.v1_endpoints.predict: 0.33701109886169434 2025-07-17 23:07:36.084 kserve.trace requestId: 1364c0a1-4b6f-4eb4-b2a7-ad1530e4bc2a, preprocess_ms: 62.111616135, explain_ms: 0, predict_ms: 274.287462234, postprocess_ms: 0.009298325 2025-07-17 23:07:36.085 uvicorn.access INFO: 127.0.0.6:0 1 - "POST /v1/models/reference-need%3Apredict HTTP/1.1" 200 OK ``` This alert resolved itself at ~1 UTC. The root cause of this issue is yet to be identified but it seems to be similar to the one identified in {https://phabricator.wikimedia.org/T346445} where the solution ended up being a fix of a memory leak.

An alert for the reference-need-predictor service was triggered on 18/07/2025. The initial alert information was: ``` reference-need-predictor-00012-deployment revision-models k8s-mlserve warning eqiad prometheus ``` I looked at the reference-need-predictor logs in logstash: https://logstash.wikimedia.org/goto/3235c894dbdbbcd08d64e3e78f82570f and after 23:07 UTC several queue-proxy errors with: ``` error reverse proxying request; sockstat: sockets: used 1078 TCP: inuse 1022 orphan 2 tw 121 alloc 11745 mem 0 UDP: inuse 0 mem 7616 UDPLITE: inuse 0 RAW: inuse 0 FRAG: inuse 0 memory 0 ``` after 23:44 UTC several istio-proxy errors with: ``` Request to probe app failed: Get "http://10.67.16.134:8012/": context deadline exceeded (Client.Timeout exceeded while awaiting headers), original URL path = /app-health/queue-proxy/readyz app URL path = / ``` after 00:11 UTC several queue-proxy errors with: ``` aggressive probe error (failed 73 times): dial tcp 127.0.0.1:8080: i/o timeout ``` Surprisingly, before the first queue-proxy error shown in the next column (on the right)above, the kserve-container seemed to be working fine: ``` INFO:root:Function run_in_process_pool took 0.27 seconds to execute. 2025-07-17 23:07:36.085 kserve.trace kserve.io.kserve.protocol.rest.v1_endpoints.predict: 0.0034160000000156288 2025-07-17 23:07:36.085 kserve.trace kserve.io.kserve.protocol.rest.v1_endpoints.predict: 0.33701109886169434 2025-07-17 23:07:36.084 kserve.trace requestId: 1364c0a1-4b6f-4eb4-b2a7-ad1530e4bc2a, preprocess_ms: 62.111616135, explain_ms: 0, predict_ms: 274.287462234, postprocess_ms: 0.009298325 2025-07-17 23:07:36.085 uvicorn.access INFO: 127.0.0.6:0 1 - "POST /v1/models/reference-need%3Apredict HTTP/1.1" 200 OK ``` This alert resolved itself at ~1 UTC. The root cause of this issue is yet to be identified but it seems to be similar to the one identified in {https://phabricator.wikimedia.org/T346445} where the solution ended up being a fix of a memory leak.