Page MenuHomePhabricator
Create Task
Maniphest T399733

Investigate reference-need-predictor alert triggered by BrokenProcessPool error
Closed, ResolvedPublic
Actions

Description

An alert for the reference-need-predictor service was triggered on 16/07/2025. The initial alert information was:

reference-need-predictor revision-models istio-system k8s-mlserve critical eqiad prometheus

I looked at the reference-need-predictor logs in logstash: https://logstash.wikimedia.org/goto/c558820a21decb143949f8319d3e6fbb and there were spikes in the minute of 10:34 UTC. They showed that the service was failing due to:

concurrent.futures.process.BrokenProcessPool: A process in the process pool was terminated abruptly while the future was running or pending.

see logs here: https://phabricator.wikimedia.org/P79250

@elukey suggested looking at the Istio dashboard since all the traffic goes through Istio before reaching the isvcs. After drilling into to 10 to 11 UTC: https://logstash.wikimedia.org/goto/efdaccbb1f9112eeb474fdb85ef502f8, we saw that the user agent is: WME/2.0 (https://enterprise.wikimedia.com/; wme_mgmt@wikimedia.org)

This alert resolved itself at ~13 UTC but we are going to investigate whether the model inputs sent by the WME client were unusually large, whether they require more resources, or if it was a different issue.

Related Objects
Search...

Event Timeline

kevinbazira created this task.Jul 16 2025, 3:39 PM
Restricted Application added a subscriber: Aklapper. · View Herald TranscriptJul 16 2025, 3:39 PM
isarantopoulos added a comment.Jul 16 2025, 3:51 PM

This seems to be the same issue described in T387019: Increased latencies in reference-quality models (ref-need). In the last comment there is a mention about the broken process pool.
I wonder if there’s some kind of memory leak causing the pod to get OOMKilled, resulting in unmet traffic demand until a new pod starts.

isarantopoulos added a project: Essential-Work.Jul 18 2025, 11:44 AM
isarantopoulos moved this task from Unsorted to Ready To Go on the Machine-Learning-Team board.
gkyziridis subscribed.EditedJul 21 2025, 2:26 PM

I tried to reproduce the error but it seems difficult.
What I saw is that we received (almost) the same time 7 requests with these ids: 146180898, 146193976, 146180765, 146193512, 146180861, 146193331, 146170512 and lang: ru at the time of the incident.
I tried to reproduce it by running a script which hits the prod endpoint iteratively:

P79522 Hit reference-need endpoint iteratively
1#!/bin/bash
2
3codes=(
4 146180898
5 146193976
6 146180765
7 146193512
8 146180861
9 146193331
10 146170512
11)
12
13for code in "${codes[@]}"; do
14 curl -s "https://inference.svc.eqiad.wmnet:30443/v1/models/reference-need:predict" \
15 -X POST \
16 -d "{\"rev_id\": $code, \"lang\": \"ru\"}" \
17 -H "Host: reference-need.revision-models.wikimedia.org"
18done
19
20
21## Response
22{"model_name":"reference-need","model_version":0,"wiki_db":"ruwiki","revision_id":146180898,"reference_need_score":0.6666666666666666}
23
24{"model_name":"reference-need","model_version":0,"wiki_db":"ruwiki","revision_id":146193976,"reference_need_score":1.0}
25
26{"model_name":"reference-need","model_version":0,"wiki_db":"ruwiki","revision_id":146180765,"reference_need_score":0.8245614035087719}
27
28{"model_name":"reference-need","model_version":0,"wiki_db":"ruwiki","revision_id":146193512,"reference_need_score":0.4166666666666667}
29
30{"model_name":"reference-need","model_version":0,"wiki_db":"ruwiki","revision_id":146180861,"reference_need_score":0.8823529411764706}
31
32{"model_name":"reference-need","model_version":0,"wiki_db":"ruwiki","revision_id":146193331,"reference_need_score":0.8461538461538461}
33
34{"model_name":"reference-need","model_version":0,"wiki_db":"ruwiki","revision_id":146170512,"reference_need_score":0.8108108108108109}

I also checked the graphana dashboard during that day/time and we can see that the CPU Usage - kserve-container was very close to the limit:
Limit: 22s
User: 18.9s
Not sure if this is reasonable or correlated to the incident.

image.png (692×516 px, 77 KB)

gkyziridis attached a referenced file: F65534588: image.png. (Show Details)Jul 21 2025, 2:57 PM
isarantopoulos added a comment.Jul 21 2025, 3:19 PM

Looking at the resources of the pod in grafana that the logs were extracted from I noticed the same pattern as before which seems to be the root of the problem: memory utilization is increasing and probably old processes are not cleared from memory.
You can find more info in the last 2 comments in the related task https://phabricator.wikimedia.org/T387019#10693506 https://phabricator.wikimedia.org/T387019#10694215

isarantopoulos closed this task as Resolved.EditedMar 27 2026, 2:50 PM
isarantopoulos claimed this task.

This task has not received any activity for quite a while. Since we haven't had any incidents about this lately, I'm gonna go ahead and resolve this.

isarantopoulos moved this task from Ready To Go to 2025-2026 Q2 Done on the Machine-Learning-Team board.Mar 27 2026, 2:57 PM
isarantopoulos moved this task from 2025-2026 Q2 Done to Task Archive on the Machine-Learning-Team board.May 20 2026, 11:12 AM
Content licensed under Creative Commons Attribution-ShareAlike (CC BY-SA) 4.0 unless otherwise noted; code licensed under GNU General Public License (GPL) 2.0 or later and other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL · Credits