Currently we lack high level visibility of containers/pods that reach or exceed configured CPU/memory requests/limits as well as CPU throttling.
We should be able to detect that in a generic fashion to have issues like T266194 addressed more proactively.
I'll add this as a meta-task to track ideas/possible work regarding this.
| Status | Subtype | Assigned | Task | ||
|---|---|---|---|---|---|
| Resolved | akosiaris | T266216 Increase visibility of container/pod ressource exhaustion | |||
| Resolved | kamila | T264625 Deploy kube-state-metrics | |||
| Resolved | kamila | T343801 Create kube-state-metrics docker image | |||
| Resolved | akosiaris | T256256 Raise an alarm on container restarts/OOMs in kubernetes | |||
| In Progress | Clement_Goubert | T357005 eventstreams regularly uses more than 95% of its memory limit | |||
| In Progress | Clement_Goubert | T357122 linkrecommendation-internal regularly uses more than 95% of its memory limit |
Change 984219 had a related patch set uploaded (by JMeybohm; author: JMeybohm):
[operations/alerts@master] Alert for containers with memory issues
Change 984219 merged by jenkins-bot:
[operations/alerts@master] Alert for containers with memory issues
Change 994196 had a related patch set uploaded (by Alexandros Kosiaris; author: Alexandros Kosiaris):
[operations/deployment-charts@master] linkrecommendation: Bump memory limit by 200Mi
Change 994197 had a related patch set uploaded (by Alexandros Kosiaris; author: Alexandros Kosiaris):
[operations/deployment-charts@master] rdf-streaming-updated: Bump taskmanager memory limit by ~33%
Change 994196 merged by jenkins-bot:
[operations/deployment-charts@master] linkrecommendation: Bump memory limit by 200Mi
The 2 patches linked worked just fine. There is taskmanager left to be +1ed by the team, but I am willing to say that we are in a better place than we used to. I am gonna resolve this
Change 994197 merged by jenkins-bot:
[operations/deployment-charts@master] rdf-streaming-updated: Bump taskmanager memory limit by ~33%