Earlier today grafana-server exploded in memory using up most/all of the grafana1002 VM's memory. The OOM killer kicked in which made things recover by themselves though.

As seen below the memory does spike up periodically too

thanos link: https://w.wiki/AG$A

From a quick look at the logs it looks like many requests to grafana image render (http://127.0.0.1:33333/render) though unclear why.

Certainly we can at least mitigate the issue by forcing memory limits on grafana-server like we did for thanos in T356788 . And also figure out the root cause