Currently there are two prometheus nodes in paws outside of the cluster (paws-prometheus-1 and paws-prometheus-2). Can these be moved as containers inside the cluster?
Description
Description
Details
Details
Related Changes in Gerrit:
| Subject | Repo | Branch | Lines +/- | |
|---|---|---|---|---|
| Move prometheus inside paws cluster | operations/puppet | production | +1 -270 |
Related Objects
Related Objects
- Mentioned In
- rPAWSa9d74fd2cb29: Install kube-state-metrics (#371)
Event Timeline
Comment Actions
<taavi> Taavi Väänänen the tl;dr is that you should indeed install kube-state-metrics to the paws cluster, and then either fix the current puppetized paws-prometheus instances to scrape them, or indeed install prometheus inside paws <Rook> Vivian Rook So in putting kube-state-metrics and prometheus inside of the paws cluster, does that let me make a graph of the number of running pods somewhere? <dcaro> David Caro Rook: if you want to setup prometheus inside the cluster, all that needs changing is the grafana config to use paws-prometheus-in-k8s as a datastore (publicly expose that k8s prometheus url)
There isn't a grafana in codfw1dev, but if we can get paws offering an endpoint for kube-state-metrics data (like: https://hub-paws-dev.codfw1dev.wmcloud.org/hub/metrics is for jupyterhub data) we should be in good shape in eqiad1 to connect to grafana, just ask how at that point.
Comment Actions
Change 995044 had a related patch set uploaded (by Vivian Rook; author: Vivian Rook):
[operations/puppet@production] Move prometheus inside paws cluster
Comment Actions
Change 995044 merged by Vivian Rook:
[operations/puppet@production] Move prometheus inside paws cluster
Comment Actions
Mentioned in SAL (#wikimedia-cloud) [2024-02-01T20:17:03Z] <Rook> prometheus and kube-state-metrics internal to cluster T355179