The integration project will need its own Prometheus server for collection and storage of metrics related to the project instances and the CI cluster overall.
We can probably base the necessary ops/puppet changes off of the role::prometheus::beta module used for deployment-prometheus01—which currently serves as a collector and data source for Beta Cluster.
| Status | Subtype | Assigned | Task | ||
|---|---|---|---|---|---|
| Open | None | T205923 Formalize the collection of CI infrastructure and tooling metrics | |||
| Open | None | T205927 Collect and expose Jenkins build metrics for visualization, reporting, and analysis | |||
| Declined | None | T205930 Set up a Prometheus server under the integration project |
See T182759: Add Prometheus exporter to Jenkins instances for the correct Prometheus target URLs of the Jenkins masters.
@fgiunchedi do you have any guidance to create a Prometheus instance on WMCS and have the exporters applied on all instances of a WMCS tenant?
The first step is what @dduvall outlined, i.e. adapt what's being done for beta prometheus server. As for exporters, ./modules/prometheus/files/usr/local/bin/prometheus-labs-targets will periodically (re)configure Prometheus to poll node-exporter metrics from all instances of the project it is deployed onto. For additional exporters you can go a similar way than production, namely generate the necessary configuration from puppet exported resources (check out prometheus::class_config for an example)
HTH!
Reflecting the reality, we are just following whatever is the default for WMCS projects. Right now we rely on https://grafana-labs.wikimedia.org/d/000000059/cloud-vps-project-board?orgId=1&var-project=integration&var-server=All and it seems good enough.
For integration we don't have much use cases beside the simple cpu/mem/disk usage tracking.