Create a new OpenStack project and Prometheus server to scrape metrics from the existing node-exporter running on virtual machines in tools and cloud-infra projects. Once these projects are configured WMCS will evaluate adding more CloudVPS projects to this configuration.
Initial steps to deploy the new monitoring stack:
[x] Create OpenStack project "metricsinfra" with wmcs-team as admins (T250210)
[x] Create a new virtual machine "prometheus01.metricsinfra.eqiad.wikimedia.cloud" (T250206#6056467)
[x] Configure Prometheus to discover scrape targets using the openstack SD configuration (https://gerrit.wikimedia.org/r/#/c/588803/)
[x] Update existing tools and cloudinfra security groups to allow prometheus to connect to the node-exporter running on TCP port 9100
[ ] Configure the Prometheus alert manager to monitor puppet status
[ ] Configure the alert manager to monitor disk capacity
[ ] Configure the alert manager to monitor host up/down state
[ ] Configure the alert manager to notify wmcs-team email and IRC #wikimedia-cloud-feed
[x] Configure a proxy to allow Grafana access to the Prometheus API - https://prometheus.wmflabs.org/cloud
[x] Add Metricsinfra Prometheus datasource to Grafana-labs
Once we have an idea on data retention and usage:
[ ] Update openstack service discovery to either monitor all projects or a specific list of projects
[ ] Configure new project template with updated security group rules