Create a new OpenStack project and Prometheus server to scrape metrics from the existing node-exporter running on virtual machines in tools and cloud-infra projects. Once these projects are configured WMCS will evaluate adding more CloudVPS projects to this configuration.
Initial steps to deploy the new monitoring stack:
[ ] Create OpenStack project "metrics-infra" with wmcs-team as admins
[ ] Create a new virtual machine "prometheus01.metrics-infra.eqiad.wikimedia.cloud"
[ ] Configure Prometheus to discover scrape targets using the openstack SD configuration
[ ] Update existing security groups to allow prometheus to connect to the node-exporter running on TCP port 9100
[ ] Configure new project template with updated security group rules
[ ] Configure the Prometheus alert manager to monitor puppet status
[ ] Configure the alert manager to monitor disk capacity
[ ] Configure the alert manager to monitor host up/down state
[ ] Configure the alert manager to notify wmcs-team email and IRC #wikimedia-cloud-feed
[ ] Configure a proxy to allow Grafana access to the Prometheus API