>>! From {T336774}:
> Ideally both answers are 'no', but regardless I'd like to move the `labs` instance of Prometheus off cloudmetrics and onto the Prometheus production hardware. My understanding is that this move will help both WMCS (one less component to think about) and Observability (less variance/snowflakes).
== Action plan ==
* [ ] Update `cr` firewall rules to allow this traffic (T343885#9119388)
* [x] Add a new `profile::prometheus::cloud` class (and rename `labs` to `cloud` in the process)
* [x] Allocate space on prometheus LVs (`eqiad` only) (`modules/prometheus/files/provision-fs.sh`)
* [x] Deploy said class to `prometheus` hosts (`eqiad` only), making sure `alertmanagers` parameter is not set so alerts won't be sent.
* [x] Verify Prometheus can pull metrics from all its jobs.
* [x] Let metrics data accumulate for TBD days
* [x] Enable alerts to be sent from `prometheus`, disable alerts from `cloudmetrics`
* [x] Audit accesses on cloudmetrics hosts for Prometheus clients
* [x] Update cloudmetrics references to point to `prometheus.svc/cloud` instead
== Followups ==
* [ ] Add thanos upload/sidecar
* [ ] Remove `labs` Prometheus instance from `cloudmetrics`
* [ ] Duplicate setup to `codfw`?
* [ ] Rename grafana datasource to `eqiad prometheus/cloud`. The following dashboards will need updating:
** Matched https://grafana.wikimedia.org/d/8KPwK6GMk/cloudcontrol-mysql-aggregated (cloudcontrol MySQL Aggregated)
** Matched https://grafana.wikimedia.org/d/cloudmetrics-prometheus-details/cloudmetrics-prometheus (Cloudmetrics - prometheus)
** Matched https://grafana.wikimedia.org/d/7TjJENEWz/wmcs-ceph-eqiad-cluster-overview (WMCS - Ceph eqiad Cluster Overview)
** Matched https://grafana.wikimedia.org/d/y0KGL0iZz/wmcs-ceph-eqiad-host-overview (WMCS - Ceph eqiad Host Overview)
** Matched https://grafana.wikimedia.org/d/wsoKtElZk/wmcs-ceph-eqiad-network-utilization (WMCS - Ceph eqiad Network Utilization)
** Matched https://grafana.wikimedia.org/d/CrAHE0iZz/wmcs-ceph-eqiad-osd-device-details (WMCS - Ceph eqiad OSD device details)
** Matched https://grafana.wikimedia.org/d/lo02I1Aiz/wmcs-ceph-eqiad-osd-overview (WMCS - Ceph eqiad OSD Overview)
** Matched https://grafana.wikimedia.org/d/-xyV8KCiz/wmcs-ceph-eqiad-pool-details (WMCS - Ceph eqiad Pool Details)
** Matched https://grafana.wikimedia.org/d/z99hzWtmk/wmcs-ceph-eqiad-pools-overview (WMCS - Ceph eqiad Pools Overview)
** Matched https://grafana.wikimedia.org/d/i7ltypbnz/wmcs-network-cost-estimation (WMCS - Network Cost Estimation)
** Matched https://grafana.wikimedia.org/d/galera-cluster-summary/wmcs-openstack-eqiad-galera-cluster-summary (WMCS - Openstack eqiad Galera cluster summary)
** Matched https://grafana.wikimedia.org/d/tN1aK6MGk/wmcs-openstack-eqiad-galera-node-specific-metrics (WMCS - Openstack eqiad Galera node specific metrics)
** Matched https://grafana.wikimedia.org/d/Kn5xm-gZk/wmcs-openstack-eqiad-rabbitmq-overview (WMCS - Openstack eqiad RabbitMQ overview)