As part of the migration of eqsin we need to re-create the prometheus instance on routed ganeti. As procedure as https://phabricator.wikimedia.org/T419430
- remove prometheus5003 from manifestes/site.pp
- setup firewall rules
#hieradata/eqsin.yaml prometheus_nodes: - prometheus5002.eqsin.wmnet - prometheus5003.eqsin.wmnet
- enable rsyncd on eqsin's instances (to be reverted)
- run puppet on prometheu5003
- Initial rsync of data from 5002 to 5003 (on 5003, as user prometheus) to be done in advance
prometheus-sync-data prometheus5002.eqsin.wmnet ops
- disable-puppet on all prometheus hosts in the site
- Add 5003, remove 5002 to prometheus_instances in puppet for the selected instance and merge the patch
# add, hieradata/common/prometheus.yaml
prometheus::instances:
ops:
port: 9900
retention_size: 3300GB
provision_lv_size: '3.6t'
hosts:
- prometheus...
- prometheus5003.eqsin.wmnet#add replica label for the new upcoming prometheus5003 node #hieradata/hosts/prometheus5003.yaml prometheus::replica_label: 'a'
- on prometheus5002
systemctl disable {prometheus,thanos-sidecar}@ops
systemctl stop {prometheus,thanos-sidecar}@ops
chmod a-x /usr/bin/thanos /usr/bin/prometheus- Final rsync of data from 5002 to 5003 (on 5003, as user prometheus)
prometheus-sync-data prometheus5002.eqsin.wmnet ops
- restart services on 5003
run-puppet-agent --force
- fix DNS entries
#templates/wmnet $ORIGIN svc.eqsin.wmnet. $INCLUDE netbox/svc.eqsin.wmnet prometheus 1H IN CNAME prometheus5003.eqsin.wmnet.
- REVERT enable rsyncd on eqsin's instances