Change Details

ATM we're supporting Buster and Bullseye for Prometheus (ops instance, deployed to every site). However for e.g. blackbox-exporter configuration options it'd be nice to have all hosts on Bullseye. Procedure to move data from old hosts to new hosts (to be tested, and adjusted, and moved to wikitech) === Preflight checks === 1. hosts are running `prometheus` role and show up in all prometheus hosts lists (e.g. `prometheus_all_nodes`) 1. ACLs on network devices have been updated 1. mysqld grants are updated (to be verified) === Migration === 1. [new host] stop puppet / prometheus / thanos-sidecar@ops 1. Initial rsync of data old -> new 1. [old host] stop puppet / thanos-sidecar@ops. Note that once thanos-sidecar@ops is stopped here then Thanos won't be able to query data for the PoP 1. Final rsync of data old -> new 1. [new host] set replica label in puppet to match the old hosts', merge the change 1. [new host] re-enable puppet and run puppet, this will restart prometheus and thanos-sidecar@ops, thus Thanos will be able to query data from the new host 1. (applicable on PoPs only) Flip DNS for `prometheus.svc` record to point to the new host 1. [old host] make sure puppet stays disabled, and thanos-sidecar@ops does not run. Ideally decom the host ASAP.

ATM we're supporting Buster and Bullseye for Prometheus (ops instance, deployed to every site). However for e.g. blackbox-exporter configuration options it'd be nice to have all hosts on Bullseye. Procedure to move data from old hosts to new hosts (to be tested, and adjusted, and moved to wikitech) === Preflight checks === 1. hosts are running `prometheus` role and show up in all prometheus hosts lists (e.g. `prometheus_all_nodes`) 1. ACLs on network devices have been updated 1. mysqld grants are updated (to be verified) === Migration === 1. [new host] stop puppet / prometheus / thanos-sidecar@ops 1. Initial rsync of data old -> new 1. [old host] stop puppet / thanos-sidecar@ops / prometheus@ops . Note that once thanos-sidecar@ops is stopped here then Thanos won't be able to query data for the PoP 1. Final rsync of data old -> new 1. [new host] chown -R prometheus:prometheus /srv/prometheus/ops 1. [new host] set replica label in puppet to match the old hosts', merge the change 1. [new host] re-enable puppet and run puppet, this will restart prometheus and thanos-sidecar@ops, thus Thanos will be able to query data from the new host 1. (applicable on PoPs only) Flip DNS for `prometheus.svc` record to point to the new host 1. [old host] make sure puppet stays disabled, and thanos-sidecar@ops does not run. Ideally decom the host ASAP.

ATM we're supporting Buster and Bullseye for Prometheus (ops instance, deployed to every site). However for e.g. blackbox-exporter configuration options it'd be nice to have all hosts on Bullseye. Procedure to move data from old hosts to new hosts (to be tested, and adjusted, and moved to wikitech) === Preflight checks === 1. hosts are running `prometheus` role and show up in all prometheus hosts lists (e.g. `prometheus_all_nodes`) 1. ACLs on network devices have been updated 1. mysqld grants are updated (to be verified) === Migration === 1. [new host] stop puppet / prometheus / thanos-sidecar@ops 1. Initial rsync of data old -> new 1. [old host] stop puppet / thanos-sidecar@ops / prometheus@ops . Note that once thanos-sidecar@ops is stopped here then Thanos won't be able to query data for the PoP 1. Final rsync of data old -> new 1. [new host] chown -R prometheus:prometheus /srv/prometheus/ops 1. [new host] set replica label in puppet to match the old hosts', merge the change 1. [new host] re-enable puppet and run puppet, this will restart prometheus and thanos-sidecar@ops, thus Thanos will be able to query data from the new host 1. (applicable on PoPs only) Flip DNS for `prometheus.svc` record to point to the new host 1. [old host] make sure puppet stays disabled, and thanos-sidecar@ops does not run. Ideally decom the host ASAP.