This task is to track the upgrade to Prometheus to v2 in beta / deployment-prep and tools. Both for testing purposes and consistency with production.
|Resolved||fgiunchedi||T220104 TEC6: Metrics monitoring infrastructure (Q4 2018/19 goal)|
|Resolved||fgiunchedi||T187987 100% of Prometheus traffic served by Prometheus v2|
|Resolved||fgiunchedi||T215272 Upgrade Prometheus to 2.7 in deployment-prep and tools|
Conversion of tools-prometheus-02 worked as expected, I've stopped v1, moved v1 metrics out of the way and installed v2 package, then started v2. Once v2 is running and starts collecting metrics into an empty storage I've ran prometheus-storage-migrator on v1 data. That took about ~5h, and once complete the migrated data can be merged back into the fresh v2 storage. Overall there has been about 3h worth of missing data, from 6am to 9am (when I've upgraded to v2)
Tools and deployment-prep are running Prometheus 2.7.1 rebuilt from unstable with k8s support and their storage has been migrated from Prometheus v1.
The former prometheus instance in beta, deployment-prometheus01 is now off as it is unused and can be deleted.