Page MenuHomePhabricator

Upgrade Prometheus to 2.7 in deployment-prep and tools
Closed, ResolvedPublic

Description

This task is to track the upgrade to Prometheus to v2 in beta / deployment-prep and tools. Both for testing purposes and consistency with production.

Event Timeline

fgiunchedi triaged this task as Normal priority.

Change 486051 had a related patch set uploaded (by Filippo Giunchedi; owner: Filippo Giunchedi):
[operations/puppet@production] prometheus: add feature flag for v2 compat

https://gerrit.wikimedia.org/r/486051

Change 486051 merged by Filippo Giunchedi:
[operations/puppet@production] prometheus: add feature flag for v2 compat

https://gerrit.wikimedia.org/r/486051

fgiunchedi moved this task from Backlog to Doing on the User-fgiunchedi board.

Change 488344 had a related patch set uploaded (by Filippo Giunchedi; owner: Filippo Giunchedi):
[operations/puppet@production] prometheus: use v2 rules for beta

https://gerrit.wikimedia.org/r/488344

Change 488344 merged by Filippo Giunchedi:
[operations/puppet@production] prometheus: use v2 rules for beta

https://gerrit.wikimedia.org/r/488344

Mentioned in SAL (#wikimedia-cloud) [2019-02-06T14:00:44Z] <godog> switch beta-prometheus to deployment-prometheus02 - T215272

Mentioned in SAL (#wikimedia-cloud) [2019-02-06T14:12:23Z] <godog> shut off deployment-prometheus01 - T215272

Mentioned in SAL (#wikimedia-cloud) [2019-02-07T08:41:13Z] <godog> upgrade prometheus-02 to prometheus 2.6 - T215272

Conversion of tools-prometheus-02 worked as expected, I've stopped v1, moved v1 metrics out of the way and installed v2 package, then started v2. Once v2 is running and starts collecting metrics into an empty storage I've ran prometheus-storage-migrator on v1 data. That took about ~5h, and once complete the migrated data can be merged back into the fresh v2 storage. Overall there has been about 3h worth of missing data, from 6am to 9am (when I've upgraded to v2)

Mentioned in SAL (#wikimedia-cloud) [2019-02-08T11:08:47Z] <godog> flip tools-prometheus.wmflabs.org to tools-prometheus-02 - T215272

fgiunchedi moved this task from Backlog to In progress on the monitoring board.Fri, Feb 8, 11:36 AM
fgiunchedi renamed this task from Upgrade Prometheus to 2.6 in deployment-prep and tools to Upgrade Prometheus to 2.7 in deployment-prep and tools.Mon, Feb 11, 10:49 AM
fgiunchedi closed this task as Resolved.Mon, Feb 11, 10:54 AM
fgiunchedi claimed this task.

Tools and deployment-prep are running Prometheus 2.7.1 rebuilt from unstable with k8s support and their storage has been migrated from Prometheus v1.

The former prometheus instance in beta, deployment-prometheus01 is now off as it is unused and can be deleted.