Page MenuHomePhabricator

Prometheus 2 breaking change
Closed, DeclinedPublic

Description

@fgiunchedi - in starting to upgrade hosts to stretch I ran across this message of sadness:

BREAKING CHANGES
The data format in Prometheus 2.0 has completely changed and is not
backwards compatible with 1.x. Prometheus 2 will create a new data directory
in /var/lib/prometheus/metrics2. The old data in /var/lib/prometheus/metrics
can not be read by the new version.
There is no conversion tool; if you want to retain access to the old data,
it is necessary to run a separate Prometheus 1.x instance on that data
directory. (This package makes no provision to allow this.)

How are you handling this in prod? Thanks!

Event Timeline

Restricted Application added a subscriber: Aklapper. · View Herald TranscriptJan 17 2018, 8:21 PM
DStrine moved this task from Triage to FR-Ops on the Fundraising-Backlog board.Jan 22 2018, 8:47 PM

Good question @cwdent. We haven't tackled the problem in production yet, though IIRC Prometheus suggests setting up a v2 instance with remote reading from the existing v1 instance. This way data that's not present in v2 will be read from v1, when enough time has passed (e.g. the Prometheus retention period) we can decom the v1 instance.

Good question @cwdent. We haven't tackled the problem in production yet, though IIRC Prometheus suggests setting up a v2 instance with remote reading from the existing v1 instance. This way data that's not present in v2 will be read from v1, when enough time has passed (e.g. the Prometheus retention period) we can decom the v1 instance.

Forgot to add, we're tracking the migration in T187987: 100% of Prometheus traffic served by Prometheus v2

@Jgreen we're using prometheus2 for the private stats server, right? Guessing this can be closed.

Jgreen closed this task as Declined.Tue, Oct 8, 3:33 PM
Jgreen moved this task from Backlog to Done on the fundraising-tech-ops board.

There's not really anything to be done here. We've archived the v1 historical data, and could, in theory, spin up a v1 instance to feed it to grafana, but it's just not worth the effort. It's a good reminder not to think of prometheus as a store for historical data, and to consider alternatives.